[SKILL] Sync SGLang skill docs (#23921)
This commit is contained in:
Xiaoyu Zhang
@@ -435,7 +435,7 @@ if torch.cuda.get_device_capability()[0] < 9:
|
||||
|
||||
JIT kernel tests live under `python/sglang/jit_kernel/tests/`. **CI does not run `pytest` in that directory directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` there (and every `bench_*.py` under `benchmark/`), collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
|
||||
|
||||
- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `stage-b-kernel-unit-1-gpu-large` (see `.github/workflows/pr-test-jit-kernel.yml`: `python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-large`).
|
||||
- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `stage-b-kernel-unit-1-gpu-large` on H100 and `stage-b-kernel-unit-1-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `stage-b-kernel-unit-8-gpu-h200`.
|
||||
- **Nightly kernel suite**: `nightly-kernel-1-gpu` with `--nightly` — typically used with `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` in CI for expanded parameter grids (see `python/sglang/jit_kernel/utils.py` → `should_run_full_tests` / `get_ci_test_range`). Wired in `.github/workflows/nightly-test-nvidia.yml` (e.g. `python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error`).
|
||||
|
||||
Registration pattern (module level, **literal** `est_time` and `suite` strings — required for AST parsing):
|
||||
@@ -444,6 +444,8 @@ Registration pattern (module level, **literal** `est_time` and `suite` strings
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-large")
|
||||
# Optional B200/SM100 registration for tests that cover Blackwell-specific code paths
|
||||
# register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-b200")
|
||||
# Optional second registration: same file also listed under the nightly kernel suite
|
||||
# register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True)
|
||||
```
|
||||
@@ -455,7 +457,9 @@ Use `register_cuda_ci(..., disabled="reason")` if the file must stay in-tree but
|
||||
**Run like CI** (from repo root):
|
||||
|
||||
```bash
|
||||
cd test && python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-large
|
||||
(cd test && python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-large)
|
||||
# For B200/SM100-specific coverage:
|
||||
(cd test && python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-b200)
|
||||
```
|
||||
|
||||
For fast iteration you can still run `pytest` on a single file locally; CI coverage is via `run_suite.py`.
|
||||
|
||||
@@ -328,6 +328,10 @@ pytest sgl-kernel/tests/test_scale.py -q
|
||||
python sgl-kernel/benchmark/bench_scale.py
|
||||
```
|
||||
|
||||
PR CI also runs `pr-test-sgl-kernel.yml`, including the B200 job
|
||||
`sgl-kernel-b200-test` when kernel changes are detected. Use that job as the
|
||||
Blackwell coverage signal for AOT `sgl-kernel` changes.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -276,6 +276,7 @@ Large suites are split across matrix jobs using the **LPT (Longest Processing Ti
|
||||
| `stage-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
|
||||
| `stage-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `stage-b-kernel-unit-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `stage-b-kernel-unit-1-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `stage-b-kernel-unit-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `stage-b-kernel-benchmark-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `stage-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
|
||||
@@ -283,10 +284,12 @@ Large suites are split across matrix jobs using the **LPT (Longest Processing Ti
|
||||
| `stage-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
|
||||
| `stage-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
|
||||
| `stage-c-test-deepep-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `stage-c-test-4-gpu-b200` | 4 | `4-gpu-b200` | — |
|
||||
| `stage-c-test-4-gpu-gb200` | 1 (no matrix) | `4-gpu-gb200` | — |
|
||||
| `stage-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
|
||||
| `stage-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
|
||||
| `stage-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
|
||||
| `stage-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
|
||||
|
||||
> **Note**: Kernel suites (`stage-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
> **Note**: Kernel suites (`stage-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `stage-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
**Workflow usage:**
|
||||
```yaml
|
||||
@@ -317,11 +320,11 @@ Determines which test suites to run based on file changes.
|
||||
| Output | Triggers |
|
||||
|--------|----------|
|
||||
| `main_package` | Stage A/B/C test suites |
|
||||
| `sgl_kernel` | Kernel wheel builds + kernel test suites |
|
||||
| `sgl_kernel` | Kernel wheel builds + kernel test suites; also switches B200 jobs to kernel-build runner labels outside `target_stage` mode |
|
||||
| `jit_kernel` | JIT kernel test workflow |
|
||||
| `multimodal_gen` | Multimodal-gen test workflow |
|
||||
|
||||
> **Note**: `sgl_kernel` is forced to `false` when `target_stage` is set, because `sgl-kernel-build-wheels` won't run and wheel artifacts won't be available.
|
||||
> **Note**: In `target_stage` mode, `sgl_kernel` is only active when `include_wheel_build=true`. Without that opt-in, kernel-change reruns fail validation instead of running a target stage without freshly built wheels. Outside `target_stage`, `sgl_kernel=true` switches B200 jobs from `4-gpu-b200` / `4-gpu-b200-low-disk` to `4-gpu-b200-kernel` / `4-gpu-b200-kernel-low-disk`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -46,11 +46,13 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
|
||||
### Step 3: Validate accuracy (sanity check)
|
||||
|
||||
```bash
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-q 20
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
|
||||
```
|
||||
|
||||
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
|
||||
- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
### Step 4: Generate the profile
|
||||
@@ -108,7 +110,7 @@ for i in $(seq 1 120); do
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-q 20
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
|
||||
# Expected: Accuracy > 0.8
|
||||
|
||||
# 4. Profile
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
---
|
||||
name: llm-torch-profiler-analysis
|
||||
description: "Unified LLM torch-profiler triage skill for `sglang`, `vllm`, and `TensorRT-LLM`. Use it to inspect an existing `trace.json(.gz)` or profile directory, or to drive live profiling against a running server and return one three-table report with kernel, overlap-opportunity, and fuse-pattern tables."
|
||||
---
|
||||
|
||||
# Unified LLM Torch Profiler Analysis
|
||||
|
||||
## Overview
|
||||
|
||||
Use this skill for `torch.profiler` analysis across:
|
||||
|
||||
- `sglang`
|
||||
- `vllm`
|
||||
- `TensorRT-LLM`
|
||||
|
||||
There is only one public workflow:
|
||||
|
||||
- `triage`
|
||||
|
||||
Preferred unified entrypoint:
|
||||
|
||||
- [scripts/analyze_llm_torch_profile.py](scripts/analyze_llm_torch_profile.py)
|
||||
|
||||
Backwards-compatibility shim (kept so older `docker exec ... analyze_sglang_torch_profile.py ...` calls keep working; it just forwards to the unified entrypoint):
|
||||
|
||||
- [scripts/analyze_sglang_torch_profile.py](scripts/analyze_sglang_torch_profile.py)
|
||||
|
||||
Markdown bundling helper:
|
||||
|
||||
- [scripts/render_triage_markdown_bundle.py](scripts/render_triage_markdown_bundle.py)
|
||||
|
||||
`triage` always prints the same three tables:
|
||||
|
||||
- kernel table
|
||||
- overlap-opportunity table
|
||||
- fuse-pattern table
|
||||
|
||||
By default, all three tables only render rows at or above `1.0%` cumulative GPU-time share.
|
||||
Rows below that are hidden by default unless the user asks for a lower cutoff.
|
||||
|
||||
Keep the fuse-pattern table source-backed and deterministic.
|
||||
Do not turn it into a fuzzy matcher.
|
||||
|
||||
If exact source-backed matching is weak but a kernel cluster is still close to a known family,
|
||||
add one short note after the tables with exactly one of:
|
||||
|
||||
- `high`
|
||||
- `medium`
|
||||
- `low`
|
||||
|
||||
## Capability Matrix
|
||||
|
||||
| Capability | SGLang | vLLM | TensorRT-LLM |
|
||||
| --- | --- | --- | --- |
|
||||
| Existing trace triage | yes | yes | yes |
|
||||
| Single-trace live capture | yes | yes, if torch profiler is enabled on server | requires profiler control endpoints |
|
||||
| Two-trace mapping+formal triage | yes | yes | yes |
|
||||
| Stage-aware live capture | yes | no | no |
|
||||
| `--profile-prefix` control | yes | usually ignored on HTTP profiler route | usually ignored on HTTP profiler route |
|
||||
|
||||
For TensorRT-LLM, live capture only works when the server exposes `/start_profile` and
|
||||
`/stop_profile`, and when the deployment already provides a shared trace path plus the
|
||||
required env vars.
|
||||
|
||||
## Validation Notes
|
||||
|
||||
This unified workflow has been validated with a `4x H100` matrix across SGLang,
|
||||
vLLM, and TensorRT-LLM. Use these model shapes as representative coverage when
|
||||
refreshing or extending the skill:
|
||||
|
||||
| Model | SGLang | vLLM | TensorRT-LLM | Result |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `mistralai/Mixtral-8x7B-Instruct-v0.1` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; benchmark probes returned direct, non-empty text |
|
||||
| `Qwen/Qwen2.5-32B-Instruct` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; benchmark probes returned direct, non-empty text |
|
||||
| `Qwen/Qwen3-32B` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; vLLM and TensorRT-LLM chat probes often emitted `<think>` prefixes |
|
||||
|
||||
To render a validated run into one markdown document:
|
||||
|
||||
```bash
|
||||
python3 scripts/render_triage_markdown_bundle.py \
|
||||
--analysis-root /path/to/analysis_root \
|
||||
--output /path/to/analysis_bundle.md
|
||||
```
|
||||
|
||||
The bundle groups by model and keeps the three tables for each framework.
|
||||
|
||||
Validation notes:
|
||||
|
||||
- all three frameworks now render kernel, overlap, and fuse tables with separate `extend/prefill` and `decode` sections when the trace contains a clean stage split
|
||||
- SGLang live capture is validated and calls the server profiler API directly instead of shelling out to `sglang.profiler`
|
||||
- SGLang trace flush can lag well beyond a few seconds, so the runner waits longer for artifacts than the earlier implementation
|
||||
- SGLang kernel-site reconstruction keeps sampling disabled in the mapping path so the optimized parser does not perturb SGLang table output; equality rechecks matched for `Mixtral-8x7B-Instruct-v0.1`, `Qwen3-32B`, and `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`
|
||||
- vLLM live capture requires `--output-dir` to match the server `torch_profiler_dir`; the validated H100 flow uses `--profiler-config {"profiler":"torch","torch_profiler_dir":"..."}` and then drives `/start_profile` and `/stop_profile`
|
||||
- TensorRT-LLM validation stays on `--backend pytorch`; the H100 flow writes the trace with `TLLM_TORCH_PROFILE_TRACE` and then analyzes the saved trace
|
||||
- the 2026-04-22 TensorRT-LLM 1.0.0 `py_executor.py` profiler setup still needed a `with_stack=True` override for table-quality Python locations; re-check this on TensorRT-LLM 1.2.1 or any 1.3.x release-candidate image before assuming the override is still required
|
||||
|
||||
## When To Use It
|
||||
|
||||
- inspect a `torch.profiler` trace or profile directory from `sglang`, `vllm`, or `TensorRT-LLM`
|
||||
- profile a live serving endpoint and analyze the result
|
||||
- summarize which kernel families dominate prefill or decode
|
||||
- map kernels back to Python code paths
|
||||
- judge whether a code path still leaves overlap opportunity
|
||||
- check whether an already-known fusion or overlap path should have applied
|
||||
|
||||
## Diffusion Backend Gate
|
||||
|
||||
For diffusion benchmark or profiling work, only analyze traces produced by the native
|
||||
SGLang diffusion backend.
|
||||
|
||||
If the run that generated the trace logs any of:
|
||||
|
||||
- `Falling back to diffusers backend`
|
||||
- `Using diffusers backend`
|
||||
- `Loaded diffusers pipeline`
|
||||
|
||||
stop the workflow instead of analyzing the trace.
|
||||
Handle it as a backend-selection issue, not as native-kernel profiler evidence.
|
||||
|
||||
## Main Flows
|
||||
|
||||
### 1. Single-trace triage from an existing profile dir or trace
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
--input /path/to/profile_dir_or_trace.json.gz
|
||||
```
|
||||
|
||||
Use this when one trace is enough.
|
||||
The overlap table stays conservative in single-trace mode and will tell you when a
|
||||
mapping/formal pair is needed.
|
||||
|
||||
### 2. Single-trace live capture from SGLang
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
--framework sglang \
|
||||
--url http://127.0.0.1:30000 \
|
||||
--output-dir /tmp/llm-profiler/sglang_profile_live \
|
||||
--num-steps 5 \
|
||||
--profile-by-stage
|
||||
```
|
||||
|
||||
The script sends `POST /start_profile` to the SGLang server directly.
|
||||
The script writes `server_args.json`, sends the probe requests after profiling is armed,
|
||||
and waits longer for trace flush than the earlier implementation.
|
||||
|
||||
### 3. Single-trace live capture from vLLM
|
||||
|
||||
Launch vLLM with torch profiler enabled, for example:
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--profiler-config '{"profiler":"torch","torch_profiler_dir":"/tmp/llm-profiler/vllm_profile"}'
|
||||
```
|
||||
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
--framework vllm \
|
||||
--url http://127.0.0.1:8000 \
|
||||
--output-dir /tmp/llm-profiler/vllm_profile \
|
||||
--num-steps 5 \
|
||||
--no-profile-by-stage
|
||||
```
|
||||
|
||||
For vLLM, `--output-dir` must point to the same `torch_profiler_dir` the server uses.
|
||||
The current vLLM profiler config already defaults `torch_profiler_with_stack=true`,
|
||||
so the runner only needs to set `torch_profiler_dir`.
|
||||
|
||||
### 4. Single-trace live capture from TensorRT-LLM
|
||||
|
||||
Use this only when the server exposes `POST /start_profile` and `POST /stop_profile`,
|
||||
and the trace path is shared with the current machine.
|
||||
|
||||
Typical env expectations are:
|
||||
|
||||
- `TLLM_PROFILE_START_STOP=1`
|
||||
- `TLLM_TORCH_PROFILE_TRACE=/shared/path/trace.json` or `.json.gz`
|
||||
|
||||
Then run:
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
--framework trtllm \
|
||||
--url http://127.0.0.1:8000 \
|
||||
--output-dir /shared/path \
|
||||
--num-steps 5 \
|
||||
--no-profile-by-stage
|
||||
```
|
||||
|
||||
If the deployment does not expose the profiler control endpoints, fall back to analyzing
|
||||
an existing trace instead of trying live capture.
|
||||
|
||||
On the current TensorRT-LLM mainline path, `py_executor.py` creates the torch profiler
|
||||
with `record_shapes=True` and `with_modules=True` but not `with_stack=True`.
|
||||
For table-quality validation, use the override generator:
|
||||
|
||||
```bash
|
||||
python3 scripts/make_trtllm_py_executor_override.py \
|
||||
--source /path/to/original/py_executor.py \
|
||||
--output /tmp/llm-profiler/py_executor_with_stack.py
|
||||
```
|
||||
|
||||
The validated TensorRT-LLM flow is:
|
||||
|
||||
1. launch `trtllm-serve` with `TLLM_TORCH_PROFILE_TRACE=/shared/path/trace.json`
|
||||
2. run a few benchmark requests
|
||||
3. analyze the emitted trace with `--input /shared/path/trace.json`
|
||||
|
||||
### 5. Two-trace triage from existing profile dirs or traces
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py triage \
|
||||
--mapping-input /path/to/graph_off_profile_dir \
|
||||
--formal-input /path/to/graph_on_profile_dir
|
||||
```
|
||||
|
||||
Use this when you need stronger overlap attribution and kernel-to-source mapping.
|
||||
|
||||
### 6. Two-trace triage from running servers
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py triage \
|
||||
--framework sglang \
|
||||
--mapping-url http://127.0.0.1:31025 \
|
||||
--formal-url http://127.0.0.1:31026 \
|
||||
--num-steps 5 \
|
||||
--profile-by-stage
|
||||
```
|
||||
|
||||
For `vllm` or `TensorRT-LLM`, use the same shape but pass:
|
||||
|
||||
- `--framework vllm` or `--framework trtllm`
|
||||
- `--mapping-output-dir ...`
|
||||
- `--formal-output-dir ...`
|
||||
- `--no-profile-by-stage`
|
||||
|
||||
## `profile_by_stage`
|
||||
|
||||
`--profile-by-stage` is only meaningful on the SGLang live-capture path.
|
||||
|
||||
- On ordinary non-PD SGLang serving, it is still useful because prefill and decode usually have very different bottlenecks.
|
||||
- On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
|
||||
- PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary `profile_by_stage`.
|
||||
- For `vllm` and `TensorRT-LLM`, disable it with `--no-profile-by-stage`.
|
||||
|
||||
## How To Choose The Triage Shape
|
||||
|
||||
### Single-trace triage
|
||||
|
||||
Use when you want the lowest-friction report:
|
||||
|
||||
- one trace is already available
|
||||
- you mainly want kernel share and fusion clues
|
||||
- you are comparing two runs side by side by running triage once per trace
|
||||
|
||||
Prefer this by default.
|
||||
|
||||
### Two-trace triage
|
||||
|
||||
Use when you need:
|
||||
|
||||
- a stronger overlap answer
|
||||
- graph-off source mapping plus graph-on final behavior
|
||||
- more trustworthy overlap recommendations in the middle table
|
||||
|
||||
1. mapping trace with graph disabled or with the lower-fusion / more-readable config
|
||||
2. formal trace with the real serving optimizations enabled
|
||||
|
||||
Do not call the mapping pass a "fast profile".
|
||||
It exists to recover `kernel -> cpu_op -> python scope`.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Single-trace workflow
|
||||
|
||||
1. If the user only wants a diagnosis, one trace is enough.
|
||||
2. Prefer one-rank traces over merged traces whenever the profiler emitted both.
|
||||
3. For a live server, let the script drive the profiler only when the framework-specific prerequisites are already met.
|
||||
4. Prefer SGLang `--profile-by-stage` unless the user explicitly wants an all-stage mixed trace.
|
||||
5. Create or clean the target trace directory before live capture so the profiler can write artifacts without permission surprises.
|
||||
|
||||
### Two-trace workflow
|
||||
|
||||
1. Produce a mapping trace first with graph disabled or the lower-fusion configuration.
|
||||
2. Produce a formal trace second with the real serving optimizations enabled.
|
||||
3. Run `triage` for the three-table report.
|
||||
4. Read the results in this order:
|
||||
- kernel table
|
||||
- overlap-opportunity table
|
||||
- fuse-pattern table
|
||||
5. Before calling something a "new" optimization idea, compare the top rows against both [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md) and [references/overlap-catalog.md](references/overlap-catalog.md). Check mainline rows first, then the `PR-backed / in-flight` sections. Prefer reporting:
|
||||
- an existing fused or overlap path that should already apply here
|
||||
- an existing path that appears disabled, unsupported, or regressed in this trace
|
||||
- an upstream pattern that is mainline elsewhere but missing locally, or still open upstream
|
||||
- a truly new opportunity only when no catalog entry fits
|
||||
6. If no exact pattern fully matches but the trace is still close to a known family, add one flat similarity note after the tables.
|
||||
Use `high`, `medium`, or `low` only.
|
||||
Base that note on the full pattern shape, not on one kernel name alone.
|
||||
Prefer semantic cues such as producer-consumer chain, source locations, CPU op names, TP context, and model-specific structure.
|
||||
Do not rewrite the script table itself to include these heuristic judgments.
|
||||
|
||||
## References
|
||||
|
||||
Load these only when needed:
|
||||
|
||||
- [references/source-map.md](references/source-map.md)
|
||||
- upstream SGLang profiler entrypoints and trace-writing paths; still most useful for SGLang-specific source follow-up
|
||||
- [references/heuristics.md](references/heuristics.md)
|
||||
- overlap labels, dependency-risk interpretation, and limits
|
||||
- [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md)
|
||||
- mixed source-backed catalog of existing fuse and overlap patterns, including mainline rows plus PR-backed / in-flight rows
|
||||
- [references/overlap-catalog.md](references/overlap-catalog.md)
|
||||
- overlap-only lookup table across LLM, VLM, diffusion, disaggregation, HiSparse, and speculative scheduling
|
||||
|
||||
## Output Contract
|
||||
|
||||
Return:
|
||||
|
||||
- trace path or generated profile path
|
||||
- framework
|
||||
- model/server args when available
|
||||
- kernel table
|
||||
- overlap-opportunity table
|
||||
- fuse-pattern table
|
||||
- optional similarity note with `high` / `medium` / `low` when exact matching is inconclusive
|
||||
- one short summary of what dominates the run
|
||||
- whether the overlap read came from single-trace triage or mapping/formal two-trace triage
|
||||
+38
-27
@@ -18,7 +18,7 @@ Use it like this:
|
||||
3. If a finding matches an existing row, report it as:
|
||||
- an existing optimization path that is missing, disabled, regressed, or unsupported for the current backend, or
|
||||
- an already-known family that should be re-applied to the current model shape.
|
||||
4. Check the `PR-backed / in-flight` sections too. If a match exists there, do not call it novel; call it an upstream or in-flight pattern instead.
|
||||
4. Check the mainline comparison sections and the `PR-backed / in-flight` sections too. If a match exists there, do not call it novel; call it an upstream or in-flight pattern instead.
|
||||
5. Only call a finding "new" when it does not fit any mainline or PR-backed row in this catalog.
|
||||
|
||||
The `vLLM-origin` sections below are comparative references. They are not
|
||||
@@ -28,16 +28,33 @@ overlap opportunity as novel.
|
||||
|
||||
The catalog is grouped by reusable optimization family, not by one specific model.
|
||||
|
||||
Refresh note `2026-04-22`: rescanned current `sglang`, `flashinfer`,
|
||||
`TensorRT-LLM`, and `vllm` mainline plus rechecked referenced PR state via the
|
||||
GitHub API on `2026-04-22`. Stable current-code families such as Qwen-style
|
||||
shared-expert top-k append, TensorRT-LLM Triton fused add+RMSNorm+FP8 quant,
|
||||
and vLLM `merge_attn_states` attention-output quant are folded into the
|
||||
mainline rows below. Closed-unmerged SGLang
|
||||
[#22410](https://github.com/sgl-project/sglang/pull/22410) and FlashInfer
|
||||
[#2840](https://github.com/flashinfer-ai/flashinfer/pull/2840) were removed
|
||||
from the PR-backed sections. Keep FlashInfer
|
||||
[#3058](https://github.com/flashinfer-ai/flashinfer/pull/3058) /
|
||||
[#3079](https://github.com/flashinfer-ai/flashinfer/pull/3079) in mind because
|
||||
that branch was reverted, and keep vLLM
|
||||
[#40057](https://github.com/vllm-project/vllm/pull/40057) in mind when using
|
||||
B200 FP4 MoE test coverage as a signal: it disables some B200 FP4 MoE layer
|
||||
tests rather than proving the kernel family is absent.
|
||||
|
||||
## 1. LLM / SRT fused-kernel families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
|
||||
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/jit_kernel/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
||||
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
||||
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/jit_kernel/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
@@ -52,6 +69,7 @@ The catalog is grouped by reusable optimization family, not by one specific mode
|
||||
| Fused MLA KV cache write + FP8 quant | `set_mla_kv_buffer_fp8_quant*`<br>`set_mla_kv_buffer_triton_fp8_quant` | `python/sglang/srt/mem_cache/utils.py`<br>`python/sglang/srt/mem_cache/memory_pool.py` | MLA / NSA KV pool path can quantize K and write directly into KV storage without a separate concat-and-quant chain | Treat standalone quant + KV-buffer write on MLA paths as missing existing fusion first. |
|
||||
| Fused MoE router / top-k / softcapping | `FusedMoeRouter`<br>`fused_moe_router*`<br>router GEMM + `topk` + `tanh` | `python/sglang/srt/layers/moe/router.py` | Single fused router kernel covers router matmul, softcapping, and top-k selection | Treat exposed router matmul + softcap + top-k chains as an existing MoE fusion family. |
|
||||
| Fused MoE grouped-topk / gate kernels | `fused_topk_deepseek`<br>`moe_fused_gate`<br>`aiter_fused_topk`<br>`kimi_k2_moe_fused_gate` | `python/sglang/srt/layers/moe/topk.py` | CUDA / ROCm / FlashInfer kernels fuse bias, grouped-topk, renorm, and routed scaling into one gate op | Check backend / model eligibility before proposing a novel router-gate fusion. |
|
||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/jit_kernel/nvfp4.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/fused_moe_method_npu.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
@@ -102,7 +120,7 @@ The catalog is grouped by reusable optimization family, not by one specific mode
|
||||
| Fused add-RMSNorm and one-pass RMSNorm | residual add plus RMSNorm still split on short hidden sizes | `python/sglang/multimodal_gen/runtime/layers/layernorm.py`<br>`python/sglang/jit_kernel/diffusion/triton/rmsnorm_onepass.py` | `fused_add_rmsnorm(...)` and `triton_one_pass_rms_norm(...)` | For short hidden-size diffusion blocks, this is already an established fusion family. |
|
||||
| Fused diffusion QK norm + RoPE | split QK norm and RoPE in diffusion attention blocks | `python/sglang/jit_kernel/diffusion/qknorm_rope.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py::apply_qk_norm_rope` | `fused_inplace_qknorm_rope(...)`, with fallback to QK norm plus `apply_flashinfer_rope_qk_inplace(...)` | Distinguish between missing fused qknorm + rope and the existing FlashInfer RoPE fallback. |
|
||||
| Z-Image fused `norm(x) * tanh(scale) + shift` | `fused_norm_tanh_mul_add`<br>`tanh(gate) * rmsnorm(x)` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py` | CuTeDSL kernel plus runtime helper for Z-Image residual-form modulation | Treat split Z-Image residual-form modulation as a missing existing diffusion fusion, not a novel idea. |
|
||||
| Z-Image fused residual modulation + next norm-scale | `fused_norm_tanh_mul_add_norm_scale`<br>`residual + tanh(gate) * rmsnorm(x)`<br>`ffn_norm1(x) * scale_mlp` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/models/dits/zimage.py` | One CuTeDSL kernel fuses the first residual-form modulation and the next normalization / scale stage | If you see this chain split in Z-Image traces, report it as a missing existing merged fusion family. |
|
||||
| Z-Image fused residual modulation + next norm-scale | `fused_norm_tanh_mul_add_norm_scale`<br>`residual + tanh(gate) * rmsnorm(x)`<br>`ffn_norm1(x) * scale_mlp` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/models/dits/zimage.py` | One CuTeDSL kernel fuses the first residual-form modulation and the next normalization / scale stage | If you see this chain split in Z-Image traces, report it as a missing existing mainline fusion family. |
|
||||
| Nunchaku fused GELU MLP | `_fused_gelu_mlp`<br>`fused_gelu_mlp` | `python/sglang/multimodal_gen/runtime/models/dits/flux.py` | Nunchaku path fuses `fc1 GEMM + GELU + shift + re-quant + fc2.lora_down` before the second GEMM | Treat split GELU-MLP on Nunchaku checkpoints as an existing fused family, not a new discovery. |
|
||||
|
||||
## 5. Diffusion kernel-overlap and async-communication families
|
||||
@@ -117,9 +135,8 @@ The catalog is grouped by reusable optimization family, not by one specific mode
|
||||
|
||||
## 6. PR-backed / in-flight fused-kernel families
|
||||
|
||||
These rows are intentionally not restricted to merged code. If the trace or
|
||||
user request is about upstream work, use these rows to avoid calling an
|
||||
already-known PR family "new".
|
||||
These rows track still-open upstream work or status-sensitive PR families.
|
||||
Stable entries should be folded into the mainline family rows above.
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -128,9 +145,7 @@ already-known PR family "new".
|
||||
| PR `#21889` fused FP4 paged dequant to FP8 + page-table remap | `_dequant_fp4_to_fp8_paged_kernel`<br>`WRITE_PT`<br>`dequant_fp4_paged_decode` | `PR #21889`<br>`python/sglang/srt/layers/attention/nsa/dequant_fp4_to_fp8.py` | Triton kernel reads FP4 pages, writes FP8 directly, and can fuse decode-side page-table remap | Treat this as an upstream in-flight decode-prep fusion family. |
|
||||
| PR `#21491` FlashInfer TRTLLM FP8 MoE with fused shared experts | `num_fused_shared_experts`<br>`trtllm_fp8_block_scale_moe` | `PR #21491`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/models/deepseek_v2.py` | FlashInfer TRTLLM FP8 MoE path can fuse shared experts inside the routed MoE kernel | On FP8 TRTLLM MoE discussions, treat fused shared experts as an upstream pattern that already has a concrete PR. |
|
||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/jit_kernel/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/jit_kernel/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||
| PR `#21952` Gemma4 fused RMSNorm + residual + scalar | `gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>`Gemma4` | `PR #21952`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/models/gemma4_causal.py` | Triton kernel fuses decoder post-FF RMSNorm, residual add, and per-layer scalar multiply into one pass | If Gemma4-style post-FF norm + residual + scalar steps appear split, treat them as an in-flight upstream Triton fuse family. |
|
||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||
| PR `#21977` TorchInductor combo-kernels horizontal Q/K norm fusion | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `PR #21977`<br>`torch._inductor.config.combo_kernels` | TorchInductor horizontally fuses sibling Q-norm and K-norm kernels, often deleting `split_with_sizes` / `clone` ladders in compiled traces | Treat separate Q/K norm ladders in compile-heavy traces as an in-flight compiler-fusion family first. |
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`sgl-kernel/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
|
||||
## 7. PR-backed / in-flight kernel-overlap families
|
||||
@@ -138,7 +153,6 @@ already-known PR family "new".
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#21877` fused down-GEMM + combine superseding SBO | `enable_fused_grouped_gemm_combine`<br>`combine`<br>`down_gemm` | `PR #21877`<br>`python/sglang/srt/server_args.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | Fused combine eliminates the standalone combine window, so SBO is intentionally disabled when this path is on | If the trace discussion is about combine overlap, first classify it as this upstream fused-overlap family. |
|
||||
| PR `#22410` hiSparse H2D transfer overlap with hit-attention | `transfer_stream`<br>`execute_h2d_async`<br>`hit-attention`<br>`merge_state` | `PR #22410`<br>`python/sglang/srt/layers/attention/nsa_backend.py`<br>`python/sglang/srt/hisparse/hisparse_coordinator.py` | hiSparse decode overlaps host-to-device KV transfer on a transfer stream with hit-attention on the compute stream before merging miss-attention work | Treat hit-attention vs H2D KV transfer windows as a concrete in-flight SGLang overlap family first. |
|
||||
|
||||
## 8. FlashInfer mainline fused-kernel families
|
||||
|
||||
@@ -151,10 +165,12 @@ only consumes a subset of that implementation.
|
||||
| FlashInfer activation / gate epilogues | `silu_and_mul`<br>`gelu_tanh_and_mul`<br>`gelu_and_mul`<br>`silu_and_mul_scaled_nvfp4_experts_quantize` | `flashinfer/activation.py`<br>`flashinfer/quantization/fp4_quantization.py` | FlashInfer covers both the plain activation-plus-mul epilogues and the NVFP4 expert-quantized extension used on MoE expert paths | Treat standalone activation, multiply, and expert-side quant ladders as one existing FlashInfer epilogue family first. |
|
||||
| FlashInfer norm / residual / quant epilogues | `rmsnorm_quant`<br>`fused_add_rmsnorm`<br>`fused_add_rmsnorm_quant`<br>`gemma_rmsnorm`<br>`gemma_fused_add_rmsnorm`<br>`fused_rmsnorm_silu`<br>`rmsnorm_fp4quant`<br>`add_rmsnorm_fp4quant` | `flashinfer/norm/__init__.py`<br>`flashinfer/cute_dsl/rmsnorm_fp4quant.py`<br>`flashinfer/cute_dsl/add_rmsnorm_fp4quant.py` | The norm family spans plain RMSNorm derivatives, residual-add epilogues, norm+activation, and direct FP8 / NVFP4 output variants instead of materializing each intermediate | Treat split residual add, norm, activation, and quant chains as one existing FlashInfer epilogue family first. |
|
||||
| FlashInfer allreduce + post-op fusion family | `allreduce_fusion`<br>`AllReduceFusionPattern`<br>`kARResidualRMSNorm`<br>`kARResidualRMSNormFP8Quant`<br>`kARResidualRMSNormFP4Quant`<br>`trtllm_mnnvl_allreduce_fusion` | `flashinfer/comm/allreduce.py`<br>`flashinfer/comm/trtllm_ar.py`<br>`flashinfer/comm/trtllm_mnnvl_ar.py` | TRTLLM and MNNVL backends fuse all-reduce with residual add, RMSNorm, and backend-appropriate quant / norm-output variants | Treat TP collective + norm (+ quant) ladders as an existing FlashInfer fused-collective family first. |
|
||||
| FlashInfer RoPE + FP8 quant / cache-update family | `rope_quantize_fp8`<br>`mla_rope_quantize_fp8`<br>`rope_quantize_fp8_append_paged_kv_cache` | `flashinfer/rope.py` | The RoPE family covers both RoPE+FP8 output and the larger decode / prefill-prep path that also writes K / V directly into paged KV cache | Treat split RoPE, quant, and cache-write ladders as one existing FlashInfer attention-prep family first. |
|
||||
| FlashInfer RoPE + FP8 quant / cache-update family | `rope_quantize_fp8`<br>`mla_rope_quantize_fp8`<br>`rope_quantize_fp8_append_paged_kv_cache`<br>`seqlen=0`<br>`batch_indices < 0` | `flashinfer/rope.py` | The RoPE family covers both RoPE+FP8 output and the larger decode / prefill-prep path that writes K / V directly into paged KV cache, including padding-token / zero-length sequence handling | Treat split RoPE, quant, cache-write, and padding-token ladders as one existing FlashInfer attention-prep family first. |
|
||||
| FlashInfer fused DeepSeek grouped-topk routing | `fused_topk_deepseek`<br>`NoAuxTc` | `flashinfer/fused_moe/fused_routing_dsv3.py` | One kernel performs sigmoid+bias, grouped score reduction, group top-k, expert top-k, and routed renorm for DeepSeek-V3-style routing | Treat router score activation -> grouped top-k -> renorm ladders as an existing FlashInfer router family first. |
|
||||
| FlashInfer fused MoE expert execution | `cutlass_fused_moe`<br>`trtllm_bf16_moe`<br>`trtllm_fp8_per_tensor_scale_moe`<br>`trtllm_fp8_block_scale_moe`<br>`trtllm_fp4_block_scale_moe`<br>`trtllm_mxint4_block_scale_moe` | `flashinfer/fused_moe/core.py` | CUTLASS and TRTLLM backends collapse expert execution, routed combine, and quantized expert variants into fused MoE runners | Treat exposed expert-side tiny GEMM ladders as matching an existing FlashInfer fused-MoE family. |
|
||||
| FlashInfer fused MoE expert execution | `cutlass_fused_moe`<br>`trtllm_bf16_moe`<br>`trtllm_fp8_per_tensor_scale_moe`<br>`trtllm_fp8_block_scale_moe`<br>`trtllm_fp4_block_scale_moe`<br>`trtllm_mxint4_block_scale_moe`<br>`non-gated` | `flashinfer/fused_moe/core.py` | CUTLASS and TRTLLM backends collapse expert execution, routed combine, and quantized expert variants into fused MoE runners, including gated and non-gated FP8 per-tensor cases | Treat exposed expert-side tiny GEMM or non-gated FP8 ladders as matching an existing FlashInfer fused-MoE family. |
|
||||
| FlashInfer CuTeDSL two-stage MoE fusion | `blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion_nvfp4`<br>`blockscaled_contiguous_grouped_gemm_finalize_fusion_nvfp4`<br>`moe_permute`<br>`moe_unpermute` | `flashinfer/fused_moe/cute_dsl/blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion.py`<br>`flashinfer/fused_moe/cute_dsl/blockscaled_contiguous_grouped_gemm_finalize_fusion.py` | The CuTeDSL path fuses gather+GEMM1+SwiGLU in the first stage and finalize+unpermute+scatter-reduce in the second stage, removing standalone `moe_permute` and `moe_unpermute` kernels | Treat multi-kernel MoE ladders around permute / finalize as one existing FlashInfer CuTeDSL family first. |
|
||||
| FlashInfer SM120 FP4 / groupwise GEMM heuristics | `cutlass_fp4_gemm_sm120`<br>`CutlassTileConfigSM120`<br>`group_gemm_nvfp4_nt_groupwise`<br>`group_gemm_mxfp4_nt_groupwise` | `flashinfer/gemm/gemm_base.py`<br>`include/flashinfer/gemm/fp4_gemm_cutlass_template_sm120.h`<br>`include/flashinfer/gemm/group_gemm_nvfp4_groupwise_sm120.cuh`<br>`csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp` | FlashInfer mainline adds SM120-oriented FP4 GEMM selection and b12x CuTeDSL fused-MoE kernels | Treat SM120 FP4 MoE/GEMM tile selection and Blackwell-lite shape restrictions as an upstream FlashInfer kernel family before inventing a local heuristic. |
|
||||
| FlashInfer MoE `routing_replay_out` support | `routing_replay_out`<br>`mPtrRoutingReplayOut`<br>`trtllm_fp8_block_scale_moe` | `flashinfer/fused_moe/core.py`<br>`csrc/trtllm_fused_moe_kernel_launcher.cu`<br>`csrc/fused_moe/noAuxTcKernels.cu` | TRTLLM-gen MoE kernels can optionally emit compact routing replay metadata without a separate routing-side reconstruction pass | Treat routing-replay writes in MoE traces as part of the upstream FlashInfer TRTLLM MoE family, not a separate postprocess opportunity. |
|
||||
|
||||
## 9. FlashInfer mainline kernel-overlap families
|
||||
|
||||
@@ -168,10 +184,7 @@ only consumes a subset of that implementation.
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#2792` RoPE + FP8 quant + paged KV append with padding-token support | `rope_quantize_fp8_append_paged_kv_cache`<br>`seqlen=0`<br>`batch_indices < 0` | `PR #2792`<br>`flashinfer/rope.py`<br>`include/flashinfer/pos_enc.cuh` | Extends the existing RoPE+quant+cache-write family to CUDA-graph padding tokens / zero-length sequences instead of introducing a separate kernel ladder | Treat split padding-token handling around RoPE+cache write as an in-flight upstream FlashInfer family first. |
|
||||
| PR `#2840` CuTeDSL MoE aux-stream overlap race fix | `aux_stream`<br>`use_prealloc`<br>`use_cuda_graph` | `PR #2840`<br>`flashinfer/fused_moe/cute_dsl/fused_moe.py` | Clarifies that async memset overlap is only safe for the preallocated / CUDA-graph case; non-graph mode falls back to main-stream zeroing to avoid races | Treat missing aux-stream overlap in non-graph traces as an intentional safety rule, not a novel opportunity. |
|
||||
| PR `#2720` PDL runtime-API migration | `cudaGridDependencySynchronize`<br>`cudaTriggerProgrammaticLaunchCompletion`<br>`inline PTX` | `PR #2720`<br>`include/flashinfer/comm/trtllm_allreduce_fusion.cuh`<br>`include/flashinfer/pos_enc.cuh` | Repo-wide migration preserves the existing PDL overlap family while replacing inline PTX with CUDA runtime APIs across norm, RoPE, attention, and MoE codepaths | Treat PDL-looking launch groups as an upstream FlashInfer overlap family even when implementation details differ across revisions. |
|
||||
| PR `#2882` FP8 per-tensor TRTLLM MoE non-gated activation | `trtllm_fp8_per_tensor_scale_moe`<br>`non-gated` | `PR #2882`<br>`csrc/trtllm_fused_moe_kernel_launcher.cu` | Extends the existing TRTLLM FP8 fused-MoE family to non-gated activations instead of requiring a separate expert path | Treat non-gated FP8 expert ladders as an in-flight upstream FlashInfer extension first. |
|
||||
|
||||
## 11. TensorRT-LLM-origin fused-kernel families
|
||||
|
||||
@@ -184,9 +197,10 @@ the current `sglang` checkout only carries an analogous implementation.
|
||||
| TensorRT-LLM FlashInfer activation / gate epilogues | `flashinfer_silu_and_mul`<br>`flashinfer_gelu_tanh_and_mul`<br>`auto_deploy::silu_and_mul`<br>post-GEMM `silu` + `mul` | `tensorrt_llm/_torch/custom_ops/flashinfer_custom_ops.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fuse_silu_mul.py`<br>`tensorrt_llm/_torch/models/modeling_gemma3.py` | Runtime custom ops and AutoDeploy rewrite `split/getitem + activation + mul` MLP epilogues into one FlashInfer op, including Gemma3 `gelu_tanh_and_mul` | Treat split gate activation + multiply as an existing TensorRT-LLM/FlashInfer epilogue family first. |
|
||||
| TensorRT-LLM FlashInfer RMSNorm family | `flashinfer_rmsnorm`<br>`flashinfer_gemma_rmsnorm`<br>`auto_deploy::flashinfer_rms_norm` | `tensorrt_llm/_torch/custom_ops/flashinfer_custom_ops.py`<br>`tensorrt_llm/_torch/modules/rms_norm.py`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/normalization/rms_norm.py` | Runtime modules and AutoDeploy can lower plain RMSNorm and Gemma RMSNorm directly to FlashInfer kernels | Treat split RMSNorm ladders as an existing TensorRT-LLM norm family before calling them novel. |
|
||||
| TensorRT-LLM FlashInfer residual add + RMSNorm | `flashinfer_fused_add_rmsnorm`<br>`flashinfer_gemma_fused_add_rmsnorm`<br>`auto_deploy::flashinfer_fused_add_rms_norm_inplace` | `tensorrt_llm/_torch/custom_ops/flashinfer_custom_ops.py`<br>`tensorrt_llm/_torch/modules/rms_norm.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fused_add_rms_norm.py` | Residual add immediately before RMSNorm can collapse to one in-place FlashInfer op, with Gemma variant support | Treat residual add + RMSNorm chains as an existing TensorRT-LLM fused epilogue family first. |
|
||||
| TensorRT-LLM Triton fused residual add + RMSNorm + FP8 quant | `triton_fused_add_rms_norm_quant_fp8`<br>`fuse_rmsnorm_quant_fp8`<br>`fp8 static quant` | `tensorrt_llm/_torch/auto_deploy/custom_ops/normalization/triton_fused_add_rms_norm_quant_fp8.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fuse_rmsnorm_quant_fp8.py` | Mainline AutoDeploy can rewrite residual-add plus RMSNorm plus FP8 static quant into one Triton op that emits BF16 norm output, FP8 quant output, and residual-add output together | Treat split add + norm + FP8 quant ladders as an existing TensorRT-LLM mainline family first. |
|
||||
| TensorRT-LLM FlashInfer RoPE with shared cos/sin cache | `flashinfer_apply_rope_with_cos_sin_cache_inplace`<br>`flashinfer_rope`<br>`cos_sin_cache` | `tensorrt_llm/_torch/modules/rotary_embedding.py`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/rope/flashinfer_rope.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/rope.py` | Runtime path applies in-place RoPE from a shared cos/sin cache, while AutoDeploy can prebuild the full cache and lower diverse RoPE graphs to `flashinfer_rope` | Treat separate cos/sin gather + RoPE application ladders as an existing TensorRT-LLM attention-prep family. |
|
||||
| TensorRT-LLM FlashInfer cached paged attention | `append_paged_kv_cache`<br>`BatchPrefillWithPagedKVCacheWrapper`<br>`BatchDecodeWithPagedKVCacheWrapper`<br>`auto_deploy::flashinfer_attention_mha_with_cache`<br>`read_cache_only` | `tensorrt_llm/_torch/attention_backend/flashinfer.py`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/attention/flashinfer_attention.py`<br>`docs/source/features/attention.md` | FlashInfer attention backend fuses metadata setup, optional paged-KV append, and prefill/decode wrapper execution, including shared-KV and read-cache-only variants in AutoDeploy | Treat metadata + KV-append + cached-attention ladders as one existing TensorRT-LLM cached-attention family first. |
|
||||
| TensorRT-LLM FlashInfer MLA regular prefill | `append_paged_mla_kv_cache`<br>`BatchPrefillWithRaggedKVCacheWrapper`<br>`flashinfer_mla` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Regular MLA prefill writes compressed KV pages and runs FlashInfer ragged prefill instead of a split append-plus-prefill ladder | Treat MLA regular-prefill prep as an existing TensorRT-LLM FlashInfer family first. |
|
||||
| TensorRT-LLM FlashInfer MLA regular prefill | `append_paged_mla_kv_cache`<br>`BatchPrefillWithRaggedKVCacheWrapper`<br>`flashinfer_mla`<br>`rank 256`<br>`gpu append kernel` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Regular MLA prefill writes compressed KV pages and runs FlashInfer ragged prefill instead of a split append-plus-prefill ladder, with rank-256 paged-KV setups using the GPU append path | Treat MLA regular-prefill prep as an existing TensorRT-LLM FlashInfer family first. |
|
||||
| TensorRT-LLM FlashInfer MLA chunked prefill with absorbed `W_kn` | `BatchMLAPagedAttentionWrapper`<br>`chunked prefill`<br>`W_kn`<br>`W_v` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Chunked prefill absorbs `W_kn` into the query-side projection, runs paged MLA attention in compressed space, then projects back with `W_v` | Treat split absorbed-proj + MLA + output-proj ladders as an existing TensorRT-LLM MLA family first. |
|
||||
| TensorRT-LLM FlashInfer MLA decode with absorbed `W_kn` + `W_v` | `plan_decode`<br>`BatchMLAPagedAttentionWrapper`<br>`decode`<br>`W_kn`<br>`W_v` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Decode path reuses the absorbed-query MLA family and projects the compressed attention output back with `W_v` | Treat similar decode-time absorbed MLA ladders as an existing TensorRT-LLM family, not a new idea. |
|
||||
| TensorRT-LLM FlashInfer fused MoE backend | `flashinfer.fused_moe`<br>`trtllm_bf16_moe`<br>`trtllm_fp8_block_scale_moe`<br>`trtllm_fp4_block_scale_moe`<br>`TRTLLM_GEN_FUSED_MOE_USE_FLASHINFER` | `tensorrt_llm/_torch/modules/fused_moe/moe_op_backend.py`<br>`tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py` | TRTLLM-gen MoE can route expert execution and quant helpers through FlashInfer instead of exposing per-expert eager ladders | Treat expert-side tiny GEMM ladders as matching an existing TensorRT-LLM FlashInfer MoE family first. |
|
||||
@@ -197,19 +211,16 @@ the current `sglang` checkout only carries an analogous implementation.
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| TensorRT-LLM multi-stream MLA attention | `multi_stream_mla_attn`<br>`record_event_passthrough`<br>`_aux`<br>`wait_event` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_attn.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | AutoDeploy rewrites MLA Q/KV forks so the KV projection runs on an auxiliary stream while the Q path stays on the caller stream | Treat exposed Q-branch vs KV-branch overlap as an existing TensorRT-LLM multi-stream family first. |
|
||||
| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node | Treat shared-expert vs routed-expert windows as an existing TensorRT-LLM branch-overlap family. |
|
||||
| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node; the same family includes synchronization rules for MLIR-fused kernels and piecewise cudagraph replay | Treat shared-expert vs routed-expert windows, including altered `multi_stream_moe` behavior under MLIR / piecewise graph modes, as an existing TensorRT-LLM branch-overlap family. |
|
||||
| TensorRT-LLM multi-stream FP8 GEMM fork parallelism | `multi_stream_gemm`<br>`trtllm_finegrained_fp8_linear`<br>`record_event_passthrough`<br>`_aux` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_gemm.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Compiler pass identifies fork points with multiple FP8 linears and moves the largest GEMM to the auxiliary stream so sibling GEMMs overlap | Treat sibling FP8 linear branches as an existing TensorRT-LLM overlap family before designing a new stream split. |
|
||||
|
||||
## 13. TensorRT-LLM-origin PR-backed / in-flight fused-kernel and kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#12674` fused residual add + RMSNorm + FP8 quant | `triton_fused_add_rms_norm_quant_fp8`<br>`residual_add`<br>`rms_norm`<br>`fp8 static quant` | `PR #12674`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fuse_rmsnorm_quant_fp8.py` | Open PR adds a pattern-matcher pass that replaces residual-add plus RMSNorm plus FP8 static quant with a fused FlashInfer 0.6.7-backed path | Treat split add + norm + FP8 quant ladders as an in-flight TensorRT-LLM family first. |
|
||||
| PR `#12519` rank-256 `flashinfer_mla` extension | `flashinfer_mla`<br>`rank 256`<br>`paged KV-cache`<br>`gpu append kernel` | `PR #12519`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Open PR extends the existing FlashInfer MLA family with a TRTLLM MLA operator, paged-KV support, and a GPU append kernel for rank-256 setups | Treat rank-256 MLA prep / decode ladders as an in-flight TensorRT-LLM MLA family, not a novel direction. |
|
||||
| PR `#12525` FlashInfer TRTLLM-gen FMHA paged-index / buffer rework | `shared paged index`<br>`trtllm-gen attention`<br>`flashinfer`<br>`kv cache buffer` | `PR #12525`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/attention/flashinfer_attention.py` | Open PR refines the existing FlashInfer TRTLLM-gen cached-attention family by disabling shared paged index and unifying KV-buffer construction | Treat these attention-prep changes as an in-flight implementation evolution of an existing family first. |
|
||||
| PR `#12544` NVFP4 KV cache support in TRTLLM-gen attention | `NVFP4 KV cache`<br>`trtllm-gen attention`<br>`flashinfer` | `PR #12544`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/attention/flashinfer_attention.py` | Open PR extends the cached-attention family so the FlashInfer-backed TRTLLM-gen path can build and consume NVFP4 KV buffers directly | Treat split KV-cache quant + buffer-build ladders as an in-flight TensorRT-LLM attention family first. |
|
||||
| PR `#12738` / `#12557` BF16 TRTLLM-gen MoE through FlashInfer | `bf16 trtllm-gen moe`<br>`flashinfer`<br>`trtllm_bf16_moe` | `PR #12738`<br>`PR #12557`<br>`tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py` | Open PRs extend the TRTLLM-gen MoE family so BF16 expert execution can route through FlashInfer instead of only CUTLASS-like paths | Treat BF16 expert ladders as an in-flight TensorRT-LLM FlashInfer MoE family. |
|
||||
| PR `#12847` `multi_stream_moe` sync fix for MLIR and piecewise cudagraphs | `multi_stream_moe`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `PR #12847`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Open PR preserves the existing multi-stream MoE overlap family while tightening synchronization when MLIR-fused kernels or piecewise cudagraph replay are present | Treat missing or altered `multi_stream_moe` overlap under MLIR / piecewise graph modes as an in-flight TensorRT-LLM rule first. |
|
||||
|
||||
## 14. vLLM-origin fused-kernel families
|
||||
|
||||
@@ -224,7 +235,7 @@ contain the same implementation.
|
||||
| vLLM-origin RMSNorm (+ residual add) + quant | `RMSNormQuantFusionPass`<br>`fused_add_rms_norm_static_fp8_quant`<br>`per_token_quant`<br>`per_group_quant` | `vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/compilation/passes/fusion/rocm_aiter_fusion.py` | Compile-time and ROCm AITER paths fuse RMSNorm or fused-add-RMSNorm with FP8 / FP4 quant output | Treat split norm/add + quant as an upstream fused family, not an unexplored direction. |
|
||||
| vLLM-origin SiLU+Mul + quant | `ActivationQuantFusionPass`<br>`SiluMulFp8*`<br>`Nvfp4`<br>`rocm_aiter` | `vllm/compilation/passes/fusion/act_quant_fusion.py`<br>`vllm/compilation/passes/fusion/rocm_aiter_fusion.py` | Activation epilogues fuse `SiLU+Mul` with FP8 / NVFP4 / AITER group quant instead of materializing the BF16 activation first | Treat standalone activation then quant kernels as matching a vLLM-origin precedent. |
|
||||
| vLLM-origin add + RMSNorm + pad | `fuse_act_padding`<br>`RocmAiterTritonAddRMSNormPadFusionPass`<br>`add_rmsnorm_pad` | `vllm/compilation/passes/fusion/rocm_aiter_fusion.py`<br>`docs/design/fusions.md` | ROCm / AITER path fuses residual add + RMSNorm directly into the padded layout expected by the next kernel | Treat norm-plus-padding ladders as an existing backend-specific fuse family first. |
|
||||
| vLLM-origin attention + output quant | `fuse_attn_quant`<br>`AttnQuantFusionPass`<br>`output_scale`<br>`output_block_scale` | `vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/backends/`<br>`docs/design/fusions.md` | Compile-time fusion pushes FP8 / NVFP4 quantization into the attention epilogue on supported Triton / FlashInfer / ROCm / AITER backends | Treat attention-output quant kernels as a known upstream epilogue fusion family before calling them novel. |
|
||||
| vLLM-origin attention + output quant | `fuse_attn_quant`<br>`AttnQuantFusionPass`<br>`merge_attn_states`<br>`output_scale`<br>`output_group_scale`<br>`output_block_scale` | `vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/ops/merge_attn_states.py`<br>`vllm/csrc/attention/merge_attn_states.cu`<br>`docs/design/fusions.md` | Compile-time fusion pushes FP8 / NVFP4 quantization into the attention epilogue on supported Triton / FlashInfer / ROCm / AITER backends, and mainline `merge_attn_states` kernels already support FP8 output when `output_scale` is provided | Treat attention-output quant and merged-attention quant epilogues as a known upstream family before calling them novel. |
|
||||
| vLLM-origin fused QK RMSNorm + RoPE | `fused_qk_norm_rope`<br>`QKNormRoPEFusionPass`<br>`qk norm + rope` | `vllm/compilation/passes/fusion/qk_norm_rope_fusion.py`<br>`vllm/_custom_ops.py`<br>`csrc/fused_qknorm_rope_kernel.cu` | Compile-time and direct custom-op paths fuse per-head Q / K RMSNorm with RoPE | Treat split QK norm + RoPE as a clear vLLM-origin precedent. |
|
||||
| vLLM-origin fused reshape + KV cache write | `reshape_and_cache`<br>`triton_reshape_and_cache_flash`<br>`kv cache write` | `vllm/v1/attention/ops/triton_reshape_and_cache_flash.py`<br>`vllm/v1/attention/backends/triton_attn.py` | Triton cache-update kernels reshape K / V into paged-cache layout and can include FP8 KV-cache scale/write logic | Treat reshape / transpose / cache-write ladders as an existing cache-store fusion family. |
|
||||
| vLLM-origin fused RoPE + KV cache update | `fuse_rope_kvcache`<br>`RopeKVCacheFusionPass`<br>`triton_rope_and_cache` | `vllm/compilation/passes/fusion/rope_kvcache_fusion.py`<br>`vllm/_aiter_ops.py`<br>`docs/design/fusions.md` | ROCm / AITER compile-time fusion combines RoPE with paged KV cache update instead of launching them separately | Treat split RoPE + cache-store as a known upstream family, especially on ROCm-like paths. |
|
||||
@@ -234,6 +245,8 @@ contain the same implementation.
|
||||
| vLLM-origin DSV3 router GEMM | `dsv3_router_gemm`<br>`allow_dsv3_router_gemm`<br>`router logits` | `vllm/_custom_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/dsv3_router_gemm_entry.cu`<br>`csrc/moe/dsv3_router_gemm_float_out.cu` | Hopper-class CUDA kernel specializes the DeepSeek router linear for small decode batches and can emit FP32 logits directly without a generic GEMM chain | Treat DeepSeek-style router linear paths as an existing upstream specialized fuse, distinct from grouped-topk itself. |
|
||||
| vLLM-origin GPT-OSS router GEMM | `gpt_oss_router_gemm`<br>`router gemm` | `vllm/_custom_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/gpt_oss_router_gemm.cu` | Model-specific CUDA kernel replaces the router linear plus bias path with one specialized GEMM op | Treat GPT-OSS-style router linear chains as an existing upstream specialized fuse. |
|
||||
| vLLM-origin DeepSeek min-latency fused QKV-A projection | `dsv3_fused_a_gemm`<br>`fused_qkv_a_proj`<br>`q_a_proj` | `vllm/model_executor/models/deepseek_v2.py`<br>`vllm/_custom_ops.py`<br>`csrc/dsv3_fused_a_gemm.cu` | Hopper-class CUDA kernel replaces the tiny-batch DeepSeek QKV-A projection path with one specialized min-latency GEMM instead of a generic linear launch | Treat small-batch DeepSeek QKV-A projection ladders as a known upstream fused kernel family first. |
|
||||
| vLLM-origin DSV3.2 fused indexer projections | `wk_weights_proj`<br>`MergedColumnParallelLinear`<br>`weights_proj` | `vllm/model_executor/models/deepseek_v2.py`<br>`vllm/model_executor/models/deepseek_mtp.py` | DSV3.2 indexer paths can fuse the `wk` and `weights_proj` projections into one GEMM and carry the matching MTP weight-loading path | Treat paired indexer projection chains as a known upstream fused linear family before calling the opportunity novel. |
|
||||
| vLLM-origin MiniMax allreduce_rms kernels | `minimax_allreduce_rms`<br>`minimax_allreduce_rmsnorm`<br>`MiniMax-M2.5`<br>`allreduce_rms` | `vllm/model_executor/models/minimax_m2.py` | TensorRT-LLM-derived MiniMax allreduce-plus-RMSNorm kernels are a concrete upstream TP decode family | Treat MiniMax TP norm + collective ladders as an upstream specialized fusion family. |
|
||||
| vLLM-origin CUTLASS scaled MM with scale / bias epilogue | `cutlass_scaled_mm`<br>`cutlass_scaled_mm_azp`<br>`scaled mm` | `vllm/_custom_ops.py`<br>`vllm/model_executor/kernels/linear/scaled_mm/cutlass.py`<br>`csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu` | CUTLASS kernels fuse activation scales, weight scales, matmul, and optional bias / AZP epilogues | Treat separate scale-mul + GEMM + bias ladders as a vLLM-origin fused linear family first. |
|
||||
| vLLM-origin fused MoE expert execution | `cpu_fused_moe`<br>`rocm_aiter_fused_moe`<br>`FusedMoE` | `vllm/model_executor/layers/fused_moe/layer.py`<br>`vllm/model_executor/layers/fused_moe/cpu_fused_moe.py`<br>`vllm/model_executor/layers/fused_moe/rocm_aiter_fused_moe.py`<br>`vllm/_aiter_ops.py` | MoE backends on CUDA / ROCm / CPU already collapse packed expert execution into fused expert kernels rather than per-expert eager GEMMs | Treat exposed expert-side tiny GEMM ladders as matching an upstream fused-MoE family. |
|
||||
| vLLM-origin fused MoE LoRA | `fused_moe_lora`<br>`fused_moe_lora_fp8`<br>`w13_shrink`<br>`w2_expand` | `vllm/lora/ops/triton_ops/fused_moe_lora_op.py`<br>`vllm/lora/ops/triton_ops/fused_moe_lora_fp8_op.py`<br>`vllm/lora/layers/fused_moe.py` | Triton kernels fuse LoRA shrink / expand work into MoE expert execution, including FP8 variants | Treat MoE-LoRA adapter work as an upstream fused family before proposing a brand new kernel. |
|
||||
@@ -245,21 +258,19 @@ contain the same implementation.
|
||||
| --- | --- | --- | --- | --- |
|
||||
| vLLM-origin AsyncTP GEMM + collective overlap | `fuse_gemm_comms`<br>`fused_matmul_reduce_scatter`<br>`fused_all_gather_matmul` | `vllm/compilation/passes/fusion/collective_fusion.py`<br>`docs/design/fusions.md` | AsyncTP overlaps GEMM with reduce-scatter / all-gather via symmetric-memory collectives | Treat GEMM+comm windows as a clear vLLM-origin overlap precedent first. |
|
||||
| vLLM-origin Sequence Parallelism staging | `enable_sp`<br>`ReduceScatter`<br>`AllGather`<br>`SequenceParallelismPass` | `vllm/compilation/passes/fusion/sequence_parallelism.py`<br>`docs/design/fusions.md` | Sequence-parallel rewrites all-reduce into RS -> local norm -> AG so later passes can overlap comm and compute | Treat RS / AG staging around norm blocks as an upstream overlap-enabling family. |
|
||||
| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/utils/torch_utils.py`<br>`vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py` | MoE shared experts can run on a dedicated aux stream and overlap with router-side work | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
|
||||
| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py`<br>`vllm/model_executor/layers/fused_moe/runner/moe_runner_base.py` | MoE shared experts can record the cloned input on `shared_experts_stream`, wait on the caller stream, run in parallel with router-side work, and rejoin before merge | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
|
||||
| vLLM-origin DCP async all-to-all overlap | `dcp_alltoall`<br>`all_to_all_single`<br>`async_op=True` | `vllm/v1/attention/ops/dcp_alltoall.py` | Output / LSE exchange uses async all-to-all handles instead of serializing collective completion on the main path | Treat DCP all-to-all windows as an upstream async-collective family. |
|
||||
|
||||
## 16. vLLM-origin PR-backed / in-flight fused-kernel and kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Open PR overlaps the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
|
||||
| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Closed PR explored overlapping the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
|
||||
| PR `#37110` Triton attention + per-group FP8 dynamic quant | `group_size=128`<br>`group_size=64`<br>`output_group_scale`<br>`per-group FP8` | `PR #37110`<br>`vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/ops/triton_unified_attention.py` | In-flight Triton attention epilogue computes per-group FP8 scales and quantizes output directly instead of launching a separate group-quant kernel | Treat attention + per-group FP8 quant as a concrete upstream vLLM family, not a novel idea. |
|
||||
| PR `#38445` MiniMax-M2 FP32 gate kernel | `fp32_router_gemm`<br>`MiniMax-M2`<br>`gate kernel` | `PR #38445`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/model_executor/models/minimax_m2.py` | Draft CUDA kernel fuses BF16->FP32 conversion and low-batch router GEMM for MiniMax-M2, replacing up to three kernels on the gate path | Treat MiniMax-M2 gate ladders as an in-flight upstream fused router family first. |
|
||||
| PR `#38621` fused QK norm + RoPE + cache + quant | `fused_qk_norm_rope_cache_quant`<br>`QK Norm + RoPE + Cache + Quant` | `PR #38621`<br>`csrc/fused_qk_norm_rope_cache_quant.cu`<br>`vllm/compilation/passes/fusion/qk_norm_rope_cache_quant_fusion.py` | Draft CUDA kernel and compile-time pass try to fuse QK RMSNorm, RoPE, KV cache write, and optional FP8 quant for small-batch decode | Treat this as an in-flight upstream fusion family before calling a similar idea novel. |
|
||||
| PR `#38684` DSV3.2 fused `wk + weights_proj` | `wk_weights_proj`<br>`MergedColumnParallelLinear`<br>`weights_proj` | `PR #38684`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/model_executor/models/deepseek_mtp.py` | Merged PR fuses the DSV3.2 indexer `wk` and `weights_proj` projections into one GEMM; FP8 weight-loading caveats are being handled in follow-up `PR #38870` | Treat paired indexer projections as a concrete upstream fused linear family before calling the opportunity novel. |
|
||||
| PR `#37646` ROCm AITER fused allreduce + RMSNorm | `rocm_aiter_fused_allreduce_rmsnorm`<br>`custom_fused_ar_rms`<br>`RocmAiterAllReduceFusionPass` | `PR #37646`<br>`vllm/_aiter_ops.py`<br>`vllm/compilation/passes/pass_manager.py` | ROCm-specific compile-time path swaps the generic all-reduce fusion pass for an AITER fused allreduce-plus-RMSNorm kernel family | Treat ROCm TP all-reduce + RMSNorm ladders as an in-flight upstream fused-collective family first. |
|
||||
| PR `#36413` FlashInfer RMSNorm + FP4 quant fusion | `fuse_norm_quant`<br>`flashinfer`<br>`NVFP4`<br>`rmsnorm + fp4 quant` | `PR #36413`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/docs/design/fusions.md` | FlashInfer-backed norm-plus-FP4 quant fusion extends the existing RMSNorm+quant family to NVFP4 flows | Treat split RMSNorm + FP4 quant ladders as an upstream in-flight family, not a fresh idea. |
|
||||
| PR `#37045` MiniMax TRTLLM `minimax_allreduce_rms` kernels | `minimax_allreduce_rms`<br>`MiniMax-M2.5`<br>`allreduce_rms` | `PR #37045`<br>`vllm/model_executor/models/minimax_m2.py` | Draft kernel ports TensorRT-LLM MiniMax allreduce-plus-RMSNorm kernels into vLLM for TP MiniMax decode | Treat MiniMax TP norm + collective ladders as an in-flight upstream specialized fusion family. |
|
||||
| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | Extends the specialized router GEMM family to GLM5 hidden size and uses PDL to overlap the router launch with the preceding fused allreduce-plus-RMS block | Treat this as an in-flight upstream router-kernel plus launch-overlap family before calling it novel. |
|
||||
|
||||
## 17. Important toggles and caveats
|
||||
@@ -312,7 +323,7 @@ FLASHINFER_REPO=${FLASHINFER_REPO:-../flashinfer}
|
||||
TRTLLM_REPO=${TRTLLM_REPO:-../TensorRT-LLM}
|
||||
VLLM_REPO=${VLLM_REPO:-../vllm}
|
||||
|
||||
rg -n "fused_add_rmsnorm|gemma_fused_add_rmsnorm|silu_and_mul|gelu_and_mul|fused_qk_rope_reshape_and_cache|fused_set_kv_buffer|fused_metadata_copy|normal_decode_set_metadata" python/sglang
|
||||
rg -n "fused_add_rmsnorm|gemma_fused_add_rmsnorm|silu_and_mul|gelu_and_mul|fused_qk_rope_reshape_and_cache|fused_set_kv_buffer|fused_metadata_copy|normal_decode_set_metadata|_append_shared_to_topk_output|fused_append_shared_experts_with_weights" python/sglang
|
||||
rg -n "MiniMaxM2RMSNormTP|fused_qknorm_rope|fused_qk_rope_cat_and_cache_mla|fused_qk_norm_mrope_3d_cache_pts_quant_shuffle|split_qkv_rmsnorm_rope|trtllm_fp8_kv_kernel|set_mla_kv_buffer_fp8_quant" python/sglang
|
||||
rg -n "FusedMoeRouter|fused_topk_deepseek|moe_fused_gate|aiter_fused_topk|fused_rms_fp8_group_quant|fast_topk_transform_fused|fused_store_index_k_cache|fused_temperature_softmax|fused_softcap" python/sglang
|
||||
rg -n "fused_qkvzba_split_reshape_cat|fused_gdn_gating|rms_norm_gated|layer_norm_gated|chunk_gated_delta_rule_fwd_kkt_solve_kernel|fused_recurrent_gated_delta_rule_update|fused_mamba_state_scatter_with_mask|_fused_gather_to_staging_kernel|_fused_scatter_from_staging_kernel" python/sglang
|
||||
@@ -322,11 +333,11 @@ rg -n "silu_and_mul|gelu_tanh_and_mul|gelu_and_mul|silu_and_mul_scaled_nvfp4_exp
|
||||
rg -n "AllReduceFusionPattern|allreduce_fusion|trigger_completion_at_end|rope_quantize_fp8|rope_quantize_fp8_append_paged_kv_cache|fused_topk_deepseek|cutlass_fused_moe|trtllm_.*_moe" "$FLASHINFER_REPO/flashinfer"
|
||||
rg -n "aux_stream|use_async_memset|split_device_green_ctx|split_device_green_ctx_by_sm_count|enable_pdl|launch_with_pdl" "$FLASHINFER_REPO/flashinfer" "$FLASHINFER_REPO/include"
|
||||
git -C "$FLASHINFER_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|pdl|stream|rope|kv|quant|topk|moe'
|
||||
rg -n "flashinfer_silu_and_mul|flashinfer_gelu_tanh_and_mul|flashinfer_rmsnorm|flashinfer_gemma_rmsnorm|flashinfer_fused_add_rmsnorm|flashinfer_apply_rope_with_cos_sin_cache_inplace" "$TRTLLM_REPO/tensorrt_llm/_torch"
|
||||
rg -n "flashinfer_silu_and_mul|flashinfer_gelu_tanh_and_mul|flashinfer_rmsnorm|flashinfer_gemma_rmsnorm|flashinfer_fused_add_rmsnorm|flashinfer_apply_rope_with_cos_sin_cache_inplace|triton_fused_add_rms_norm_quant_fp8|fuse_rmsnorm_quant_fp8" "$TRTLLM_REPO/tensorrt_llm/_torch"
|
||||
rg -n "flashinfer_attention_mha_with_cache|append_paged_kv_cache|flashinfer_mla|append_paged_mla_kv_cache|flashinfer_cached_ssm|selective_state_update|flashinfer.fused_moe" "$TRTLLM_REPO/tensorrt_llm/_torch" "$TRTLLM_REPO/docs/source"
|
||||
rg -n "multi_stream_moe|multi_stream_mla_attn|multi_stream_gemm|record_event_passthrough|begin_aux_stream_passthrough|end_aux_stream_passthrough|wait_aux_stream_passthrough" "$TRTLLM_REPO/tensorrt_llm/_torch"
|
||||
git -C "$TRTLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|flashinfer|mla|kv cache|multi-stream|stream|rope|rmsnorm|moe'
|
||||
rg -n "fused_add_rms_norm|fused_qk_norm_rope|grouped_topk|topk_softmax|topk_sigmoid|dsv3_router_gemm|dsv3_fused_a_gemm|concat_and_cache_mla_rope_fused|gpt_oss_router_gemm|cutlass_scaled_mm|cpu_fused_moe|fused_moe_lora|triton_pos_embed_interpolate" "$VLLM_REPO/vllm" "$VLLM_REPO/csrc"
|
||||
rg -n "fused_add_rms_norm|merge_attn_states|fused_qk_norm_rope|grouped_topk|topk_softmax|topk_sigmoid|dsv3_router_gemm|dsv3_fused_a_gemm|concat_and_cache_mla_rope_fused|gpt_oss_router_gemm|cutlass_scaled_mm|cpu_fused_moe|fused_moe_lora|triton_pos_embed_interpolate" "$VLLM_REPO/vllm" "$VLLM_REPO/csrc"
|
||||
rg -n "fuse_allreduce_rms|fuse_norm_quant|fuse_act_quant|fuse_attn_quant|enable_qk_norm_rope_fusion|fuse_rope_kvcache|enable_sp|fuse_gemm_comms|RocmAiter|dcp_alltoall|shared_experts_stream|TRTLLM_ENABLE_PDL|wk_weights_proj" "$VLLM_REPO/vllm" "$VLLM_REPO/docs/design/fusions.md" "$VLLM_REPO/csrc"
|
||||
git -C "$VLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|triton|cuda|rope|kv cache|topk|router|allreduce|reduce-scatter|all-gather|all_to_all|quant'
|
||||
# GitHub PR scan terms for the connector or web UI:
|
||||
+22
-18
@@ -16,8 +16,8 @@ Use it like this:
|
||||
3. If a match exists in the mainline sections, report it as an existing
|
||||
overlap family that is missing, disabled, regressed, or unsupported on the
|
||||
current backend.
|
||||
4. If a match exists only in the `PR-backed / in-flight` section, report it as
|
||||
an upstream overlap pattern, not a novel idea.
|
||||
4. If a match exists only in the `PR-backed / in-flight`
|
||||
section, report it as an upstream overlap pattern, not a novel idea.
|
||||
5. Only call an overlap opportunity "new" when no row in this file or
|
||||
`fuse-overlap-catalog.md` fits.
|
||||
|
||||
@@ -26,6 +26,18 @@ necessarily present in the checked-out `sglang` tree, but they should still be
|
||||
treated as upstream or analogous kernel-overlap families before labeling an
|
||||
overlap opportunity as novel.
|
||||
|
||||
Refresh note `2026-04-22`: rescanned current `sglang`, `flashinfer`,
|
||||
`TensorRT-LLM`, and `vllm` mainline overlap paths plus rechecked referenced PR
|
||||
state via the GitHub API on `2026-04-22`. Closed-unmerged SGLang
|
||||
[#22410](https://github.com/sgl-project/sglang/pull/22410) and FlashInfer
|
||||
[#2840](https://github.com/flashinfer-ai/flashinfer/pull/2840) were removed
|
||||
from the PR-backed sections. SGLang
|
||||
[#21877](https://github.com/sgl-project/sglang/pull/21877), FlashInfer
|
||||
[#2720](https://github.com/flashinfer-ai/flashinfer/pull/2720), and vLLM
|
||||
[#35968](https://github.com/vllm-project/vllm/pull/35968) /
|
||||
[#39301](https://github.com/vllm-project/vllm/pull/39301) remain useful
|
||||
upstream overlap references as of this refresh.
|
||||
|
||||
## 1. LLM / SRT kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
@@ -64,7 +76,6 @@ overlap opportunity as novel.
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#21877` fused down-GEMM + combine superseding SBO | `enable_fused_grouped_gemm_combine`<br>`combine`<br>`down_gemm` | `PR #21877`<br>`python/sglang/srt/server_args.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | Fused combine eliminates the standalone combine window, so SBO is intentionally disabled when this path is on | If the trace discussion is about combine overlap, first classify it as this upstream fused-overlap family. |
|
||||
| PR `#22410` hiSparse H2D transfer overlap with hit-attention | `transfer_stream`<br>`execute_h2d_async`<br>`hit-attention`<br>`merge_state` | `PR #22410`<br>`python/sglang/srt/layers/attention/nsa_backend.py`<br>`python/sglang/srt/hisparse/hisparse_coordinator.py` | hiSparse decode overlaps host-to-device KV transfer on a transfer stream with hit-attention on the compute stream before running miss-attention and merge | Treat hit-attention vs H2D KV transfer windows as an in-flight SGLang overlap family first. |
|
||||
|
||||
## 5. FlashInfer kernel-overlap families
|
||||
|
||||
@@ -82,7 +93,6 @@ checkout only calls part of that implementation.
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#2840` CuTeDSL MoE aux-stream overlap race fix | `aux_stream`<br>`use_prealloc`<br>`use_cuda_graph` | `PR #2840`<br>`flashinfer/fused_moe/cute_dsl/fused_moe.py` | Clarifies that async memset overlap is only safe for the preallocated / CUDA-graph case; non-graph mode falls back to main-stream zeroing to avoid races | Treat missing aux-stream overlap in non-graph traces as an intentional safety rule, not a novel opportunity. |
|
||||
| PR `#2720` PDL runtime-API migration | `cudaGridDependencySynchronize`<br>`cudaTriggerProgrammaticLaunchCompletion`<br>`inline PTX` | `PR #2720`<br>`include/flashinfer/comm/trtllm_allreduce_fusion.cuh`<br>`include/flashinfer/pos_enc.cuh` | Repo-wide migration preserves the existing PDL overlap family while replacing inline PTX with CUDA runtime APIs across norm, RoPE, attention, and MoE codepaths | Treat PDL-looking launch groups as an upstream FlashInfer overlap family even when implementation details differ across revisions. |
|
||||
|
||||
## 7. TensorRT-LLM-origin kernel-overlap families
|
||||
@@ -94,32 +104,26 @@ AutoDeploy rather than same-stream PDL windows.
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| TensorRT-LLM multi-stream MLA attention | `multi_stream_mla_attn`<br>`record_event_passthrough`<br>`_aux`<br>`wait_event` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_attn.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | AutoDeploy rewrites MLA Q/KV forks so the KV projection runs on an auxiliary stream while the Q path stays on the caller stream | Treat exposed Q-branch vs KV-branch overlap as an existing TensorRT-LLM multi-stream family first. |
|
||||
| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node | Treat shared-expert vs routed-expert windows as an existing TensorRT-LLM branch-overlap family. |
|
||||
| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node; the same family includes synchronization rules for MLIR-fused kernels and piecewise cudagraph replay | Treat shared-expert vs routed-expert windows, including altered behavior under MLIR / piecewise graph modes, as an existing TensorRT-LLM branch-overlap family. |
|
||||
| TensorRT-LLM multi-stream FP8 GEMM fork parallelism | `multi_stream_gemm`<br>`trtllm_finegrained_fp8_linear`<br>`record_event_passthrough`<br>`_aux` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_gemm.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Compiler pass identifies fork points with multiple FP8 linears and moves the largest GEMM to the auxiliary stream so sibling GEMMs overlap | Treat sibling FP8 linear branches as an existing TensorRT-LLM overlap family before designing a new stream split. |
|
||||
|
||||
## 8. TensorRT-LLM-origin PR-backed / in-flight kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#12847` `multi_stream_moe` sync fix for MLIR and piecewise cudagraphs | `multi_stream_moe`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `PR #12847`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Open PR preserves the existing multi-stream MoE overlap family while tightening synchronization when MLIR-fused kernels or piecewise cudagraph replay are present | Treat missing or altered `multi_stream_moe` overlap under MLIR / piecewise graph modes as an in-flight TensorRT-LLM rule first. |
|
||||
|
||||
## 9. vLLM-origin kernel-overlap families
|
||||
## 8. vLLM-origin kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| vLLM-origin AsyncTP GEMM + collective overlap | `fuse_gemm_comms`<br>`fused_matmul_reduce_scatter`<br>`fused_all_gather_matmul` | `vllm/compilation/passes/fusion/collective_fusion.py`<br>`docs/design/fusions.md` | AsyncTP overlaps GEMM with reduce-scatter / all-gather via symmetric-memory collectives | Treat GEMM+comm windows as a clear vLLM-origin overlap precedent first. |
|
||||
| vLLM-origin Sequence Parallelism staging | `enable_sp`<br>`ReduceScatter`<br>`AllGather`<br>`SequenceParallelismPass` | `vllm/compilation/passes/fusion/sequence_parallelism.py`<br>`docs/design/fusions.md` | Sequence-parallel rewrites all-reduce into RS -> local norm -> AG so later passes can overlap comm and compute | Treat RS / AG staging around norm blocks as an upstream overlap-enabling family. |
|
||||
| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/utils/torch_utils.py`<br>`vllm/model_executor/layers/fused_moe/runner/default_moe_runner.py` | MoE shared experts can run on a dedicated aux stream and overlap with router-side work | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
|
||||
| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py`<br>`vllm/model_executor/layers/fused_moe/runner/moe_runner_base.py` | MoE shared experts can record the cloned input on `shared_experts_stream`, wait on the caller stream, run in parallel with router-side work, and rejoin before merge | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
|
||||
| vLLM-origin DCP async all-to-all overlap | `dcp_alltoall`<br>`all_to_all_single`<br>`async_op=True` | `vllm/v1/attention/ops/dcp_alltoall.py` | Output / LSE exchange uses async all-to-all handles instead of serializing collective completion on the main path | Treat DCP all-to-all windows as an upstream async-collective family. |
|
||||
|
||||
## 10. vLLM-origin PR-backed / in-flight kernel-overlap families
|
||||
## 9. vLLM-origin PR-backed / in-flight kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Open PR overlaps the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
|
||||
| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Closed PR explored overlapping the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
|
||||
| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | The GLM5 router GEMM path explicitly uses PDL so the router kernel can overlap with the preceding fused allreduce-plus-RMS block on supported GPUs | Treat router-GEMM launch overlap on GLM5-like traces as an in-flight upstream family first. |
|
||||
|
||||
## 11. Important toggles and caveats
|
||||
## 10. Important toggles and caveats
|
||||
|
||||
| Toggle / env | Location | Effect on trace interpretation |
|
||||
| --- | --- | --- |
|
||||
@@ -141,7 +145,7 @@ AutoDeploy rather than same-stream PDL windows.
|
||||
| `PassConfig.enable_sp` | `vllm/config/compilation.py` | Enables vLLM's sequence-parallel staging family that creates RS / AG overlap opportunities. |
|
||||
| `PassConfig.fuse_gemm_comms` | `vllm/config/compilation.py` | Enables AsyncTP GEMM + collective overlap and auto-enables `enable_sp` when valid. |
|
||||
|
||||
## 12. Suggested refresh commands
|
||||
## 11. Suggested refresh commands
|
||||
|
||||
These commands are only for maintainers refreshing this catalog by rescanning
|
||||
the local source trees. They are not used by the triage scripts at runtime.
|
||||
@@ -160,7 +164,7 @@ git -C "$FLASHINFER_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overl
|
||||
rg -n "multi_stream_moe|multi_stream_mla_attn|multi_stream_gemm|record_event_passthrough|begin_aux_stream_passthrough|end_aux_stream_passthrough|wait_aux_stream_passthrough" "$TRTLLM_REPO/tensorrt_llm/_torch"
|
||||
rg -n "mlir_elementwise_fusion|piecewise|cudagraph|caller_stream.synchronize" "$TRTLLM_REPO/tensorrt_llm/_torch"
|
||||
git -C "$TRTLLM_REPO" log --all --format='%h %s' | rg -i 'overlap|multi-stream|aux stream|cudagraph|mlir|stream|flashinfer|moe|mla'
|
||||
rg -n "fuse_gemm_comms|enable_sp|fused_matmul_reduce_scatter|fused_all_gather_matmul|shared_experts_stream|dcp_alltoall|async_op=True|aux_stream|maybe_execute_in_parallel" "$VLLM_REPO/vllm" "$VLLM_REPO/docs/design/fusions.md"
|
||||
rg -n "fuse_gemm_comms|enable_sp|fused_matmul_reduce_scatter|fused_all_gather_matmul|shared_experts_stream|maybe_sync_shared_experts_stream|dcp_alltoall|async_op=True|aux_stream|maybe_execute_in_parallel" "$VLLM_REPO/vllm" "$VLLM_REPO/docs/design/fusions.md"
|
||||
git -C "$VLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|allreduce|reduce-scatter|all-gather|all_to_all|stream|multi-stream|triton|cuda|router'
|
||||
# GitHub PR scan terms for the connector or web UI:
|
||||
# "fused OR overlap repo:sgl-project/sglang"
|
||||
@@ -0,0 +1,806 @@
|
||||
"""Compact triage entrypoint for unified LLM torch-profiler analysis."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
import triage_kernel_helpers as kernel_helpers
|
||||
import triage_overlap_helpers as overlap_helpers
|
||||
from profile_common import (
|
||||
discover_trace_targets,
|
||||
framework_display_name,
|
||||
load_server_args,
|
||||
load_trace_json,
|
||||
parse_stage,
|
||||
resolve_framework,
|
||||
run_profiler,
|
||||
)
|
||||
|
||||
MIN_RENDER_SHARE_PCT = 1.0
|
||||
MAPPING_KERNEL_SAMPLE_LIMIT_PER_NAME = 16
|
||||
|
||||
|
||||
def build_triage_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="analyze_llm_torch_profile.py",
|
||||
description=(
|
||||
"Compact LLM torch-profiler triage entrypoint for SGLang, vLLM, and "
|
||||
"TensorRT-LLM. "
|
||||
"This prints three tables: kernel mapping, overlap opportunities, "
|
||||
"and fuse opportunities. "
|
||||
"Use either a single trace/profile input or a mapping+formal two-trace pair."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--framework",
|
||||
type=str,
|
||||
default="auto",
|
||||
choices=["auto", "sglang", "vllm", "trtllm", "tllm", "tensorrt-llm"],
|
||||
help=(
|
||||
"Serving framework. Use auto to detect from trace contents, path hints, "
|
||||
"or URL features."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Single trace file or profile directory to triage.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--url",
|
||||
type=str,
|
||||
default=None,
|
||||
help=(
|
||||
"Running server URL for single-trace triage. SGLang supports direct "
|
||||
"capture through its profiler HTTP API. vLLM and TensorRT-LLM require "
|
||||
"a server-side torch-profiler output path exposed via --output-dir."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help=(
|
||||
"Trace output dir when using --url. For vLLM this should match the "
|
||||
"server's torch_profiler_dir. For TensorRT-LLM it should match the "
|
||||
"directory or file path configured by TLLM_TORCH_PROFILE_TRACE."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--profile-prefix",
|
||||
type=str,
|
||||
default="triage-trace",
|
||||
help=(
|
||||
"Profile prefix when generating a trace from --url. SGLang uses it "
|
||||
"directly; vLLM and TensorRT-LLM may ignore it on the HTTP profiler path."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-input",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Graph-off mapping trace file or directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-url",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Running graph-off server URL for the mapping trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-input",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Formal graph-on trace file or directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-url",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Running graph-on server URL for the formal trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-output-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Trace output dir when using --mapping-url.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-output-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Trace output dir when using --formal-url.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-profile-prefix",
|
||||
type=str,
|
||||
default="mapping-trace",
|
||||
help="Profile prefix for the mapping trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-profile-prefix",
|
||||
type=str,
|
||||
default="formal-trace",
|
||||
help="Profile prefix for the formal trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-steps",
|
||||
type=int,
|
||||
default=5,
|
||||
help="Profiler steps when generating traces from URLs.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--profile-by-stage", action=argparse.BooleanOptionalAction, default=True
|
||||
)
|
||||
parser.add_argument(
|
||||
"--merge-profiles", action=argparse.BooleanOptionalAction, default=False
|
||||
)
|
||||
parser.add_argument("--probe-requests", type=int, default=1)
|
||||
parser.add_argument(
|
||||
"--probe-prompt",
|
||||
type=str,
|
||||
default=(
|
||||
"Repeat the word profiler many times with spaces so the server performs several decode steps. "
|
||||
"Do not add explanations."
|
||||
),
|
||||
)
|
||||
parser.add_argument("--probe-max-new-tokens", type=int, default=None)
|
||||
parser.add_argument("--probe-delay", type=float, default=0.5)
|
||||
parser.add_argument(
|
||||
"--start-step",
|
||||
type=int,
|
||||
default=None,
|
||||
help="SGLang-only profiler start step when generating traces from URLs.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pid-substring",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Restrict overlap analysis to PIDs containing this substring.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--kernel-table-limit",
|
||||
type=int,
|
||||
default=0,
|
||||
help="How many kernel rows to print per stage. Use 0 for all kernels.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--overlap-table-limit",
|
||||
type=int,
|
||||
default=0,
|
||||
help="How many overlap rows to print per stage. Use 0 for all kernels.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def parse_triage_args(argv: Sequence[str]) -> argparse.Namespace:
|
||||
parser = build_triage_parser()
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
single_trace_mode = bool(args.input) or bool(args.url)
|
||||
dual_trace_mode = any(
|
||||
[
|
||||
args.mapping_input,
|
||||
args.mapping_url,
|
||||
args.formal_input,
|
||||
args.formal_url,
|
||||
]
|
||||
)
|
||||
|
||||
if single_trace_mode and dual_trace_mode:
|
||||
parser.error(
|
||||
"Use either single-trace mode (--input/--url) or two-trace mode "
|
||||
"(--mapping-* plus --formal-*), not both."
|
||||
)
|
||||
|
||||
if single_trace_mode:
|
||||
if bool(args.input) == bool(args.url):
|
||||
parser.error("Provide exactly one of --input or --url.")
|
||||
return args
|
||||
|
||||
if bool(args.mapping_input) == bool(args.mapping_url):
|
||||
parser.error("Provide exactly one of --mapping-input or --mapping-url.")
|
||||
if bool(args.formal_input) == bool(args.formal_url):
|
||||
parser.error("Provide exactly one of --formal-input or --formal-url.")
|
||||
return args
|
||||
|
||||
|
||||
def resolve_profile_targets(
|
||||
*,
|
||||
label: str,
|
||||
input_path: Optional[str],
|
||||
url: Optional[str],
|
||||
output_dir: Optional[str],
|
||||
profile_prefix: Optional[str],
|
||||
args: argparse.Namespace,
|
||||
) -> Tuple[List[Path], Optional[dict], str]:
|
||||
if bool(input_path) == bool(url):
|
||||
raise ValueError(f"{label} trace requires exactly one of input path or URL.")
|
||||
|
||||
if url:
|
||||
framework = resolve_framework(
|
||||
args.framework,
|
||||
input_path=Path(output_dir).resolve() if output_dir else None,
|
||||
url=url,
|
||||
)
|
||||
target_dir = run_profiler(
|
||||
url=url,
|
||||
output_dir=output_dir,
|
||||
num_steps=args.num_steps,
|
||||
profile_by_stage=args.profile_by_stage,
|
||||
merge_profiles=args.merge_profiles,
|
||||
profile_prefix=profile_prefix,
|
||||
probe_requests=max(0, args.probe_requests),
|
||||
probe_prompt=args.probe_prompt,
|
||||
probe_max_new_tokens=args.probe_max_new_tokens,
|
||||
probe_delay=args.probe_delay,
|
||||
start_step=args.start_step,
|
||||
framework=framework,
|
||||
framework_hint_path=output_dir,
|
||||
)
|
||||
traces, server_args = discover_trace_targets(target_dir, all_traces=False)
|
||||
resolved_framework = resolve_framework(
|
||||
args.framework,
|
||||
input_path=target_dir,
|
||||
url=url,
|
||||
server_args=server_args,
|
||||
)
|
||||
return traces, server_args, resolved_framework
|
||||
|
||||
resolved = Path(input_path).resolve()
|
||||
traces, server_args = discover_trace_targets(resolved, all_traces=False)
|
||||
if server_args is None:
|
||||
server_args = load_server_args(resolved)
|
||||
framework = resolve_framework(
|
||||
args.framework, input_path=resolved, server_args=server_args
|
||||
)
|
||||
return traces, server_args, framework
|
||||
|
||||
|
||||
def build_mapping_kernel_map(trace_paths: Sequence[Path], framework: str) -> dict:
|
||||
stage_site_stats = defaultdict(
|
||||
lambda: defaultdict(lambda: defaultdict(kernel_helpers.MappingSiteAggregate))
|
||||
)
|
||||
stage_kernel_categories: Dict[str, Dict[str, str]] = defaultdict(dict)
|
||||
global_site_stats = defaultdict(
|
||||
lambda: defaultdict(kernel_helpers.MappingSiteAggregate)
|
||||
)
|
||||
global_kernel_categories: Dict[str, str] = {}
|
||||
|
||||
for trace_path in trace_paths:
|
||||
trace = load_trace_json(trace_path)
|
||||
kernels, cpu_ops, python_frames, launch_events, _, _ = (
|
||||
kernel_helpers.extract_trace_data(trace)
|
||||
)
|
||||
if not kernels:
|
||||
continue
|
||||
cpu_ops_by_external_id = kernel_helpers.build_cpu_op_index(cpu_ops)
|
||||
launches_by_correlation = kernel_helpers.build_launch_index(launch_events)
|
||||
site_context_cache = {}
|
||||
default_stage = parse_stage(trace_path)
|
||||
for stage, stage_kernels in kernel_helpers.group_kernels_by_stage(
|
||||
kernels, default_stage
|
||||
).items():
|
||||
sampled_stage_kernels = (
|
||||
stage_kernels
|
||||
if framework == "sglang"
|
||||
else sample_kernels_for_mapping(stage_kernels)
|
||||
)
|
||||
local_site_stats = kernel_helpers.aggregate_kernel_sites(
|
||||
sampled_stage_kernels,
|
||||
cpu_ops_by_external_id,
|
||||
python_frames,
|
||||
launches_by_correlation=launches_by_correlation,
|
||||
site_context_cache=site_context_cache,
|
||||
)
|
||||
kernel_categories = {
|
||||
kernel.canonical_name: kernel.category for kernel in stage_kernels
|
||||
}
|
||||
kernel_helpers.merge_site_stats(stage_site_stats[stage], local_site_stats)
|
||||
kernel_helpers.merge_site_stats(global_site_stats, local_site_stats)
|
||||
stage_kernel_categories[stage].update(kernel_categories)
|
||||
global_kernel_categories.update(kernel_categories)
|
||||
|
||||
stage_payloads = {
|
||||
stage: kernel_helpers.build_stage_payload(
|
||||
dict(site_stats), stage_kernel_categories.get(stage, {})
|
||||
)
|
||||
for stage, site_stats in stage_site_stats.items()
|
||||
}
|
||||
global_payload = kernel_helpers.build_stage_payload(
|
||||
dict(global_site_stats), global_kernel_categories
|
||||
)
|
||||
return {"stages": stage_payloads, "global": global_payload}
|
||||
|
||||
|
||||
def stage_index(stage: str) -> int:
|
||||
return {"extend": 0, "prefill": 0, "decode": 1, "all": 2}.get(stage, 99)
|
||||
|
||||
|
||||
def sample_kernels_for_mapping(
|
||||
kernels: Sequence[kernel_helpers.KernelEvent],
|
||||
per_name_limit: int = MAPPING_KERNEL_SAMPLE_LIMIT_PER_NAME,
|
||||
) -> List[kernel_helpers.KernelEvent]:
|
||||
if per_name_limit <= 0:
|
||||
return list(kernels)
|
||||
|
||||
grouped: Dict[str, List[kernel_helpers.KernelEvent]] = defaultdict(list)
|
||||
for kernel in kernels:
|
||||
grouped[kernel.canonical_name].append(kernel)
|
||||
|
||||
sampled: List[kernel_helpers.KernelEvent] = []
|
||||
for kernel_name in sorted(grouped):
|
||||
items = grouped[kernel_name]
|
||||
if len(items) <= per_name_limit:
|
||||
sampled.extend(items)
|
||||
continue
|
||||
for sample_idx in range(per_name_limit):
|
||||
pos = round(sample_idx * (len(items) - 1) / (per_name_limit - 1))
|
||||
sampled.append(items[pos])
|
||||
sampled.sort(key=lambda kernel: (kernel.ts, kernel.name))
|
||||
return sampled
|
||||
|
||||
|
||||
def stage_display(stage: str) -> str:
|
||||
return kernel_helpers.stage_label(stage)
|
||||
|
||||
|
||||
def pick_stage_value(stage_to_value: Dict[str, object], stage: str) -> Optional[object]:
|
||||
if stage in stage_to_value:
|
||||
return stage_to_value[stage]
|
||||
if "all" in stage_to_value:
|
||||
return stage_to_value["all"]
|
||||
if len(stage_to_value) == 1:
|
||||
return next(iter(stage_to_value.values()))
|
||||
return None
|
||||
|
||||
|
||||
def render_stages(stage_to_value: Dict[str, object]) -> List[str]:
|
||||
stages = set(stage_to_value)
|
||||
if any(stage != "all" for stage in stages):
|
||||
stages.discard("all")
|
||||
return sorted(stages, key=stage_index)
|
||||
|
||||
|
||||
def build_overlap_stage_bundle_map(
|
||||
trace_paths: Sequence[Path],
|
||||
*,
|
||||
label_prefix: str,
|
||||
server_args: Optional[dict],
|
||||
pid_substring: Optional[str],
|
||||
) -> Dict[str, overlap_helpers.TraceBundle]:
|
||||
stage_bundles: Dict[str, overlap_helpers.TraceBundle] = {}
|
||||
for trace_path in sorted(
|
||||
trace_paths, key=lambda item: (stage_index(parse_stage(item)), item.name)
|
||||
):
|
||||
trace_json = load_trace_json(trace_path)
|
||||
raw_events = trace_json.get(
|
||||
"traceEvents",
|
||||
trace_json if isinstance(trace_json, list) else [],
|
||||
)
|
||||
events, pid = overlap_helpers.extract_kernel_events(trace_json, pid_substring)
|
||||
if not events:
|
||||
continue
|
||||
default_stage = parse_stage(trace_path)
|
||||
stage_groups = overlap_helpers.group_events_by_stage(events, default_stage)
|
||||
for stage in render_stages(stage_groups):
|
||||
if stage in stage_bundles:
|
||||
continue
|
||||
stage_bundles[stage] = overlap_helpers.TraceBundle(
|
||||
label=f"{label_prefix}-{stage}",
|
||||
trace_path=trace_path,
|
||||
server_args=server_args,
|
||||
raw_events=raw_events,
|
||||
events=stage_groups[stage],
|
||||
pid=pid,
|
||||
)
|
||||
if "all" in stage_groups and not stage_bundles:
|
||||
stage_bundles["all"] = overlap_helpers.TraceBundle(
|
||||
label=f"{label_prefix}-all",
|
||||
trace_path=trace_path,
|
||||
server_args=server_args,
|
||||
raw_events=raw_events,
|
||||
events=stage_groups["all"],
|
||||
pid=pid,
|
||||
)
|
||||
return stage_bundles
|
||||
|
||||
|
||||
def group_rows_by_stage(rows: Sequence[dict]) -> List[Tuple[str, List[dict]]]:
|
||||
grouped: Dict[str, List[dict]] = defaultdict(list)
|
||||
for row in rows:
|
||||
grouped[str(row.get("stage") or "all")].append(row)
|
||||
return [
|
||||
(stage, grouped[stage]) for stage in sorted(grouped.keys(), key=stage_index)
|
||||
]
|
||||
|
||||
|
||||
def render_kernel_table_for_stage(rows: Sequence[dict]) -> List[str]:
|
||||
lines = [
|
||||
"| Kernel | Category | GPU time | Share | Launches | Python location (site share) | CPU op |",
|
||||
"| --- | --- | ---: | ---: | ---: | --- | --- |",
|
||||
]
|
||||
if not rows:
|
||||
lines.append(
|
||||
"| No kernel rows at or above 1.0% share. | - | - | - | - | - | - |"
|
||||
)
|
||||
return lines
|
||||
for row in rows:
|
||||
lines.append(
|
||||
"| {kernel} | {category} | {gpu_time} | {share:.1f}% | {launches} | {location} | {cpu_op} |".format(
|
||||
kernel=kernel_helpers.escape_md_cell(row["kernel"]),
|
||||
category=kernel_helpers.escape_md_cell(row["category"]),
|
||||
gpu_time=kernel_helpers.format_ms(row["total_us"]),
|
||||
share=row["share_pct"],
|
||||
launches=row["launches"],
|
||||
location=kernel_helpers.escape_md_cell(row["location"]),
|
||||
cpu_op=kernel_helpers.escape_md_cell(row["cpu_op"]),
|
||||
)
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def render_stage_section_tables(
|
||||
rows: Sequence[dict],
|
||||
*,
|
||||
render_stage_fn,
|
||||
stage_label_prefix: str = "#####",
|
||||
) -> List[str]:
|
||||
if not rows:
|
||||
return render_stage_fn([])
|
||||
stage_groups = group_rows_by_stage(rows)
|
||||
if len(stage_groups) == 1 and stage_groups[0][0] == "all":
|
||||
return render_stage_fn(stage_groups[0][1])
|
||||
|
||||
lines: List[str] = []
|
||||
for index, (stage, stage_rows) in enumerate(stage_groups):
|
||||
lines.append(f"{stage_label_prefix} {stage_display(stage)}")
|
||||
lines.extend(render_stage_fn(stage_rows))
|
||||
if index != len(stage_groups) - 1:
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def render_kernel_tables(rows: Sequence[dict]) -> List[str]:
|
||||
return render_stage_section_tables(
|
||||
rows, render_stage_fn=render_kernel_table_for_stage
|
||||
)
|
||||
|
||||
|
||||
def render_overlap_table_for_stage(rows: Sequence[dict]) -> List[str]:
|
||||
lines = [
|
||||
"| Priority | Verdict | Kernel | Python scope | Formal signal | Dep risk | Recommendation |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- |",
|
||||
]
|
||||
if not rows:
|
||||
lines.append(
|
||||
"| - | - | No rows cleared the 1.0% reporting bar. Use mapping/formal mode for overlap attribution. | - | - | - | - |"
|
||||
)
|
||||
return lines
|
||||
for row in rows:
|
||||
formal_signal = (
|
||||
f"{row['total_us']:.1f} us, share {row['share_pct']:.1f}%, "
|
||||
f"excl {row['exclusive_ratio'] * 100:.1f}% / hid {row['hidden_ratio'] * 100:.1f}%"
|
||||
)
|
||||
lines.append(
|
||||
"| "
|
||||
+ " | ".join(
|
||||
[
|
||||
row["priority"],
|
||||
row["verdict"],
|
||||
kernel_helpers.escape_md_cell(row["kernel"]),
|
||||
kernel_helpers.escape_md_cell(row["python_scope"]),
|
||||
kernel_helpers.escape_md_cell(formal_signal),
|
||||
overlap_helpers.dependency_risk_label(row["dependency_signal"]),
|
||||
row["recommendation"],
|
||||
]
|
||||
)
|
||||
+ " |"
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def render_overlap_tables(rows: Sequence[dict]) -> List[str]:
|
||||
return render_stage_section_tables(
|
||||
rows,
|
||||
render_stage_fn=render_overlap_table_for_stage,
|
||||
)
|
||||
|
||||
|
||||
def render_fuse_table_for_stage(rows: Sequence[dict]) -> List[str]:
|
||||
lines = [
|
||||
"| Pattern | Confidence | Related GPU time | Share | Evidence kernels | Current kernel Python location | Candidate fused Python path | Rationale |",
|
||||
"| --- | --- | ---: | ---: | --- | --- | --- | --- |",
|
||||
]
|
||||
if not rows:
|
||||
lines.append(
|
||||
"| No medium-confidence source-backed fusion opportunity matched this trace. | - | - | - | - | - | - | - |"
|
||||
)
|
||||
return lines
|
||||
for row in rows:
|
||||
lines.append(
|
||||
"| {pattern} | {confidence} | {gpu_time} | {share:.1f}% | {evidence} | {current_locations} | {candidate_path} | {rationale} |".format(
|
||||
pattern=kernel_helpers.escape_md_cell(row["pattern"]),
|
||||
confidence=kernel_helpers.escape_md_cell(row["confidence"]),
|
||||
gpu_time=kernel_helpers.format_ms(row["related_us"]),
|
||||
share=row["share_pct"],
|
||||
evidence=kernel_helpers.escape_md_cell(row["evidence"]),
|
||||
current_locations=kernel_helpers.escape_md_cell(
|
||||
row["current_locations"]
|
||||
),
|
||||
candidate_path=kernel_helpers.escape_md_cell(row["candidate_path"]),
|
||||
rationale=kernel_helpers.escape_md_cell(row["rationale"]),
|
||||
)
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def render_fuse_tables(rows: Sequence[dict]) -> List[str]:
|
||||
return render_stage_section_tables(
|
||||
rows,
|
||||
render_stage_fn=render_fuse_table_for_stage,
|
||||
)
|
||||
|
||||
|
||||
def run_triage(args: argparse.Namespace) -> int:
|
||||
single_trace_mode = bool(args.input) or bool(args.url)
|
||||
if single_trace_mode:
|
||||
formal_traces, formal_server_args, formal_framework = resolve_profile_targets(
|
||||
label="input",
|
||||
input_path=args.input,
|
||||
url=args.url,
|
||||
output_dir=args.output_dir,
|
||||
profile_prefix=args.profile_prefix,
|
||||
args=args,
|
||||
)
|
||||
mapping_traces = formal_traces
|
||||
mapping_server_args = formal_server_args
|
||||
mapping_framework = formal_framework
|
||||
else:
|
||||
mapping_traces, mapping_server_args, mapping_framework = (
|
||||
resolve_profile_targets(
|
||||
label="mapping",
|
||||
input_path=args.mapping_input,
|
||||
url=args.mapping_url,
|
||||
output_dir=args.mapping_output_dir,
|
||||
profile_prefix=args.mapping_profile_prefix,
|
||||
args=args,
|
||||
)
|
||||
)
|
||||
formal_traces, formal_server_args, formal_framework = resolve_profile_targets(
|
||||
label="formal",
|
||||
input_path=args.formal_input,
|
||||
url=args.formal_url,
|
||||
output_dir=args.formal_output_dir,
|
||||
profile_prefix=args.formal_profile_prefix,
|
||||
args=args,
|
||||
)
|
||||
|
||||
mapping_kernel_map = build_mapping_kernel_map(mapping_traces, mapping_framework)
|
||||
|
||||
kernel_rows_rendered: List[dict] = []
|
||||
fuse_rows_rendered: List[dict] = []
|
||||
formal_stage_payloads: Dict[str, dict] = {}
|
||||
|
||||
for formal_trace in formal_traces:
|
||||
trace = load_trace_json(formal_trace)
|
||||
kernels, cpu_ops, python_frames, launch_events, _, _ = (
|
||||
kernel_helpers.extract_trace_data(trace)
|
||||
)
|
||||
if not kernels:
|
||||
continue
|
||||
default_stage = parse_stage(formal_trace)
|
||||
stage_groups = kernel_helpers.group_kernels_by_stage(kernels, default_stage)
|
||||
formal_cpu_ops_by_external_id = kernel_helpers.build_cpu_op_index(cpu_ops)
|
||||
formal_launches_by_correlation = kernel_helpers.build_launch_index(
|
||||
launch_events
|
||||
)
|
||||
formal_site_context_cache = {}
|
||||
for stage_name, stage_kernels in stage_groups.items():
|
||||
local_site_stats = kernel_helpers.aggregate_kernel_sites(
|
||||
stage_kernels,
|
||||
formal_cpu_ops_by_external_id,
|
||||
python_frames,
|
||||
launches_by_correlation=formal_launches_by_correlation,
|
||||
site_context_cache=formal_site_context_cache,
|
||||
)
|
||||
formal_stage_payloads[stage_name] = kernel_helpers.build_stage_payload(
|
||||
local_site_stats,
|
||||
{kernel.canonical_name: kernel.category for kernel in stage_kernels},
|
||||
)
|
||||
trace_total_us = sum(kernel.dur for kernel in kernels)
|
||||
for stage in sorted(stage_groups, key=stage_index):
|
||||
stage_kernels = stage_groups[stage]
|
||||
if not stage_kernels:
|
||||
continue
|
||||
total_us = sum(kernel.dur for kernel in stage_kernels)
|
||||
if (
|
||||
stage == "all"
|
||||
and default_stage == "all"
|
||||
and kernel_helpers.pct(total_us, trace_total_us) < MIN_RENDER_SHARE_PCT
|
||||
):
|
||||
continue
|
||||
kernel_stats = kernel_helpers.aggregate(
|
||||
stage_kernels, key_fn=lambda item: item.canonical_name
|
||||
)
|
||||
kernel_categories = {
|
||||
kernel.canonical_name: kernel.category for kernel in stage_kernels
|
||||
}
|
||||
full_kernel_rows = kernel_helpers.build_kernel_rows(
|
||||
stage=stage,
|
||||
kernel_stats=kernel_stats,
|
||||
kernel_categories=kernel_categories,
|
||||
local_stage_payload=formal_stage_payloads.get(stage, {"kernels": {}}),
|
||||
external_kernel_map=mapping_kernel_map,
|
||||
)
|
||||
visible_kernel_rows = kernel_helpers.limit_kernel_rows(
|
||||
full_kernel_rows, args.kernel_table_limit
|
||||
)
|
||||
for row in visible_kernel_rows:
|
||||
share_pct = kernel_helpers.pct(row.total_us, total_us)
|
||||
if share_pct < MIN_RENDER_SHARE_PCT:
|
||||
continue
|
||||
kernel_rows_rendered.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"kernel": row.name,
|
||||
"category": row.category,
|
||||
"total_us": row.total_us,
|
||||
"share_pct": share_pct,
|
||||
"launches": row.aggregate.count,
|
||||
"location": row.location,
|
||||
"cpu_op": row.cpu_op,
|
||||
}
|
||||
)
|
||||
for item in kernel_helpers.detect_fusion_opportunities(
|
||||
kernel_rows=full_kernel_rows,
|
||||
total_us=total_us,
|
||||
server_args=formal_server_args or mapping_server_args,
|
||||
framework=formal_framework,
|
||||
):
|
||||
share_pct = kernel_helpers.pct(item.related_us, total_us)
|
||||
if share_pct < MIN_RENDER_SHARE_PCT:
|
||||
continue
|
||||
fuse_rows_rendered.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"pattern": item.pattern,
|
||||
"confidence": item.confidence,
|
||||
"related_us": item.related_us,
|
||||
"share_pct": share_pct,
|
||||
"evidence": item.evidence,
|
||||
"current_locations": item.current_locations,
|
||||
"candidate_path": item.candidate_path,
|
||||
"rationale": item.rationale,
|
||||
}
|
||||
)
|
||||
|
||||
overlap_rows_rendered: List[dict] = []
|
||||
if not single_trace_mode:
|
||||
mapping_overlap_bundles = build_overlap_stage_bundle_map(
|
||||
mapping_traces,
|
||||
label_prefix="mapping",
|
||||
server_args=mapping_server_args,
|
||||
pid_substring=args.pid_substring,
|
||||
)
|
||||
formal_overlap_bundles = build_overlap_stage_bundle_map(
|
||||
formal_traces,
|
||||
label_prefix="formal",
|
||||
server_args=formal_server_args,
|
||||
pid_substring=args.pid_substring,
|
||||
)
|
||||
for stage in render_stages(formal_overlap_bundles):
|
||||
formal_bundle = pick_stage_value(formal_overlap_bundles, stage)
|
||||
mapping_bundle = pick_stage_value(mapping_overlap_bundles, stage)
|
||||
if formal_bundle is None or mapping_bundle is None:
|
||||
continue
|
||||
formal_bundle.overlap_stats = overlap_helpers.analyze_overlap(
|
||||
formal_bundle.events
|
||||
)
|
||||
aggregates = overlap_helpers.aggregate_events(formal_bundle.events)
|
||||
source_map = overlap_helpers.build_kernel_source_map(
|
||||
mapping_bundle,
|
||||
kernel_map_entry_lookup=lambda stage_name, kernel_name: (
|
||||
kernel_helpers.lookup_kernel_map_entry(
|
||||
mapping_kernel_map, stage_name, kernel_name
|
||||
)
|
||||
if mapping_kernel_map
|
||||
else None
|
||||
),
|
||||
stage=stage,
|
||||
)
|
||||
source_map = overlap_helpers.merge_source_map_from_kernel_payload(
|
||||
source_map,
|
||||
pick_stage_value(formal_stage_payloads, stage),
|
||||
)
|
||||
stage_rows = overlap_helpers.build_action_rows(
|
||||
aggregates,
|
||||
source_map,
|
||||
formal_bundle.events,
|
||||
formal_bundle.overlap_stats["total_busy_us"],
|
||||
table_limit=max(0, args.overlap_table_limit),
|
||||
)
|
||||
for row in stage_rows:
|
||||
if row.share_pct < MIN_RENDER_SHARE_PCT:
|
||||
continue
|
||||
overlap_rows_rendered.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"priority": row.priority,
|
||||
"verdict": row.verdict,
|
||||
"kernel": row.kernel,
|
||||
"python_scope": row.python_scope,
|
||||
"total_us": row.total_us,
|
||||
"share_pct": row.share_pct,
|
||||
"exclusive_ratio": row.exclusive_ratio,
|
||||
"hidden_ratio": row.hidden_ratio,
|
||||
"dependency_signal": row.dependency_signal,
|
||||
"recommendation": row.recommendation,
|
||||
}
|
||||
)
|
||||
|
||||
lines: List[str] = []
|
||||
lines.append("Triage View")
|
||||
lines.append(f"Mode: {'single-trace' if single_trace_mode else 'mapping-formal'}")
|
||||
if single_trace_mode:
|
||||
lines.append(f"Framework: {framework_display_name(formal_framework)}")
|
||||
lines.append(f"Input traces: {', '.join(str(path) for path in formal_traces)}")
|
||||
else:
|
||||
if mapping_framework == formal_framework:
|
||||
lines.append(f"Framework: {framework_display_name(formal_framework)}")
|
||||
else:
|
||||
lines.append(
|
||||
f"Mapping framework: {framework_display_name(mapping_framework)}"
|
||||
)
|
||||
lines.append(
|
||||
f"Formal framework: {framework_display_name(formal_framework)}"
|
||||
)
|
||||
lines.append(
|
||||
f"Mapping traces: {', '.join(str(path) for path in mapping_traces)}"
|
||||
)
|
||||
lines.append(f"Formal traces: {', '.join(str(path) for path in formal_traces)}")
|
||||
if formal_server_args or mapping_server_args:
|
||||
server_args = formal_server_args or mapping_server_args
|
||||
model = server_args.get("model_path") or server_args.get("model")
|
||||
if model:
|
||||
lines.append(f"Model: {model}")
|
||||
lines.append("")
|
||||
lines.append("Kernel Table")
|
||||
lines.extend(render_kernel_tables(kernel_rows_rendered))
|
||||
lines.append("")
|
||||
lines.append("Overlap Opportunity Table")
|
||||
lines.extend(render_overlap_tables(overlap_rows_rendered))
|
||||
lines.append("")
|
||||
lines.append("Fuse Opportunity Table")
|
||||
lines.extend(render_fuse_tables(fuse_rows_rendered))
|
||||
print("\n".join(lines).rstrip())
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
argv = list(argv or sys.argv[1:])
|
||||
triage_parser = build_triage_parser()
|
||||
|
||||
if not argv or argv[0] in {"-h", "--help"}:
|
||||
triage_parser.print_help()
|
||||
return 0
|
||||
|
||||
if argv[0] == "triage":
|
||||
argv = argv[1:]
|
||||
elif not argv[0].startswith("-"):
|
||||
triage_parser.error(
|
||||
"This skill exposes only the triage workflow. "
|
||||
"Use single-trace mode (--input/--url) or mapping+formal two-trace mode."
|
||||
)
|
||||
return 2
|
||||
|
||||
return run_triage(parse_triage_args(argv))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Backwards-compatibility shim for the unified LLM torch-profiler entrypoint.
|
||||
|
||||
The real implementation now lives in ``analyze_llm_torch_profile`` because this
|
||||
skill covers SGLang, vLLM, and TensorRT-LLM. Older scripts and runbooks that
|
||||
still invoke ``analyze_sglang_torch_profile.py`` keep working by forwarding to
|
||||
that module.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
|
||||
from analyze_llm_torch_profile import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
"""Generate a TensorRT-LLM py_executor override for stable torch-profiler capture."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
START_MARKER = "torch_profiler = torch.profiler.profile("
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProfileCallSpan:
|
||||
start: int
|
||||
end: int
|
||||
block: str
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Create a py_executor.py override that enables with_stack=True for "
|
||||
"TensorRT-LLM torch-profiler traces."
|
||||
)
|
||||
)
|
||||
parser.add_argument("--source", required=True, help="Original py_executor.py path.")
|
||||
parser.add_argument("--output", required=True, help="Override file path to write.")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_profile_call_span(text: str) -> ProfileCallSpan:
|
||||
start = text.find(START_MARKER)
|
||||
if start == -1:
|
||||
raise SystemExit("Could not find torch profiler setup in source file.")
|
||||
|
||||
open_paren = text.find("(", start)
|
||||
if open_paren == -1:
|
||||
raise SystemExit("Malformed torch profiler setup in source file.")
|
||||
|
||||
depth = 0
|
||||
for index in range(open_paren, len(text)):
|
||||
char = text[index]
|
||||
if char == "(":
|
||||
depth += 1
|
||||
elif char == ")":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return ProfileCallSpan(
|
||||
start=start,
|
||||
end=index + 1,
|
||||
block=text[start : index + 1],
|
||||
)
|
||||
raise SystemExit("Could not find the end of the torch profiler call.")
|
||||
|
||||
|
||||
def inject_with_stack(block: str) -> str:
|
||||
if "with_stack=" in block:
|
||||
return block
|
||||
|
||||
lines = block.splitlines()
|
||||
if not lines:
|
||||
raise SystemExit("Unexpected torch profiler block format.")
|
||||
|
||||
last_line = lines[-1]
|
||||
if not last_line.strip():
|
||||
raise SystemExit("Unexpected torch profiler block terminator.")
|
||||
|
||||
if last_line.strip() == ")":
|
||||
if len(lines) < 2:
|
||||
raise SystemExit("Could not find the last torch profiler argument line.")
|
||||
last_arg_index = len(lines) - 2
|
||||
last_arg_line = lines[last_arg_index]
|
||||
indent = last_arg_line[: len(last_arg_line) - len(last_arg_line.lstrip())]
|
||||
if not last_arg_line.rstrip().endswith(","):
|
||||
lines[last_arg_index] = last_arg_line.rstrip() + ","
|
||||
lines.insert(len(lines) - 1, f"{indent}with_stack=True")
|
||||
return "\n".join(lines)
|
||||
|
||||
if not last_line.rstrip().endswith(")"):
|
||||
raise SystemExit("Unexpected torch profiler block terminator.")
|
||||
|
||||
indent = last_line[: len(last_line) - len(last_line.lstrip())]
|
||||
last_arg_text = last_line.rstrip()[:-1].rstrip()
|
||||
if not last_arg_text.endswith(","):
|
||||
last_arg_text += ","
|
||||
lines[-1] = last_arg_text
|
||||
lines.append(f"{indent}with_stack=True)")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def inject_rank0_trace_guard(text: str) -> str:
|
||||
needle = (
|
||||
" enable_torch_trace = bool(torch_trace_path and profile_start_stop)\n"
|
||||
)
|
||||
replacement = (
|
||||
" # Multi-rank PyTorch backend workers race on the same chrome-trace "
|
||||
"path.\n"
|
||||
" # Keep the full torch-profiler trace on rank 0 and let the other "
|
||||
"ranks\n"
|
||||
" # continue with CUDA-profiler gating only.\n"
|
||||
" enable_torch_trace = bool(\n"
|
||||
" torch_trace_path and profile_start_stop and self.dist.rank == 0\n"
|
||||
" )\n"
|
||||
)
|
||||
if replacement in text:
|
||||
return text
|
||||
if needle not in text:
|
||||
raise SystemExit("Could not find enable_torch_trace assignment in source file.")
|
||||
return text.replace(needle, replacement, 1)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
source = Path(args.source).expanduser().resolve()
|
||||
output = Path(args.output).expanduser().resolve()
|
||||
text = source.read_text(encoding="utf-8")
|
||||
span = find_profile_call_span(text)
|
||||
patched_block = inject_with_stack(span.block)
|
||||
patched = (
|
||||
text
|
||||
if patched_block == span.block
|
||||
else (text[: span.start] + patched_block + text[span.end :])
|
||||
)
|
||||
patched = inject_rank0_trace_guard(patched)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(patched, encoding="utf-8")
|
||||
print(output)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,230 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a small correctness and latency probe against an LLM server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib import request
|
||||
|
||||
from profile_common import extract_openai_chat_text
|
||||
|
||||
DEFAULT_PROMPTS = [
|
||||
"用一句中文介绍上海。",
|
||||
"What is 2+2? Answer briefly.",
|
||||
"Write one short haiku about GPUs.",
|
||||
]
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Send a few short requests to an LLM server and record latency plus "
|
||||
"sample outputs."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--framework",
|
||||
required=True,
|
||||
choices=("sglang", "vllm", "trtllm"),
|
||||
help="Serving framework.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--url",
|
||||
required=True,
|
||||
help="Server base URL, for example http://127.0.0.1:30000.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
default=None,
|
||||
help="OpenAI model id. Auto-discovered for vLLM and TensorRT-LLM when omitted.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--requests",
|
||||
type=int,
|
||||
default=6,
|
||||
help="How many probe requests to send.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-tokens",
|
||||
type=int,
|
||||
default=48,
|
||||
help="Generation length for each request.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--timeout",
|
||||
type=float,
|
||||
default=180.0,
|
||||
help="Per-request timeout in seconds.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Optional prompt override. Repeat to add more prompts.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
default=None,
|
||||
help="Optional JSON output path.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def post_json(url: str, payload: Dict[str, Any], timeout: float) -> Dict[str, Any]:
|
||||
req = request.Request(
|
||||
url=url,
|
||||
data=json.dumps(payload).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read()
|
||||
return json.loads(raw.decode("utf-8")) if raw else {}
|
||||
|
||||
|
||||
def get_json(url: str, timeout: float) -> Dict[str, Any]:
|
||||
req = request.Request(url=url, method="GET")
|
||||
with request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read()
|
||||
return json.loads(raw.decode("utf-8")) if raw else {}
|
||||
|
||||
|
||||
def discover_openai_model(base_url: str, timeout: float) -> str:
|
||||
payload = get_json(base_url.rstrip("/") + "/v1/models", timeout=timeout)
|
||||
data = payload.get("data")
|
||||
if not isinstance(data, list) or not data:
|
||||
raise RuntimeError(f"No models returned by {base_url.rstrip('/')}/v1/models")
|
||||
first = data[0]
|
||||
if isinstance(first, dict) and first.get("id"):
|
||||
return str(first["id"])
|
||||
raise RuntimeError(f"Malformed /v1/models payload from {base_url.rstrip('/')}")
|
||||
|
||||
|
||||
def p95(values: List[float]) -> Optional[float]:
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
index = max(0, math.ceil(len(ordered) * 0.95) - 1)
|
||||
return ordered[index]
|
||||
|
||||
|
||||
def sglang_request(base_url: str, prompt: str, max_tokens: int, timeout: float) -> str:
|
||||
payload = {
|
||||
"text": prompt,
|
||||
"sampling_params": {
|
||||
"temperature": 0.0,
|
||||
"max_new_tokens": max_tokens,
|
||||
},
|
||||
"stream": False,
|
||||
}
|
||||
body = post_json(base_url.rstrip("/") + "/generate", payload, timeout=timeout)
|
||||
return str(body.get("text", ""))
|
||||
|
||||
|
||||
def openai_request(
|
||||
base_url: str,
|
||||
model: str,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
timeout: float,
|
||||
) -> Dict[str, str]:
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.0,
|
||||
"max_tokens": max_tokens,
|
||||
"stream": False,
|
||||
}
|
||||
body = post_json(
|
||||
base_url.rstrip("/") + "/v1/chat/completions",
|
||||
payload,
|
||||
timeout=timeout,
|
||||
)
|
||||
text, source = extract_openai_chat_text(body)
|
||||
return {"text": text, "source": source}
|
||||
|
||||
|
||||
def run_probe(args: argparse.Namespace) -> Dict[str, Any]:
|
||||
prompts = args.prompt or list(DEFAULT_PROMPTS)
|
||||
model = args.model
|
||||
if args.framework in {"vllm", "trtllm"} and not model:
|
||||
model = discover_openai_model(args.url, timeout=args.timeout)
|
||||
|
||||
latencies: List[float] = []
|
||||
samples: List[Dict[str, Any]] = []
|
||||
errors: List[Dict[str, str]] = []
|
||||
|
||||
for request_idx in range(args.requests):
|
||||
prompt = prompts[request_idx % len(prompts)]
|
||||
start = time.time()
|
||||
try:
|
||||
if args.framework == "sglang":
|
||||
text = sglang_request(
|
||||
args.url,
|
||||
prompt,
|
||||
max_tokens=args.max_tokens,
|
||||
timeout=args.timeout,
|
||||
)
|
||||
source = "generate.text"
|
||||
else:
|
||||
assert model is not None
|
||||
result = openai_request(
|
||||
args.url,
|
||||
model,
|
||||
prompt,
|
||||
max_tokens=args.max_tokens,
|
||||
timeout=args.timeout,
|
||||
)
|
||||
text = result["text"]
|
||||
source = result["source"]
|
||||
elapsed = time.time() - start
|
||||
latencies.append(elapsed)
|
||||
samples.append(
|
||||
{
|
||||
"prompt": prompt,
|
||||
"latency_s": round(elapsed, 3),
|
||||
"content": text[:240],
|
||||
"source": source,
|
||||
"non_empty": bool(text.strip()),
|
||||
}
|
||||
)
|
||||
except Exception as exc: # pragma: no cover - runtime probe path
|
||||
errors.append({"prompt": prompt, "error": repr(exc)})
|
||||
|
||||
return {
|
||||
"framework": args.framework,
|
||||
"url": args.url,
|
||||
"model": model,
|
||||
"requests": args.requests,
|
||||
"success": len(samples),
|
||||
"errors": len(errors),
|
||||
"all_non_empty": (
|
||||
all(sample["non_empty"] for sample in samples) if samples else False
|
||||
),
|
||||
"avg_latency_s": round(statistics.mean(latencies), 3) if latencies else None,
|
||||
"p95_latency_s": round(p95(latencies), 3) if latencies else None,
|
||||
"samples": samples[:3],
|
||||
"error_samples": errors[:3],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
summary = run_probe(args)
|
||||
rendered = json.dumps(summary, ensure_ascii=False, indent=2)
|
||||
print(rendered)
|
||||
if args.output:
|
||||
output_path = Path(args.output).expanduser().resolve()
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(rendered + "\n", encoding="utf-8")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,880 @@
|
||||
"""Shared helpers for unified LLM torch-profiler skill scripts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Callable, Dict, Iterable, List, Optional, Sequence, Tuple
|
||||
from urllib import request
|
||||
|
||||
STAGE_ORDER = {"extend": 0, "prefill": 0, "decode": 1, "all": 2}
|
||||
FRAMEWORK_LABELS = {
|
||||
"auto": "auto",
|
||||
"sglang": "SGLang",
|
||||
"vllm": "vLLM",
|
||||
"trtllm": "TensorRT-LLM",
|
||||
}
|
||||
TRACE_FILE_PATTERNS = (
|
||||
"*.trace.json",
|
||||
"*.trace.json.gz",
|
||||
"*.pt.trace.json",
|
||||
"*.pt.trace.json.gz",
|
||||
"*.json",
|
||||
"*.json.gz",
|
||||
)
|
||||
TRACE_FILE_IGNORE_NAMES = {
|
||||
"server_args.json",
|
||||
"metadata.json",
|
||||
"config.json",
|
||||
}
|
||||
TRACE_METADATA_NAMES = {
|
||||
"process_name",
|
||||
"thread_name",
|
||||
"process_sort_index",
|
||||
"thread_sort_index",
|
||||
}
|
||||
NON_KERNEL_TRACE_CATEGORIES = ("python_function", "cpu_op", "trace")
|
||||
PYTHON_SCOPE_NAME_PREFIXES = ("python/", "nn.module:")
|
||||
|
||||
|
||||
@lru_cache(maxsize=65536)
|
||||
def _normalize_text_cached(text: str) -> str:
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return ""
|
||||
for token in (" ", "\t", "\n", "\r", "\v", "\f"):
|
||||
if token in text:
|
||||
return " ".join(text.split())
|
||||
return text
|
||||
|
||||
|
||||
def normalize_text(value: object) -> str:
|
||||
return _normalize_text_cached(value if isinstance(value, str) else str(value))
|
||||
|
||||
|
||||
def canonicalize_framework(value: object) -> str:
|
||||
lowered = normalize_text(value).lower().replace("_", "-")
|
||||
aliases = {
|
||||
"": "auto",
|
||||
"auto": "auto",
|
||||
"sglang": "sglang",
|
||||
"sgl": "sglang",
|
||||
"vllm": "vllm",
|
||||
"trt": "trtllm",
|
||||
"tllm": "trtllm",
|
||||
"trtllm": "trtllm",
|
||||
"tensorrt-llm": "trtllm",
|
||||
"tensorrtllm": "trtllm",
|
||||
}
|
||||
return aliases.get(lowered, "auto")
|
||||
|
||||
|
||||
def framework_display_name(value: object) -> str:
|
||||
return FRAMEWORK_LABELS.get(canonicalize_framework(value), str(value))
|
||||
|
||||
|
||||
@lru_cache(maxsize=65536)
|
||||
def _normalize_repo_relative_path_cached(text: str) -> str:
|
||||
text = text.replace("\\", "/")
|
||||
lowered = text.lower()
|
||||
for marker, normalized_marker in (
|
||||
("python/sglang/", "python/sglang/"),
|
||||
("sgl_kernel/", "sgl_kernel/"),
|
||||
("vllm/", "vllm/"),
|
||||
("tensorrt_llm/", "tensorrt_llm/"),
|
||||
("tensorrt-llm/", "tensorrt_llm/"),
|
||||
):
|
||||
idx = lowered.find(marker)
|
||||
if idx != -1:
|
||||
suffix = text[idx + len(marker) :].lstrip("/")
|
||||
return f"{normalized_marker}{suffix}".lstrip("/")
|
||||
idx = lowered.find("sglang/")
|
||||
if idx != -1:
|
||||
return ("python/" + text[idx:]).lstrip("/")
|
||||
return text.lstrip("/")
|
||||
|
||||
|
||||
def normalize_repo_relative_path(path: object) -> str:
|
||||
return _normalize_repo_relative_path_cached(normalize_text(path))
|
||||
|
||||
|
||||
def contains_any_keyword(text: str, keywords: Iterable[str]) -> bool:
|
||||
return any(keyword in text for keyword in keywords)
|
||||
|
||||
|
||||
def coerce_optional_int(value: object) -> Optional[int]:
|
||||
if value in (None, "", "None"):
|
||||
return None
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
return int(value) if value.is_integer() else None
|
||||
try:
|
||||
return int(str(value))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def extract_trace_events(trace: object) -> Sequence[dict]:
|
||||
if isinstance(trace, dict):
|
||||
events = trace.get("traceEvents", [])
|
||||
return events if isinstance(events, list) else []
|
||||
if isinstance(trace, list):
|
||||
return trace
|
||||
return []
|
||||
|
||||
|
||||
def is_trace_metadata_name(name: object) -> bool:
|
||||
return str(name) in TRACE_METADATA_NAMES
|
||||
|
||||
|
||||
def is_complete_duration_event(event: dict) -> bool:
|
||||
if event.get("ph") != "X":
|
||||
return False
|
||||
dur = event.get("dur")
|
||||
ts = event.get("ts")
|
||||
if dur is None or ts is None:
|
||||
return False
|
||||
try:
|
||||
return float(dur) > 0
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def is_annotation_event(name: object, category: object) -> bool:
|
||||
lowered_name = normalize_text(name).lower()
|
||||
lowered_category = normalize_text(category).lower()
|
||||
return "annotation" in lowered_category or lowered_name.startswith("## call ")
|
||||
|
||||
|
||||
def is_non_kernel_trace_category(category: object) -> bool:
|
||||
lowered_category = normalize_text(category).lower()
|
||||
return any(token in lowered_category for token in NON_KERNEL_TRACE_CATEGORIES)
|
||||
|
||||
|
||||
def looks_like_python_scope_name(name: object) -> bool:
|
||||
lowered_name = normalize_text(name).lower()
|
||||
return ".py(" in lowered_name or lowered_name.startswith(PYTHON_SCOPE_NAME_PREFIXES)
|
||||
|
||||
|
||||
def has_stream_marker(args: Optional[dict]) -> bool:
|
||||
trace_args = args or {}
|
||||
return "stream" in trace_args or "cuda_stream" in trace_args
|
||||
|
||||
|
||||
def load_trace_json(path: Path) -> dict:
|
||||
if path.suffix == ".gz":
|
||||
with gzip.open(path, "rt", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
|
||||
|
||||
def load_server_args(path: Path) -> Optional[dict]:
|
||||
resolved = path.resolve()
|
||||
candidate_dirs: List[Path] = []
|
||||
if resolved.is_file():
|
||||
candidate_dirs.extend([resolved.parent, resolved.parent.parent])
|
||||
else:
|
||||
candidate_dirs.extend([resolved, resolved.parent])
|
||||
|
||||
seen: set[Path] = set()
|
||||
for candidate_dir in candidate_dirs:
|
||||
if candidate_dir in seen:
|
||||
continue
|
||||
seen.add(candidate_dir)
|
||||
candidate = candidate_dir / "server_args.json"
|
||||
if candidate.exists():
|
||||
with open(candidate, "r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
return None
|
||||
|
||||
|
||||
def try_get_json(url: str, timeout: float = 60.0) -> Optional[object]:
|
||||
try:
|
||||
with request.urlopen(url, timeout=timeout) as response:
|
||||
raw = response.read()
|
||||
except Exception:
|
||||
return None
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return json.loads(raw.decode("utf-8"))
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
|
||||
def _flatten_chat_text_parts(value: object) -> List[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
text = value.strip()
|
||||
return [text] if text else []
|
||||
if isinstance(value, list):
|
||||
parts: List[str] = []
|
||||
for item in value:
|
||||
parts.extend(_flatten_chat_text_parts(item))
|
||||
return parts
|
||||
if isinstance(value, dict):
|
||||
parts: List[str] = []
|
||||
text_keys = (
|
||||
"text",
|
||||
"content",
|
||||
"reasoning_content",
|
||||
"reasoning",
|
||||
"output_text",
|
||||
)
|
||||
if any(key in value for key in text_keys):
|
||||
for key in text_keys:
|
||||
parts.extend(_flatten_chat_text_parts(value.get(key)))
|
||||
if parts:
|
||||
return parts
|
||||
item_type = normalize_text(value.get("type")).lower()
|
||||
if item_type in {"text", "output_text", "input_text"}:
|
||||
for key in ("text", "content", "value"):
|
||||
parts.extend(_flatten_chat_text_parts(value.get(key)))
|
||||
elif item_type in {"reasoning", "thinking"}:
|
||||
for key in ("text", "content", "reasoning_content", "reasoning"):
|
||||
parts.extend(_flatten_chat_text_parts(value.get(key)))
|
||||
return parts
|
||||
return []
|
||||
|
||||
|
||||
def flatten_chat_text(value: object) -> str:
|
||||
return "\n".join(_flatten_chat_text_parts(value)).strip()
|
||||
|
||||
|
||||
def extract_openai_chat_text(body: object) -> Tuple[str, str]:
|
||||
if not isinstance(body, dict):
|
||||
return "", "invalid_body"
|
||||
|
||||
choices = body.get("choices")
|
||||
if not isinstance(choices, list) or not choices:
|
||||
fallback = flatten_chat_text(body.get("output_text"))
|
||||
if fallback:
|
||||
return fallback, "body.output_text"
|
||||
return "", "missing_choices"
|
||||
|
||||
first_choice = choices[0]
|
||||
if not isinstance(first_choice, dict):
|
||||
return "", "invalid_choice"
|
||||
|
||||
message = first_choice.get("message")
|
||||
if isinstance(message, dict):
|
||||
for key in ("content", "reasoning_content", "reasoning"):
|
||||
text = flatten_chat_text(message.get(key))
|
||||
if text:
|
||||
return text, f"message.{key}"
|
||||
|
||||
for key in ("text", "content", "reasoning_content", "reasoning"):
|
||||
text = flatten_chat_text(first_choice.get(key))
|
||||
if text:
|
||||
return text, f"choice.{key}"
|
||||
|
||||
delta = first_choice.get("delta")
|
||||
if isinstance(delta, dict):
|
||||
for key in ("content", "reasoning_content", "reasoning"):
|
||||
text = flatten_chat_text(delta.get(key))
|
||||
if text:
|
||||
return text, f"delta.{key}"
|
||||
|
||||
fallback = flatten_chat_text(body.get("output_text"))
|
||||
if fallback:
|
||||
return fallback, "body.output_text"
|
||||
return "", "empty"
|
||||
|
||||
|
||||
def detect_framework_from_text(text: object) -> Optional[str]:
|
||||
lowered = normalize_text(text).lower()
|
||||
if not lowered:
|
||||
return None
|
||||
if any(
|
||||
token in lowered
|
||||
for token in (
|
||||
"tensorrt_llm",
|
||||
"tensorrt-llm",
|
||||
"trtllm",
|
||||
"pyexecutor",
|
||||
)
|
||||
):
|
||||
return "trtllm"
|
||||
if "vllm" in lowered:
|
||||
return "vllm"
|
||||
if any(token in lowered for token in ("python/sglang/", "sgl_kernel/", "sglang/")):
|
||||
return "sglang"
|
||||
return None
|
||||
|
||||
|
||||
def detect_framework_from_server_args(server_args: Optional[dict]) -> Optional[str]:
|
||||
if not isinstance(server_args, dict) or not server_args:
|
||||
return None
|
||||
lowered_keys = {normalize_text(key).lower() for key in server_args}
|
||||
if lowered_keys & {
|
||||
"attention_backend",
|
||||
"sampling_backend",
|
||||
"disable_cuda_graph",
|
||||
"disable_piecewise_cuda_graph",
|
||||
"chunked_prefill_size",
|
||||
"schedule_policy",
|
||||
}:
|
||||
return "sglang"
|
||||
return detect_framework_from_text(json.dumps(server_args, sort_keys=True))
|
||||
|
||||
|
||||
def detect_framework_from_trace(trace: object) -> Optional[str]:
|
||||
text_samples: List[str] = []
|
||||
for event in extract_trace_events(trace)[:256]:
|
||||
text_samples.extend(
|
||||
[
|
||||
str(event.get("name", "")),
|
||||
str(event.get("cat", "")),
|
||||
str(event.get("pid", "")),
|
||||
]
|
||||
)
|
||||
trace_args = event.get("args")
|
||||
if isinstance(trace_args, dict):
|
||||
for key, value in list(trace_args.items())[:8]:
|
||||
text_samples.append(str(key))
|
||||
if isinstance(value, str):
|
||||
text_samples.append(value)
|
||||
return detect_framework_from_text(" ".join(text_samples))
|
||||
|
||||
|
||||
def detect_framework_from_path(path: Path) -> Optional[str]:
|
||||
hint = detect_framework_from_text(str(path))
|
||||
if hint:
|
||||
return hint
|
||||
server_args = load_server_args(path)
|
||||
hint = detect_framework_from_server_args(server_args)
|
||||
if hint:
|
||||
return hint
|
||||
if path.is_file():
|
||||
try:
|
||||
return detect_framework_from_trace(load_trace_json(path))
|
||||
except Exception:
|
||||
return None
|
||||
trace_files = discover_trace_files(path, recursive=True, limit=3)
|
||||
for trace_file in trace_files:
|
||||
try:
|
||||
hint = detect_framework_from_trace(load_trace_json(trace_file))
|
||||
except Exception:
|
||||
hint = None
|
||||
if hint:
|
||||
return hint
|
||||
return None
|
||||
|
||||
|
||||
def detect_framework_from_url(
|
||||
url: str, output_dir: Optional[str] = None
|
||||
) -> Optional[str]:
|
||||
hint = detect_framework_from_text(output_dir or "")
|
||||
if hint:
|
||||
return hint
|
||||
server_info = try_get_json(url.rstrip("/") + "/server_info")
|
||||
if isinstance(server_info, dict) and (
|
||||
"internal_states" in server_info
|
||||
or "tokenizer_path" in server_info
|
||||
or "prefill" in server_info
|
||||
or "decode" in server_info
|
||||
):
|
||||
return "sglang"
|
||||
models = try_get_json(url.rstrip("/") + "/v1/models")
|
||||
if isinstance(models, dict) and isinstance(models.get("data"), list):
|
||||
return "vllm"
|
||||
return None
|
||||
|
||||
|
||||
def resolve_framework(
|
||||
requested: object,
|
||||
*,
|
||||
input_path: Optional[Path] = None,
|
||||
url: Optional[str] = None,
|
||||
server_args: Optional[dict] = None,
|
||||
) -> str:
|
||||
explicit = canonicalize_framework(requested)
|
||||
if explicit != "auto":
|
||||
return explicit
|
||||
for hint in (
|
||||
detect_framework_from_server_args(server_args),
|
||||
detect_framework_from_path(input_path) if input_path else None,
|
||||
(
|
||||
detect_framework_from_url(url, str(input_path) if input_path else None)
|
||||
if url
|
||||
else None
|
||||
),
|
||||
):
|
||||
if hint:
|
||||
return hint
|
||||
return "sglang"
|
||||
|
||||
|
||||
def parse_stage(path: Path) -> str:
|
||||
name = path.name.lower()
|
||||
if "-extend" in name or "-prefill" in name:
|
||||
return "extend"
|
||||
if "-decode" in name:
|
||||
return "decode"
|
||||
return "all"
|
||||
|
||||
|
||||
def parse_tp_rank(path: Path) -> Optional[int]:
|
||||
for pattern in (
|
||||
r"(?:^|[_-])tp(\d+)(?:[_.-]|$)",
|
||||
r"TP-(\d+)",
|
||||
r"(?:^|[_-])rank(\d+)(?:[_.-]|$)",
|
||||
r"(?:^|[_-])worker(\d+)(?:[_.-]|$)",
|
||||
):
|
||||
match = re.search(pattern, path.name, re.IGNORECASE)
|
||||
if match:
|
||||
return int(match.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def file_looks_like_trace(path: Path) -> bool:
|
||||
name = path.name.lower()
|
||||
if name in TRACE_FILE_IGNORE_NAMES:
|
||||
return False
|
||||
if path.is_dir():
|
||||
return False
|
||||
if any(name.endswith(suffix) for suffix in (".trace.json", ".trace.json.gz")):
|
||||
return True
|
||||
if ".pt.trace.json" in name:
|
||||
return True
|
||||
if not any(name.endswith(suffix) for suffix in (".json", ".json.gz")):
|
||||
return False
|
||||
try:
|
||||
trace = load_trace_json(path)
|
||||
except Exception:
|
||||
return False
|
||||
if isinstance(trace, dict):
|
||||
return isinstance(trace.get("traceEvents"), list)
|
||||
if isinstance(trace, list):
|
||||
return bool(trace) and all(isinstance(item, dict) for item in trace[:8])
|
||||
return False
|
||||
|
||||
|
||||
def discover_trace_files(
|
||||
path: Path,
|
||||
*,
|
||||
recursive: bool,
|
||||
limit: Optional[int] = None,
|
||||
) -> List[Path]:
|
||||
if path.is_file():
|
||||
return [path] if file_looks_like_trace(path) else []
|
||||
|
||||
candidates: List[Path] = []
|
||||
seen: set[Path] = set()
|
||||
for pattern in TRACE_FILE_PATTERNS:
|
||||
iterator = path.rglob(pattern) if recursive else path.glob(pattern)
|
||||
for candidate in iterator:
|
||||
resolved = candidate.resolve()
|
||||
if resolved in seen:
|
||||
continue
|
||||
seen.add(resolved)
|
||||
candidates.append(resolved)
|
||||
candidates = [
|
||||
candidate
|
||||
for candidate in candidates
|
||||
if candidate.exists() and file_looks_like_trace(candidate)
|
||||
]
|
||||
candidates.sort(key=lambda item: item.stat().st_mtime)
|
||||
if limit is not None and limit >= 0:
|
||||
return candidates[-limit:] if limit else []
|
||||
return candidates
|
||||
|
||||
|
||||
def newest_trace_dir(path: Path) -> Path:
|
||||
if path.is_file():
|
||||
return path.parent
|
||||
direct = discover_trace_files(path, recursive=False)
|
||||
if direct:
|
||||
return path
|
||||
traces = discover_trace_files(path, recursive=True)
|
||||
trace_dirs = list({trace.parent for trace in traces})
|
||||
if not trace_dirs:
|
||||
raise FileNotFoundError(f"No trace files found under {path}")
|
||||
trace_dirs.sort(
|
||||
key=lambda item: max(
|
||||
trace.stat().st_mtime for trace in traces if trace.parent == item
|
||||
)
|
||||
)
|
||||
return trace_dirs[-1]
|
||||
|
||||
|
||||
def discover_trace_targets(
|
||||
path: Path, all_traces: bool
|
||||
) -> Tuple[List[Path], Optional[dict]]:
|
||||
if path.is_file():
|
||||
return [path], load_server_args(path)
|
||||
|
||||
trace_dir = newest_trace_dir(path)
|
||||
traces = discover_trace_files(trace_dir, recursive=False)
|
||||
if not traces:
|
||||
raise FileNotFoundError(f"No trace files found under {trace_dir}")
|
||||
|
||||
non_merged = [trace for trace in traces if not trace.name.startswith("merged-")]
|
||||
selected = non_merged or traces
|
||||
if not all_traces:
|
||||
ranks = sorted(
|
||||
{
|
||||
rank
|
||||
for rank in (parse_tp_rank(trace) for trace in selected)
|
||||
if rank is not None
|
||||
}
|
||||
)
|
||||
if ranks:
|
||||
rank = 0 if 0 in ranks else ranks[0]
|
||||
selected = [trace for trace in selected if parse_tp_rank(trace) == rank]
|
||||
grouped: Dict[str, List[Path]] = defaultdict(list)
|
||||
for trace in selected:
|
||||
grouped[parse_stage(trace)].append(trace)
|
||||
selected = [
|
||||
sorted(group, key=lambda item: item.stat().st_mtime)[-1]
|
||||
for group in grouped.values()
|
||||
]
|
||||
|
||||
selected.sort(key=lambda item: (STAGE_ORDER.get(parse_stage(item), 99), item.name))
|
||||
return selected, load_server_args(trace_dir)
|
||||
|
||||
|
||||
def post_json(
|
||||
url: str, payload: Optional[dict] = None, timeout: float = 60.0
|
||||
) -> Optional[dict]:
|
||||
req = request.Request(
|
||||
url=url,
|
||||
data=(None if payload is None else json.dumps(payload).encode("utf-8")),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with request.urlopen(req, timeout=timeout) as response:
|
||||
raw = response.read()
|
||||
return json.loads(raw.decode("utf-8")) if raw else None
|
||||
|
||||
|
||||
def send_probe_request(
|
||||
url: str,
|
||||
prompt: str,
|
||||
max_new_tokens: int,
|
||||
sampling_seed: int,
|
||||
framework: str,
|
||||
model: Optional[str] = None,
|
||||
) -> None:
|
||||
framework = canonicalize_framework(framework)
|
||||
if framework == "sglang":
|
||||
payload = {
|
||||
"text": prompt,
|
||||
"sampling_params": {
|
||||
"sampling_seed": sampling_seed,
|
||||
"temperature": 0.0,
|
||||
"max_new_tokens": max_new_tokens,
|
||||
},
|
||||
"stream": False,
|
||||
}
|
||||
post_json(url.rstrip("/") + "/generate", payload, timeout=300.0)
|
||||
return
|
||||
|
||||
resolved_model = model or discover_openai_model(url)
|
||||
chat_payload = {
|
||||
"model": resolved_model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.0,
|
||||
"max_tokens": max_new_tokens,
|
||||
"stream": False,
|
||||
}
|
||||
try:
|
||||
post_json(url.rstrip("/") + "/v1/chat/completions", chat_payload, timeout=300.0)
|
||||
return
|
||||
except Exception:
|
||||
completion_payload = {
|
||||
"model": resolved_model,
|
||||
"prompt": prompt,
|
||||
"temperature": 0.0,
|
||||
"max_tokens": max_new_tokens,
|
||||
"stream": False,
|
||||
}
|
||||
post_json(
|
||||
url.rstrip("/") + "/v1/completions",
|
||||
completion_payload,
|
||||
timeout=300.0,
|
||||
)
|
||||
|
||||
|
||||
def discover_openai_model(url: str) -> str:
|
||||
payload = try_get_json(url.rstrip("/") + "/v1/models", timeout=60.0)
|
||||
if not isinstance(payload, dict):
|
||||
raise RuntimeError(f"Could not read {url.rstrip('/')}/v1/models")
|
||||
data = payload.get("data")
|
||||
if not isinstance(data, list) or not data:
|
||||
raise RuntimeError(f"No models returned by {url.rstrip('/')}/v1/models")
|
||||
first = data[0]
|
||||
if isinstance(first, dict) and first.get("id"):
|
||||
return str(first["id"])
|
||||
raise RuntimeError(f"Malformed /v1/models payload from {url.rstrip('/')}")
|
||||
|
||||
|
||||
def ensure_remote_profiler_output_path(
|
||||
output_dir: Optional[str], framework: str
|
||||
) -> Path:
|
||||
if not output_dir:
|
||||
raise ValueError(
|
||||
f"{framework_display_name(framework)} live capture requires --output-dir "
|
||||
"to point at the server-side torch profiler trace path that is visible "
|
||||
"from this machine."
|
||||
)
|
||||
output_path = Path(output_dir).expanduser().resolve()
|
||||
if output_path.suffix in {".json", ".gz"}:
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
return output_path
|
||||
|
||||
|
||||
def wait_for_profiler_artifact(path: Path, timeout_s: float = 60.0) -> Path:
|
||||
deadline = time.time() + timeout_s
|
||||
while time.time() < deadline:
|
||||
if path.is_file() and file_looks_like_trace(path):
|
||||
return path
|
||||
if path.exists():
|
||||
trace_files = discover_trace_files(path, recursive=True)
|
||||
if trace_files:
|
||||
return newest_trace_dir(path)
|
||||
if path.is_dir():
|
||||
child_dirs = [item for item in path.iterdir() if item.is_dir()]
|
||||
if child_dirs:
|
||||
child_dirs.sort(key=lambda item: item.stat().st_mtime)
|
||||
newest_child = child_dirs[-1]
|
||||
child_traces = discover_trace_files(newest_child, recursive=True)
|
||||
if child_traces:
|
||||
return newest_child
|
||||
time.sleep(0.5)
|
||||
return path
|
||||
|
||||
|
||||
def start_remote_profiler(url: str, framework: str) -> None:
|
||||
try:
|
||||
post_json(url.rstrip("/") + "/start_profile", timeout=60.0)
|
||||
except Exception as exc:
|
||||
if framework == "vllm":
|
||||
raise RuntimeError(
|
||||
"vLLM live torch profiling requires the server to be launched with "
|
||||
'--profiler-config \'{"profiler":"torch","torch_profiler_dir":"..."}\' '
|
||||
"and to expose POST /start_profile."
|
||||
) from exc
|
||||
if framework == "trtllm":
|
||||
raise RuntimeError(
|
||||
"TensorRT-LLM live torch profiling requires "
|
||||
"a server build that exposes POST /start_profile plus the env vars "
|
||||
"TLLM_PROFILE_START_STOP=1 and TLLM_TORCH_PROFILE_TRACE=/shared/path."
|
||||
) from exc
|
||||
raise
|
||||
|
||||
|
||||
def stop_remote_profiler(url: str, framework: str) -> None:
|
||||
try:
|
||||
post_json(url.rstrip("/") + "/stop_profile", timeout=300.0)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(
|
||||
f"Failed to stop {framework_display_name(framework)} profiler via "
|
||||
f"{url.rstrip('/')}/stop_profile"
|
||||
) from exc
|
||||
|
||||
|
||||
def run_remote_profiler(
|
||||
url: str,
|
||||
output_dir: Optional[str],
|
||||
framework: str,
|
||||
probe_requests: int,
|
||||
probe_prompt: str,
|
||||
probe_max_new_tokens: Optional[int],
|
||||
probe_delay: float,
|
||||
num_steps: int,
|
||||
) -> Path:
|
||||
framework = canonicalize_framework(framework)
|
||||
output_path = ensure_remote_profiler_output_path(output_dir, framework)
|
||||
start_remote_profiler(url, framework)
|
||||
stop_error: Optional[BaseException] = None
|
||||
try:
|
||||
if probe_requests > 0:
|
||||
# Some profiler endpoints need a brief setup window after
|
||||
# POST /start_profile. A very short delay can send probes too early
|
||||
# and miss the profiling window entirely.
|
||||
time.sleep(max(5.0, probe_delay))
|
||||
effective_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
|
||||
model = (
|
||||
discover_openai_model(url) if framework in {"vllm", "trtllm"} else None
|
||||
)
|
||||
for request_idx in range(probe_requests):
|
||||
send_probe_request(
|
||||
url=url,
|
||||
prompt=probe_prompt,
|
||||
max_new_tokens=effective_max_new_tokens,
|
||||
sampling_seed=request_idx,
|
||||
framework=framework,
|
||||
model=model,
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
stop_remote_profiler(url, framework)
|
||||
except BaseException as exc: # pragma: no cover - preserve original failure
|
||||
stop_error = exc
|
||||
if stop_error is not None:
|
||||
raise stop_error
|
||||
return wait_for_profiler_artifact(output_path)
|
||||
|
||||
|
||||
def run_sglang_profiler(
|
||||
url: str,
|
||||
output_dir: Optional[str],
|
||||
num_steps: int,
|
||||
profile_by_stage: bool,
|
||||
merge_profiles: bool,
|
||||
profile_prefix: Optional[str],
|
||||
probe_requests: int,
|
||||
probe_prompt: str,
|
||||
probe_max_new_tokens: Optional[int],
|
||||
probe_delay: float,
|
||||
start_step: Optional[int] = None,
|
||||
) -> Path:
|
||||
if output_dir is None:
|
||||
output_dir = tempfile.mkdtemp(prefix="sglang-torch-profile-")
|
||||
output_root = Path(output_dir).resolve()
|
||||
output_root.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_root / str(time.time())
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
server_args = try_get_json(url.rstrip("/") + "/server_info", timeout=60.0)
|
||||
if server_args is not None:
|
||||
with open(output_path / "server_args.json", "w", encoding="utf-8") as handle:
|
||||
json.dump(server_args, handle)
|
||||
|
||||
payload = {
|
||||
"output_dir": str(output_path),
|
||||
"num_steps": str(num_steps),
|
||||
"activities": ["CPU", "GPU"],
|
||||
"profile_by_stage": profile_by_stage,
|
||||
"merge_profiles": merge_profiles,
|
||||
"profile_prefix": profile_prefix,
|
||||
}
|
||||
if start_step is not None:
|
||||
payload["start_step"] = str(start_step)
|
||||
|
||||
req = request.Request(
|
||||
url.rstrip("/") + "/start_profile",
|
||||
data=json.dumps(payload).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with request.urlopen(req, timeout=300.0):
|
||||
pass
|
||||
|
||||
if probe_requests > 0:
|
||||
time.sleep(max(0.0, probe_delay))
|
||||
effective_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
|
||||
for request_idx in range(probe_requests):
|
||||
send_probe_request(
|
||||
url=url,
|
||||
prompt=probe_prompt,
|
||||
max_new_tokens=effective_max_new_tokens,
|
||||
sampling_seed=request_idx,
|
||||
framework="sglang",
|
||||
)
|
||||
|
||||
return wait_for_profiler_artifact(output_path, timeout_s=180.0)
|
||||
|
||||
|
||||
def run_profiler(
|
||||
url: str,
|
||||
output_dir: Optional[str],
|
||||
num_steps: int,
|
||||
profile_by_stage: bool,
|
||||
merge_profiles: bool,
|
||||
profile_prefix: Optional[str],
|
||||
probe_requests: int,
|
||||
probe_prompt: str,
|
||||
probe_max_new_tokens: Optional[int],
|
||||
probe_delay: float,
|
||||
start_step: Optional[int] = None,
|
||||
framework: str = "auto",
|
||||
framework_hint_path: Optional[str] = None,
|
||||
) -> Path:
|
||||
resolved_framework = resolve_framework(
|
||||
framework,
|
||||
url=url,
|
||||
input_path=(
|
||||
Path(framework_hint_path).expanduser().resolve()
|
||||
if framework_hint_path
|
||||
else None
|
||||
),
|
||||
)
|
||||
if resolved_framework == "sglang":
|
||||
return run_sglang_profiler(
|
||||
url=url,
|
||||
output_dir=output_dir,
|
||||
num_steps=num_steps,
|
||||
profile_by_stage=profile_by_stage,
|
||||
merge_profiles=merge_profiles,
|
||||
profile_prefix=profile_prefix,
|
||||
probe_requests=probe_requests,
|
||||
probe_prompt=probe_prompt,
|
||||
probe_max_new_tokens=probe_max_new_tokens,
|
||||
probe_delay=probe_delay,
|
||||
start_step=start_step,
|
||||
)
|
||||
if start_step is not None:
|
||||
raise ValueError("--start-step is only supported for SGLang live capture.")
|
||||
if profile_by_stage:
|
||||
raise ValueError(
|
||||
"--profile-by-stage is only supported for SGLang live capture. "
|
||||
"Disable it when profiling vLLM or TensorRT-LLM."
|
||||
)
|
||||
if merge_profiles:
|
||||
raise ValueError(
|
||||
"--merge-profiles is only supported for SGLang live capture. "
|
||||
"Disable it when profiling vLLM or TensorRT-LLM."
|
||||
)
|
||||
if profile_prefix:
|
||||
print(
|
||||
f"Note: {framework_display_name(resolved_framework)} ignores "
|
||||
"--profile-prefix on the HTTP profiler control path.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return run_remote_profiler(
|
||||
url=url,
|
||||
output_dir=output_dir,
|
||||
framework=resolved_framework,
|
||||
probe_requests=probe_requests,
|
||||
probe_prompt=probe_prompt,
|
||||
probe_max_new_tokens=probe_max_new_tokens,
|
||||
probe_delay=probe_delay,
|
||||
num_steps=num_steps,
|
||||
)
|
||||
|
||||
|
||||
def select_heaviest_pid(
|
||||
events: Sequence[dict],
|
||||
event_filter: Callable[[dict], bool],
|
||||
pid_substring: Optional[str] = None,
|
||||
preferred_substrings: Iterable[str] = (),
|
||||
) -> Optional[str]:
|
||||
durations: Counter = Counter()
|
||||
for event in events:
|
||||
if not event_filter(event):
|
||||
continue
|
||||
pid = str(event.get("pid"))
|
||||
if pid_substring and pid_substring not in pid:
|
||||
continue
|
||||
durations[pid] += float(event["dur"])
|
||||
if not durations:
|
||||
return None
|
||||
|
||||
for substring in preferred_substrings:
|
||||
preferred = [pid for pid in durations if substring in pid]
|
||||
if preferred:
|
||||
return max(preferred, key=lambda pid: durations[pid])
|
||||
return max(durations, key=lambda pid: durations[pid])
|
||||
@@ -0,0 +1,259 @@
|
||||
"""Bundle one or more triage text reports into a single markdown document."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
FRAMEWORK_LABELS = {
|
||||
"sglang": "SGLang",
|
||||
"vllm": "vLLM",
|
||||
"trtllm": "TensorRT-LLM",
|
||||
}
|
||||
|
||||
FRAMEWORK_ORDER = {"sglang": 0, "vllm": 1, "trtllm": 2}
|
||||
|
||||
|
||||
def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Render multiple profiler triage text outputs into one markdown file. "
|
||||
"Input files are expected to be the existing analysis_*.txt outputs "
|
||||
"already emitted by analyze_llm_torch_profile.py."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--analysis-root",
|
||||
type=str,
|
||||
default=None,
|
||||
help=(
|
||||
"Root directory to scan recursively for analysis_*.txt files. "
|
||||
"Parent directory names are used as model section ids."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--analysis-file",
|
||||
action="append",
|
||||
default=[],
|
||||
help=(
|
||||
"Explicit analysis file entry. Use either PATH or LABEL=PATH. "
|
||||
"When LABEL is omitted, the parent directory name is used."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--title",
|
||||
type=str,
|
||||
default="Unified LLM Torch Profiler Triage Bundle",
|
||||
help="Top-level markdown title.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Write the bundled markdown to this file. Prints to stdout when omitted.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--include-toc",
|
||||
action=argparse.BooleanOptionalAction,
|
||||
default=True,
|
||||
help="Include a simple table of contents.",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
if not args.analysis_root and not args.analysis_file:
|
||||
parser.error("Provide at least one of --analysis-root or --analysis-file.")
|
||||
return args
|
||||
|
||||
|
||||
def framework_key_from_path(path: Path) -> str:
|
||||
lowered = path.name.lower()
|
||||
if "sglang" in lowered:
|
||||
return "sglang"
|
||||
if "vllm" in lowered:
|
||||
return "vllm"
|
||||
if "trtllm" in lowered or "tensorrt" in lowered:
|
||||
return "trtllm"
|
||||
return "other"
|
||||
|
||||
|
||||
def framework_label(framework_key: str) -> str:
|
||||
return FRAMEWORK_LABELS.get(framework_key, framework_key)
|
||||
|
||||
|
||||
def discover_analysis_files(root: Path) -> List[Tuple[str, Path]]:
|
||||
entries: List[Tuple[str, Path]] = []
|
||||
for path in sorted(root.rglob("analysis*.txt")):
|
||||
entries.append((path.parent.name, path))
|
||||
return entries
|
||||
|
||||
|
||||
def parse_explicit_entry(raw: str) -> Tuple[str, Path]:
|
||||
if "=" in raw:
|
||||
label, path_text = raw.split("=", 1)
|
||||
path = Path(path_text).expanduser().resolve()
|
||||
return label.strip(), path
|
||||
path = Path(raw).expanduser().resolve()
|
||||
return path.parent.name, path
|
||||
|
||||
|
||||
def slugify(text: str) -> str:
|
||||
chars = []
|
||||
last_dash = False
|
||||
for char in text.lower():
|
||||
if char.isalnum():
|
||||
chars.append(char)
|
||||
last_dash = False
|
||||
elif not last_dash:
|
||||
chars.append("-")
|
||||
last_dash = True
|
||||
return "".join(chars).strip("-")
|
||||
|
||||
|
||||
def extract_model_name(report_text: str) -> Optional[str]:
|
||||
for line in report_text.splitlines():
|
||||
if line.startswith("Model: "):
|
||||
return line.split("Model: ", 1)[1].strip()
|
||||
return None
|
||||
|
||||
|
||||
def choose_model_display_name(
|
||||
current: Optional[str],
|
||||
candidate: Optional[str],
|
||||
*,
|
||||
label: str,
|
||||
) -> str:
|
||||
if candidate and candidate != label:
|
||||
if not current or current == label:
|
||||
return candidate
|
||||
if len(candidate) > len(current):
|
||||
return candidate
|
||||
return current
|
||||
if current:
|
||||
return current
|
||||
return label
|
||||
|
||||
|
||||
def normalize_report_text(report_text: str) -> str:
|
||||
text = report_text.replace("\r\n", "\n").strip()
|
||||
if not text:
|
||||
return "_Empty analysis output._"
|
||||
heading_map = {
|
||||
"Triage View": "#### Triage View",
|
||||
"Kernel Table": "#### Kernel Table",
|
||||
"Overlap Opportunity Table": "#### Overlap Opportunity Table",
|
||||
"Fuse Opportunity Table": "#### Fuse Opportunity Table",
|
||||
}
|
||||
normalized_lines = []
|
||||
for line in text.splitlines():
|
||||
normalized_lines.append(heading_map.get(line, line))
|
||||
return "\n".join(normalized_lines)
|
||||
|
||||
|
||||
def build_bundle_markdown(
|
||||
*,
|
||||
title: str,
|
||||
labeled_paths: Sequence[Tuple[str, Path]],
|
||||
include_toc: bool,
|
||||
) -> str:
|
||||
grouped: Dict[str, List[Tuple[str, Path, str]]] = defaultdict(list)
|
||||
model_display: Dict[str, str] = {}
|
||||
|
||||
for label, path in labeled_paths:
|
||||
raw_text = path.read_text(encoding="utf-8")
|
||||
report_text = normalize_report_text(raw_text)
|
||||
model_name = extract_model_name(report_text)
|
||||
grouped[label].append((framework_key_from_path(path), path, report_text))
|
||||
model_display[label] = choose_model_display_name(
|
||||
model_display.get(label),
|
||||
model_name,
|
||||
label=label,
|
||||
)
|
||||
|
||||
ordered_labels = sorted(
|
||||
grouped,
|
||||
key=lambda item: (model_display[item].lower(), item.lower()),
|
||||
)
|
||||
|
||||
lines: List[str] = [f"# {title}", ""]
|
||||
lines.append(
|
||||
f"_Generated on {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')}_"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if include_toc:
|
||||
lines.append("## Contents")
|
||||
lines.append("")
|
||||
for label in ordered_labels:
|
||||
lines.append(
|
||||
f"- [{model_display[label]}](#{slugify(model_display[label])})"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
for label in ordered_labels:
|
||||
display_name = model_display[label]
|
||||
lines.append(f"## {display_name}")
|
||||
lines.append("")
|
||||
lines.append(f"Model id: `{label}`")
|
||||
lines.append("")
|
||||
|
||||
records = sorted(
|
||||
grouped[label],
|
||||
key=lambda item: (
|
||||
FRAMEWORK_ORDER.get(item[0], 99),
|
||||
item[1].name.lower(),
|
||||
),
|
||||
)
|
||||
|
||||
for framework_key, path, report_text in records:
|
||||
lines.append(f"### {framework_label(framework_key)}")
|
||||
lines.append("")
|
||||
lines.append(f"Source: `{path}`")
|
||||
lines.append("")
|
||||
lines.append(report_text)
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines).rstrip() + "\n"
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
args = parse_args(argv)
|
||||
|
||||
labeled_paths: List[Tuple[str, Path]] = []
|
||||
if args.analysis_root:
|
||||
labeled_paths.extend(
|
||||
discover_analysis_files(Path(args.analysis_root).expanduser().resolve())
|
||||
)
|
||||
for raw_entry in args.analysis_file:
|
||||
labeled_paths.append(parse_explicit_entry(raw_entry))
|
||||
|
||||
existing = []
|
||||
missing = []
|
||||
for label, path in labeled_paths:
|
||||
if path.is_file():
|
||||
existing.append((label, path))
|
||||
else:
|
||||
missing.append(str(path))
|
||||
if missing:
|
||||
raise SystemExit("Missing analysis files:\n" + "\n".join(missing))
|
||||
if not existing:
|
||||
raise SystemExit("No analysis files found.")
|
||||
|
||||
markdown = build_bundle_markdown(
|
||||
title=args.title,
|
||||
labeled_paths=existing,
|
||||
include_toc=args.include_toc,
|
||||
)
|
||||
|
||||
if args.output:
|
||||
output_path = Path(args.output).expanduser().resolve()
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(markdown, encoding="utf-8")
|
||||
else:
|
||||
print(markdown, end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+775
-310
File diff suppressed because it is too large
Load Diff
+644
-453
File diff suppressed because it is too large
Load Diff
@@ -1,3 +1,8 @@
|
||||
---
|
||||
name: sglang-bisect-ci-regression
|
||||
description: Investigate consistently failing SGLang CI tests by extracting the failure signature from scheduled or rerun workflows, bisecting the passing/failing commit window, checking runner or hardware specificity, and optionally reproducing on a remote GPU host.
|
||||
---
|
||||
|
||||
# SGLang Bisect CI Regression
|
||||
|
||||
Investigate a consistently failing CI test to find the root cause - whether it's a code regression from a specific PR, a hardware/runner-specific issue, or an environment change. Optionally reproduce the failure on a remote GPU server.
|
||||
|
||||
@@ -1,189 +0,0 @@
|
||||
---
|
||||
name: sglang-torch-profiler-analysis
|
||||
description: "Compact SGLang torch-profiler triage skill. Use when Codex should inspect an existing `trace.json(.gz)` or profile directory, trigger `sglang.profiler` against a live server, and return one compact report with kernel, overlap-opportunity, and fuse-pattern tables. Single-trace triage is enough for quick diagnosis; mapping+formal two-trace triage gives stronger overlap conclusions."
|
||||
---
|
||||
|
||||
# SGLang Torch Profiler Analysis
|
||||
|
||||
## Overview
|
||||
|
||||
Use this skill for SGLang `torch.profiler` analysis.
|
||||
|
||||
There is only one public workflow:
|
||||
|
||||
- `triage`
|
||||
|
||||
Use the unified entrypoint:
|
||||
|
||||
- [scripts/analyze_sglang_torch_profile.py](scripts/analyze_sglang_torch_profile.py)
|
||||
|
||||
`triage` always prints the same three tables:
|
||||
|
||||
- kernel table
|
||||
- overlap-opportunity table
|
||||
- fuse-pattern table
|
||||
|
||||
By default, all three tables only render rows at or above `1.0%` cumulative GPU-time share.
|
||||
Treat anything below that as noise unless the user explicitly asks for a lower cutoff.
|
||||
|
||||
The script-level fuse-pattern table should stay source-backed and deterministic.
|
||||
Do not build a fuzzy string-matching engine into the script for typo-tolerance.
|
||||
|
||||
If exact/source-backed matching is weak but the agent judges that a cluster of kernels
|
||||
still looks semantically close to a known pattern, add a short AI note after the table
|
||||
with one of these labels:
|
||||
|
||||
- `high`: very likely the same pattern family; naming drift or minor implementation reshaping is the main uncertainty
|
||||
- `medium`: several signals line up, but one important piece is still ambiguous
|
||||
- `low`: weak resemblance only; mention it only if it is still worth a human follow-up
|
||||
|
||||
## When To Use It
|
||||
|
||||
- inspect an SGLang torch profiler trace or profile directory
|
||||
- profile a live SGLang server and immediately analyze the output
|
||||
- summarize which kernel families dominate prefill or decode
|
||||
- map kernels back to Python code paths
|
||||
- judge whether a code path still has overlap headroom
|
||||
- check whether an already-known fusion or overlap path should have applied
|
||||
|
||||
## Diffusion Backend Gate
|
||||
|
||||
For diffusion benchmark or profiling work, only analyze traces produced by the native
|
||||
SGLang diffusion backend.
|
||||
|
||||
If the run that generated the trace logs any of:
|
||||
- `Falling back to diffusers backend`
|
||||
- `Using diffusers backend`
|
||||
- `Loaded diffusers pipeline`
|
||||
|
||||
stop the workflow instead of analyzing the trace. Treat it as a backend-selection issue,
|
||||
not as valid SGLang diffusion profiler evidence.
|
||||
|
||||
## Main Flows
|
||||
|
||||
### 1. Single-trace triage from an existing profile dir or trace
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_sglang_torch_profile.py \
|
||||
--input /path/to/profile_dir_or_trace.json.gz
|
||||
```
|
||||
|
||||
Use this when you want the fastest read on kernel share and likely fused-kernel pattern matches.
|
||||
The overlap table stays conservative in single-trace mode and will tell you when a mapping/formal pair is needed.
|
||||
|
||||
### 2. Single-trace triage from a running server
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_sglang_torch_profile.py \
|
||||
--url http://127.0.0.1:30000 \
|
||||
--num-steps 5 \
|
||||
--profile-by-stage
|
||||
```
|
||||
|
||||
### 3. Two-trace triage from existing profile dirs or traces
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_sglang_torch_profile.py triage \
|
||||
--mapping-input /path/to/graph_off_profile_dir \
|
||||
--formal-input /path/to/graph_on_profile_dir
|
||||
```
|
||||
|
||||
Use this when you need stronger overlap conclusions and cleaner kernel-to-source attribution.
|
||||
|
||||
### 4. Two-trace triage from running servers
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_sglang_torch_profile.py triage \
|
||||
--mapping-url http://127.0.0.1:31025 \
|
||||
--formal-url http://127.0.0.1:31026 \
|
||||
--num-steps 5 \
|
||||
--profile-by-stage
|
||||
```
|
||||
|
||||
## `profile_by_stage`
|
||||
|
||||
`profile_by_stage` is not only for PD disaggregation.
|
||||
|
||||
- On ordinary non-PD serving, it is still useful because prefill and decode usually have very different bottlenecks.
|
||||
- On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
|
||||
- PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary `profile_by_stage`.
|
||||
|
||||
## How To Choose The Triage Shape
|
||||
|
||||
### Single-trace triage
|
||||
|
||||
Use when you want the lowest-friction report:
|
||||
|
||||
- one trace is already available
|
||||
- you mainly want kernel share and fusion clues
|
||||
- you are comparing two runs side by side by running triage once per trace
|
||||
|
||||
This is the recommended default.
|
||||
|
||||
### Two-trace triage
|
||||
|
||||
Use when you need:
|
||||
|
||||
- a stronger answer about overlap headroom
|
||||
- graph-off source mapping plus graph-on final behavior
|
||||
- more trustworthy overlap recommendations in the middle table
|
||||
|
||||
1. mapping trace with `--disable-cuda-graph --disable-piecewise-cuda-graph`
|
||||
2. formal trace with the real serving optimizations enabled
|
||||
|
||||
Do not call the mapping pass a "fast profile". It exists to recover `kernel -> cpu_op -> python scope`.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Single-trace workflow
|
||||
|
||||
1. If the user only wants a quick diagnosis, one trace is enough.
|
||||
2. Prefer rank-local `TP-0` traces over merged traces.
|
||||
3. For a live server, this skill can call `sglang.profiler` and automatically send a small probe request.
|
||||
4. Prefer `--profile-by-stage` even on standard serving unless the user explicitly wants an all-stage mixed trace.
|
||||
|
||||
### Two-trace workflow
|
||||
|
||||
1. Produce a mapping trace first with graph disabled.
|
||||
2. Produce a formal trace second with graph enabled and the real serving flags kept on.
|
||||
3. Run `triage` for the compact three-table report.
|
||||
4. Read the results in this order:
|
||||
- kernel table
|
||||
- overlap-opportunity table
|
||||
- fuse-pattern table
|
||||
5. Before calling something a "new" optimization idea, compare the top rows against both [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md) and [references/overlap-catalog.md](references/overlap-catalog.md). Always check the `PR-backed / in-flight` sections too. Prefer reporting:
|
||||
- an existing fused or overlap path that should already apply here
|
||||
- an existing path that appears disabled, unsupported, or regressed in this trace
|
||||
- an upstream PR-backed pattern that already exists but is not merged into the checked-out tree
|
||||
- a truly new opportunity only when no catalog entry fits
|
||||
6. If no exact pattern fully matches but the trace still looks semantically close to a known family, add one flat `AI similarity judgment` note after the tables.
|
||||
Use `high`, `medium`, or `low` only.
|
||||
Base that note on the full pattern shape, not on one kernel name alone.
|
||||
Prefer semantic cues such as producer-consumer chain, source locations, CPU op names, TP context, and model-specific structure.
|
||||
Do not rewrite the script table itself to include these heuristic judgments.
|
||||
|
||||
## References
|
||||
|
||||
Load these only when needed:
|
||||
|
||||
- [references/source-map.md](references/source-map.md)
|
||||
- upstream SGLang profiler entrypoints and trace-writing source paths
|
||||
- [references/heuristics.md](references/heuristics.md)
|
||||
- overlap labels, dependency-risk interpretation, and limits
|
||||
- [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md)
|
||||
- mixed source-backed catalog of existing fuse and overlap patterns, including PR-backed / in-flight rows
|
||||
- [references/overlap-catalog.md](references/overlap-catalog.md)
|
||||
- overlap-only lookup table across LLM, VLM, diffusion, disaggregation, HiSparse, and speculative scheduling
|
||||
|
||||
## Output Contract
|
||||
|
||||
Return:
|
||||
|
||||
- trace path or generated profile path
|
||||
- model/server args when available
|
||||
- kernel table
|
||||
- overlap-opportunity table
|
||||
- fuse-pattern table
|
||||
- optional `AI similarity judgment` note with `high` / `medium` / `low` when exact matching is inconclusive
|
||||
- one short conclusion about what dominates the run
|
||||
- whether the overlap conclusion came from single-trace triage or mapping/formal two-trace triage
|
||||
-601
@@ -1,601 +0,0 @@
|
||||
"""Compact triage entrypoint for SGLang torch-profiler analysis."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
import triage_kernel_helpers as kernel_helpers
|
||||
import triage_overlap_helpers as overlap_helpers
|
||||
from profile_common import (
|
||||
discover_trace_targets,
|
||||
load_server_args,
|
||||
load_trace_json,
|
||||
parse_stage,
|
||||
run_profiler,
|
||||
)
|
||||
|
||||
MIN_RENDER_SHARE_PCT = 1.0
|
||||
|
||||
|
||||
def build_triage_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="analyze_sglang_torch_profile.py",
|
||||
description=(
|
||||
"Compact SGLang torch-profiler triage entrypoint. "
|
||||
"This prints three tables: kernel mapping, overlap opportunities, "
|
||||
"and fuse opportunities. "
|
||||
"Use either a single trace/profile input or a mapping+formal two-trace pair."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Single trace file or profile directory to triage.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--url",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Running SGLang server URL for single-trace triage.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Trace output dir when using --url.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--profile-prefix",
|
||||
type=str,
|
||||
default="triage-trace",
|
||||
help="Profile prefix when generating a single trace from --url.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-input",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Graph-off mapping trace file or directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-url",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Running graph-off SGLang server URL for the mapping trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-input",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Formal graph-on trace file or directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-url",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Running graph-on SGLang server URL for the formal trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-output-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Trace output dir when using --mapping-url.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-output-dir",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Trace output dir when using --formal-url.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-profile-prefix",
|
||||
type=str,
|
||||
default="mapping-trace",
|
||||
help="Profile prefix for the mapping trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--formal-profile-prefix",
|
||||
type=str,
|
||||
default="formal-trace",
|
||||
help="Profile prefix for the formal trace.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-steps",
|
||||
type=int,
|
||||
default=5,
|
||||
help="Profiler steps when generating traces from URLs.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--profile-by-stage", action=argparse.BooleanOptionalAction, default=True
|
||||
)
|
||||
parser.add_argument(
|
||||
"--merge-profiles", action=argparse.BooleanOptionalAction, default=False
|
||||
)
|
||||
parser.add_argument("--probe-requests", type=int, default=1)
|
||||
parser.add_argument(
|
||||
"--probe-prompt",
|
||||
type=str,
|
||||
default=(
|
||||
"Repeat the word profiler many times with spaces so the server performs several decode steps. "
|
||||
"Do not add explanations."
|
||||
),
|
||||
)
|
||||
parser.add_argument("--probe-max-new-tokens", type=int, default=None)
|
||||
parser.add_argument("--probe-delay", type=float, default=0.5)
|
||||
parser.add_argument(
|
||||
"--start-step",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Pass through to sglang.profiler when generating traces from URLs.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pid-substring",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Restrict overlap analysis to PIDs containing this substring.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--kernel-table-limit",
|
||||
type=int,
|
||||
default=0,
|
||||
help="How many kernel rows to print per stage. Use 0 for all kernels.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--overlap-table-limit",
|
||||
type=int,
|
||||
default=0,
|
||||
help="How many overlap rows to print per stage. Use 0 for all kernels.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def parse_triage_args(argv: Sequence[str]) -> argparse.Namespace:
|
||||
parser = build_triage_parser()
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
single_trace_mode = bool(args.input) or bool(args.url)
|
||||
dual_trace_mode = any(
|
||||
[
|
||||
args.mapping_input,
|
||||
args.mapping_url,
|
||||
args.formal_input,
|
||||
args.formal_url,
|
||||
]
|
||||
)
|
||||
|
||||
if single_trace_mode and dual_trace_mode:
|
||||
parser.error(
|
||||
"Use either single-trace mode (--input/--url) or two-trace mode "
|
||||
"(--mapping-* plus --formal-*), not both."
|
||||
)
|
||||
|
||||
if single_trace_mode:
|
||||
if bool(args.input) == bool(args.url):
|
||||
parser.error("Provide exactly one of --input or --url.")
|
||||
return args
|
||||
|
||||
if bool(args.mapping_input) == bool(args.mapping_url):
|
||||
parser.error("Provide exactly one of --mapping-input or --mapping-url.")
|
||||
if bool(args.formal_input) == bool(args.formal_url):
|
||||
parser.error("Provide exactly one of --formal-input or --formal-url.")
|
||||
return args
|
||||
|
||||
|
||||
def resolve_profile_targets(
|
||||
*,
|
||||
label: str,
|
||||
input_path: Optional[str],
|
||||
url: Optional[str],
|
||||
output_dir: Optional[str],
|
||||
profile_prefix: Optional[str],
|
||||
args: argparse.Namespace,
|
||||
) -> Tuple[List[Path], Optional[dict]]:
|
||||
if bool(input_path) == bool(url):
|
||||
raise ValueError(f"{label} trace requires exactly one of input path or URL.")
|
||||
|
||||
if url:
|
||||
target_dir = run_profiler(
|
||||
url=url,
|
||||
output_dir=output_dir,
|
||||
num_steps=args.num_steps,
|
||||
profile_by_stage=args.profile_by_stage,
|
||||
merge_profiles=args.merge_profiles,
|
||||
profile_prefix=profile_prefix,
|
||||
probe_requests=max(0, args.probe_requests),
|
||||
probe_prompt=args.probe_prompt,
|
||||
probe_max_new_tokens=args.probe_max_new_tokens,
|
||||
probe_delay=args.probe_delay,
|
||||
start_step=args.start_step,
|
||||
)
|
||||
traces, server_args = discover_trace_targets(target_dir, all_traces=False)
|
||||
return traces, server_args
|
||||
|
||||
resolved = Path(input_path).resolve()
|
||||
traces, server_args = discover_trace_targets(resolved, all_traces=False)
|
||||
if server_args is None:
|
||||
server_args = load_server_args(resolved)
|
||||
return traces, server_args
|
||||
|
||||
|
||||
def build_mapping_kernel_map(trace_paths: Sequence[Path]) -> dict:
|
||||
stage_site_stats = defaultdict(
|
||||
lambda: defaultdict(lambda: defaultdict(kernel_helpers.MappingSiteAggregate))
|
||||
)
|
||||
stage_kernel_categories: Dict[str, Dict[str, str]] = defaultdict(dict)
|
||||
global_site_stats = defaultdict(
|
||||
lambda: defaultdict(kernel_helpers.MappingSiteAggregate)
|
||||
)
|
||||
global_kernel_categories: Dict[str, str] = {}
|
||||
|
||||
for trace_path in trace_paths:
|
||||
trace = load_trace_json(trace_path)
|
||||
kernels, cpu_ops, python_frames, launch_events, _, _ = (
|
||||
kernel_helpers.extract_trace_data(trace)
|
||||
)
|
||||
cpu_ops_by_external_id = kernel_helpers.build_cpu_op_index(cpu_ops)
|
||||
launches_by_correlation = kernel_helpers.build_launch_index(launch_events)
|
||||
local_site_stats = kernel_helpers.aggregate_kernel_sites(
|
||||
kernels,
|
||||
cpu_ops_by_external_id,
|
||||
python_frames,
|
||||
launches_by_correlation=launches_by_correlation,
|
||||
)
|
||||
stage = parse_stage(trace_path)
|
||||
kernel_categories = {
|
||||
kernel.canonical_name: kernel.category for kernel in kernels
|
||||
}
|
||||
kernel_helpers.merge_site_stats(stage_site_stats[stage], local_site_stats)
|
||||
kernel_helpers.merge_site_stats(global_site_stats, local_site_stats)
|
||||
stage_kernel_categories[stage].update(kernel_categories)
|
||||
global_kernel_categories.update(kernel_categories)
|
||||
|
||||
stage_payloads = {
|
||||
stage: kernel_helpers.build_stage_payload(
|
||||
dict(site_stats), stage_kernel_categories.get(stage, {})
|
||||
)
|
||||
for stage, site_stats in stage_site_stats.items()
|
||||
}
|
||||
global_payload = kernel_helpers.build_stage_payload(
|
||||
dict(global_site_stats), global_kernel_categories
|
||||
)
|
||||
return {"stages": stage_payloads, "global": global_payload}
|
||||
|
||||
|
||||
def stage_index(stage: str) -> int:
|
||||
return {"extend": 0, "prefill": 0, "decode": 1, "all": 2}.get(stage, 99)
|
||||
|
||||
|
||||
def stage_display(stage: str) -> str:
|
||||
return kernel_helpers.stage_label(stage)
|
||||
|
||||
|
||||
def pick_trace_for_stage(stage_to_trace: Dict[str, Path], stage: str) -> Optional[Path]:
|
||||
if stage in stage_to_trace:
|
||||
return stage_to_trace[stage]
|
||||
if "all" in stage_to_trace:
|
||||
return stage_to_trace["all"]
|
||||
if len(stage_to_trace) == 1:
|
||||
return next(iter(stage_to_trace.values()))
|
||||
return None
|
||||
|
||||
|
||||
def build_stage_trace_map(trace_paths: Sequence[Path]) -> Dict[str, Path]:
|
||||
stage_map: Dict[str, Path] = {}
|
||||
for trace_path in sorted(
|
||||
trace_paths, key=lambda item: (stage_index(parse_stage(item)), item.name)
|
||||
):
|
||||
stage_map[parse_stage(trace_path)] = trace_path
|
||||
return stage_map
|
||||
|
||||
|
||||
def render_kernel_table(rows: Sequence[dict]) -> List[str]:
|
||||
lines = [
|
||||
"| Stage | Kernel | Category | GPU time | Share | Launches | Python location (site share) | CPU op |",
|
||||
"| --- | --- | --- | ---: | ---: | ---: | --- | --- |",
|
||||
]
|
||||
for row in rows:
|
||||
lines.append(
|
||||
"| {stage} | {kernel} | {category} | {gpu_time} | {share:.1f}% | {launches} | {location} | {cpu_op} |".format(
|
||||
stage=kernel_helpers.escape_md_cell(stage_display(row["stage"])),
|
||||
kernel=kernel_helpers.escape_md_cell(row["kernel"]),
|
||||
category=kernel_helpers.escape_md_cell(row["category"]),
|
||||
gpu_time=kernel_helpers.format_ms(row["total_us"]),
|
||||
share=row["share_pct"],
|
||||
launches=row["launches"],
|
||||
location=kernel_helpers.escape_md_cell(row["location"]),
|
||||
cpu_op=kernel_helpers.escape_md_cell(row["cpu_op"]),
|
||||
)
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def render_overlap_table(rows: Sequence[dict]) -> List[str]:
|
||||
lines = [
|
||||
"| Stage | Priority | Verdict | Kernel | Python scope | Formal signal | Dep risk | Recommendation |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- | --- |",
|
||||
]
|
||||
if not rows:
|
||||
lines.append(
|
||||
"| - | - | - | No actionable overlap rows. Use mapping/formal two-trace triage for stronger overlap conclusions. | - | - | - | - |"
|
||||
)
|
||||
return lines
|
||||
for row in rows:
|
||||
formal_signal = (
|
||||
f"{row['total_us']:.1f} us, share {row['share_pct']:.1f}%, "
|
||||
f"excl {row['exclusive_ratio'] * 100:.1f}% / hid {row['hidden_ratio'] * 100:.1f}%"
|
||||
)
|
||||
lines.append(
|
||||
"| "
|
||||
+ " | ".join(
|
||||
[
|
||||
kernel_helpers.escape_md_cell(stage_display(row["stage"])),
|
||||
row["priority"],
|
||||
row["verdict"],
|
||||
kernel_helpers.escape_md_cell(row["kernel"]),
|
||||
kernel_helpers.escape_md_cell(row["python_scope"]),
|
||||
kernel_helpers.escape_md_cell(formal_signal),
|
||||
overlap_helpers.dependency_risk_label(row["dependency_signal"]),
|
||||
row["recommendation"],
|
||||
]
|
||||
)
|
||||
+ " |"
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def render_fuse_table(rows: Sequence[dict]) -> List[str]:
|
||||
lines = [
|
||||
"| Stage | Pattern | Confidence | Related GPU time | Share | Evidence kernels | Current kernel Python location | Candidate fused Python path | Rationale |",
|
||||
"| --- | --- | --- | ---: | ---: | --- | --- | --- | --- |",
|
||||
]
|
||||
if not rows:
|
||||
lines.append(
|
||||
"| - | No medium-confidence source-backed fusion opportunity matched this trace. | - | - | - | - | - | - | - |"
|
||||
)
|
||||
return lines
|
||||
for row in rows:
|
||||
lines.append(
|
||||
"| {stage} | {pattern} | {confidence} | {gpu_time} | {share:.1f}% | {evidence} | {current_locations} | {candidate_path} | {rationale} |".format(
|
||||
stage=kernel_helpers.escape_md_cell(stage_display(row["stage"])),
|
||||
pattern=kernel_helpers.escape_md_cell(row["pattern"]),
|
||||
confidence=kernel_helpers.escape_md_cell(row["confidence"]),
|
||||
gpu_time=kernel_helpers.format_ms(row["related_us"]),
|
||||
share=row["share_pct"],
|
||||
evidence=kernel_helpers.escape_md_cell(row["evidence"]),
|
||||
current_locations=kernel_helpers.escape_md_cell(
|
||||
row["current_locations"]
|
||||
),
|
||||
candidate_path=kernel_helpers.escape_md_cell(row["candidate_path"]),
|
||||
rationale=kernel_helpers.escape_md_cell(row["rationale"]),
|
||||
)
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
def run_triage(args: argparse.Namespace) -> int:
|
||||
single_trace_mode = bool(args.input) or bool(args.url)
|
||||
if single_trace_mode:
|
||||
formal_traces, formal_server_args = resolve_profile_targets(
|
||||
label="input",
|
||||
input_path=args.input,
|
||||
url=args.url,
|
||||
output_dir=args.output_dir,
|
||||
profile_prefix=args.profile_prefix,
|
||||
args=args,
|
||||
)
|
||||
mapping_traces = formal_traces
|
||||
mapping_server_args = formal_server_args
|
||||
else:
|
||||
mapping_traces, mapping_server_args = resolve_profile_targets(
|
||||
label="mapping",
|
||||
input_path=args.mapping_input,
|
||||
url=args.mapping_url,
|
||||
output_dir=args.mapping_output_dir,
|
||||
profile_prefix=args.mapping_profile_prefix,
|
||||
args=args,
|
||||
)
|
||||
formal_traces, formal_server_args = resolve_profile_targets(
|
||||
label="formal",
|
||||
input_path=args.formal_input,
|
||||
url=args.formal_url,
|
||||
output_dir=args.formal_output_dir,
|
||||
profile_prefix=args.formal_profile_prefix,
|
||||
args=args,
|
||||
)
|
||||
|
||||
mapping_kernel_map = build_mapping_kernel_map(mapping_traces)
|
||||
|
||||
kernel_rows_rendered: List[dict] = []
|
||||
fuse_rows_rendered: List[dict] = []
|
||||
|
||||
for formal_trace in formal_traces:
|
||||
trace = load_trace_json(formal_trace)
|
||||
kernels, _, _, _, _, _ = kernel_helpers.extract_trace_data(trace)
|
||||
if not kernels:
|
||||
continue
|
||||
stage = parse_stage(formal_trace)
|
||||
total_us = sum(kernel.dur for kernel in kernels)
|
||||
kernel_stats = kernel_helpers.aggregate(
|
||||
kernels, key_fn=lambda item: item.canonical_name
|
||||
)
|
||||
kernel_categories = {
|
||||
kernel.canonical_name: kernel.category for kernel in kernels
|
||||
}
|
||||
full_kernel_rows = kernel_helpers.build_kernel_rows(
|
||||
stage=stage,
|
||||
kernel_stats=kernel_stats,
|
||||
kernel_categories=kernel_categories,
|
||||
local_stage_payload=mapping_kernel_map.get("stages", {}).get(
|
||||
stage, {"kernels": {}}
|
||||
),
|
||||
external_kernel_map=mapping_kernel_map,
|
||||
)
|
||||
visible_kernel_rows = kernel_helpers.limit_kernel_rows(
|
||||
full_kernel_rows, args.kernel_table_limit
|
||||
)
|
||||
for row in visible_kernel_rows:
|
||||
share_pct = kernel_helpers.pct(row.total_us, total_us)
|
||||
if share_pct < MIN_RENDER_SHARE_PCT:
|
||||
continue
|
||||
kernel_rows_rendered.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"kernel": row.name,
|
||||
"category": row.category,
|
||||
"total_us": row.total_us,
|
||||
"share_pct": share_pct,
|
||||
"launches": row.aggregate.count,
|
||||
"location": row.location,
|
||||
"cpu_op": row.cpu_op,
|
||||
}
|
||||
)
|
||||
for item in kernel_helpers.detect_fusion_opportunities(
|
||||
stage=stage,
|
||||
kernel_rows=full_kernel_rows,
|
||||
total_us=total_us,
|
||||
server_args=formal_server_args or mapping_server_args,
|
||||
):
|
||||
share_pct = kernel_helpers.pct(item.related_us, total_us)
|
||||
if share_pct < MIN_RENDER_SHARE_PCT:
|
||||
continue
|
||||
fuse_rows_rendered.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"pattern": item.pattern,
|
||||
"confidence": item.confidence,
|
||||
"related_us": item.related_us,
|
||||
"share_pct": share_pct,
|
||||
"evidence": item.evidence,
|
||||
"current_locations": item.current_locations,
|
||||
"candidate_path": item.candidate_path,
|
||||
"rationale": item.rationale,
|
||||
}
|
||||
)
|
||||
|
||||
mapping_stage_map = build_stage_trace_map(mapping_traces)
|
||||
formal_stage_map = build_stage_trace_map(formal_traces)
|
||||
overlap_rows_rendered: List[dict] = []
|
||||
if not single_trace_mode:
|
||||
for stage in sorted(formal_stage_map, key=stage_index):
|
||||
formal_trace = formal_stage_map[stage]
|
||||
mapping_trace = pick_trace_for_stage(mapping_stage_map, stage)
|
||||
if mapping_trace is None:
|
||||
continue
|
||||
mapping_trace_json = load_trace_json(mapping_trace)
|
||||
mapping_events, mapping_pid = overlap_helpers.extract_kernel_events(
|
||||
mapping_trace_json, args.pid_substring
|
||||
)
|
||||
if not mapping_events:
|
||||
continue
|
||||
formal_trace_json = load_trace_json(formal_trace)
|
||||
formal_events, formal_pid = overlap_helpers.extract_kernel_events(
|
||||
formal_trace_json, args.pid_substring
|
||||
)
|
||||
if not formal_events:
|
||||
continue
|
||||
mapping_bundle = overlap_helpers.TraceBundle(
|
||||
label=f"mapping-{stage}",
|
||||
trace_path=mapping_trace,
|
||||
server_args=mapping_server_args,
|
||||
raw_events=mapping_trace_json.get(
|
||||
"traceEvents",
|
||||
mapping_trace_json if isinstance(mapping_trace_json, list) else [],
|
||||
),
|
||||
events=mapping_events,
|
||||
pid=mapping_pid,
|
||||
)
|
||||
formal_bundle = overlap_helpers.TraceBundle(
|
||||
label=f"formal-{stage}",
|
||||
trace_path=formal_trace,
|
||||
server_args=formal_server_args,
|
||||
raw_events=formal_trace_json.get(
|
||||
"traceEvents",
|
||||
formal_trace_json if isinstance(formal_trace_json, list) else [],
|
||||
),
|
||||
events=formal_events,
|
||||
pid=formal_pid,
|
||||
)
|
||||
formal_bundle.overlap_stats = overlap_helpers.analyze_overlap(
|
||||
formal_bundle.events
|
||||
)
|
||||
aggregates = overlap_helpers.aggregate_events(formal_bundle.events)
|
||||
source_map = overlap_helpers.build_kernel_source_map(mapping_bundle)
|
||||
stage_rows = overlap_helpers.build_action_rows(
|
||||
aggregates,
|
||||
source_map,
|
||||
formal_bundle.events,
|
||||
formal_bundle.overlap_stats["total_busy_us"],
|
||||
table_limit=max(0, args.overlap_table_limit),
|
||||
)
|
||||
for row in stage_rows:
|
||||
if row.share_pct < MIN_RENDER_SHARE_PCT:
|
||||
continue
|
||||
overlap_rows_rendered.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"priority": row.priority,
|
||||
"verdict": row.verdict,
|
||||
"kernel": row.kernel,
|
||||
"python_scope": row.python_scope,
|
||||
"total_us": row.total_us,
|
||||
"share_pct": row.share_pct,
|
||||
"exclusive_ratio": row.exclusive_ratio,
|
||||
"hidden_ratio": row.hidden_ratio,
|
||||
"dependency_signal": row.dependency_signal,
|
||||
"recommendation": row.recommendation,
|
||||
}
|
||||
)
|
||||
|
||||
lines: List[str] = []
|
||||
lines.append("Triage View")
|
||||
if single_trace_mode:
|
||||
lines.append(f"Input traces: {', '.join(str(path) for path in formal_traces)}")
|
||||
else:
|
||||
lines.append(
|
||||
f"Mapping traces: {', '.join(str(path) for path in mapping_traces)}"
|
||||
)
|
||||
lines.append(f"Formal traces: {', '.join(str(path) for path in formal_traces)}")
|
||||
if formal_server_args or mapping_server_args:
|
||||
server_args = formal_server_args or mapping_server_args
|
||||
model = server_args.get("model_path") or server_args.get("model")
|
||||
if model:
|
||||
lines.append(f"Model: {model}")
|
||||
lines.append("")
|
||||
lines.append("Kernel Table")
|
||||
lines.extend(render_kernel_table(kernel_rows_rendered))
|
||||
lines.append("")
|
||||
lines.append("Overlap Opportunity Table")
|
||||
lines.extend(render_overlap_table(overlap_rows_rendered))
|
||||
lines.append("")
|
||||
lines.append("Fuse Opportunity Table")
|
||||
lines.extend(render_fuse_table(fuse_rows_rendered))
|
||||
print("\n".join(lines).rstrip())
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
argv = list(argv or sys.argv[1:])
|
||||
triage_parser = build_triage_parser()
|
||||
|
||||
if not argv or argv[0] in {"-h", "--help"}:
|
||||
triage_parser.print_help()
|
||||
return 0
|
||||
|
||||
if argv[0] == "triage":
|
||||
argv = argv[1:]
|
||||
elif not argv[0].startswith("-"):
|
||||
triage_parser.error(
|
||||
"This skill now exposes only the compact triage workflow. "
|
||||
"Use single-trace mode (--input/--url) or mapping+formal two-trace mode."
|
||||
)
|
||||
return 2
|
||||
|
||||
return run_triage(parse_triage_args(argv))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -1,333 +0,0 @@
|
||||
"""Shared helpers for SGLang torch-profiler skill scripts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Callable, Dict, Iterable, List, Optional, Sequence, Tuple
|
||||
from urllib import request
|
||||
|
||||
STAGE_ORDER = {"extend": 0, "prefill": 0, "decode": 1, "all": 2}
|
||||
TRACE_METADATA_NAMES = {
|
||||
"process_name",
|
||||
"thread_name",
|
||||
"process_sort_index",
|
||||
"thread_sort_index",
|
||||
}
|
||||
NON_KERNEL_TRACE_CATEGORIES = ("python_function", "cpu_op", "trace")
|
||||
PYTHON_SCOPE_NAME_PREFIXES = ("python/", "nn.module:")
|
||||
|
||||
|
||||
def normalize_text(value: object) -> str:
|
||||
return re.sub(r"\s+", " ", str(value)).strip()
|
||||
|
||||
|
||||
def normalize_repo_relative_path(path: object) -> str:
|
||||
text = normalize_text(path).replace("\\", "/")
|
||||
for marker in ("python/sglang/", "sgl_kernel/"):
|
||||
idx = text.find(marker)
|
||||
if idx != -1:
|
||||
return text[idx:].lstrip("/")
|
||||
idx = text.find("sglang/")
|
||||
if idx != -1:
|
||||
return ("python/" + text[idx:]).lstrip("/")
|
||||
return text.lstrip("/")
|
||||
|
||||
|
||||
def contains_any_keyword(text: str, keywords: Iterable[str]) -> bool:
|
||||
return any(keyword in text for keyword in keywords)
|
||||
|
||||
|
||||
def coerce_optional_int(value: object) -> Optional[int]:
|
||||
if value in (None, "", "None"):
|
||||
return None
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
return int(value) if value.is_integer() else None
|
||||
try:
|
||||
return int(str(value))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def extract_trace_events(trace: object) -> Sequence[dict]:
|
||||
if isinstance(trace, dict):
|
||||
events = trace.get("traceEvents", [])
|
||||
return events if isinstance(events, list) else []
|
||||
if isinstance(trace, list):
|
||||
return trace
|
||||
return []
|
||||
|
||||
|
||||
def is_trace_metadata_name(name: object) -> bool:
|
||||
return str(name) in TRACE_METADATA_NAMES
|
||||
|
||||
|
||||
def is_complete_duration_event(event: dict) -> bool:
|
||||
if event.get("ph") != "X":
|
||||
return False
|
||||
dur = event.get("dur")
|
||||
ts = event.get("ts")
|
||||
if dur is None or ts is None:
|
||||
return False
|
||||
try:
|
||||
return float(dur) > 0
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def is_annotation_event(name: object, category: object) -> bool:
|
||||
lowered_name = normalize_text(name).lower()
|
||||
lowered_category = normalize_text(category).lower()
|
||||
return "annotation" in lowered_category or lowered_name.startswith("## call ")
|
||||
|
||||
|
||||
def is_non_kernel_trace_category(category: object) -> bool:
|
||||
lowered_category = normalize_text(category).lower()
|
||||
return any(token in lowered_category for token in NON_KERNEL_TRACE_CATEGORIES)
|
||||
|
||||
|
||||
def looks_like_python_scope_name(name: object) -> bool:
|
||||
lowered_name = normalize_text(name).lower()
|
||||
return ".py(" in lowered_name or lowered_name.startswith(PYTHON_SCOPE_NAME_PREFIXES)
|
||||
|
||||
|
||||
def has_stream_marker(args: Optional[dict]) -> bool:
|
||||
trace_args = args or {}
|
||||
return "stream" in trace_args or "cuda_stream" in trace_args
|
||||
|
||||
|
||||
def load_trace_json(path: Path) -> dict:
|
||||
if path.suffix == ".gz":
|
||||
with gzip.open(path, "rt", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
|
||||
|
||||
def load_server_args(path: Path) -> Optional[dict]:
|
||||
resolved = path.resolve()
|
||||
candidate_dirs: List[Path] = []
|
||||
if resolved.is_file():
|
||||
candidate_dirs.extend([resolved.parent, resolved.parent.parent])
|
||||
else:
|
||||
candidate_dirs.extend([resolved, resolved.parent])
|
||||
|
||||
seen: set[Path] = set()
|
||||
for candidate_dir in candidate_dirs:
|
||||
if candidate_dir in seen:
|
||||
continue
|
||||
seen.add(candidate_dir)
|
||||
candidate = candidate_dir / "server_args.json"
|
||||
if candidate.exists():
|
||||
with open(candidate, "r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
return None
|
||||
|
||||
|
||||
def parse_stage(path: Path) -> str:
|
||||
name = path.name.lower()
|
||||
if "-extend" in name or "-prefill" in name:
|
||||
return "extend"
|
||||
if "-decode" in name:
|
||||
return "decode"
|
||||
return "all"
|
||||
|
||||
|
||||
def parse_tp_rank(path: Path) -> Optional[int]:
|
||||
match = re.search(r"TP-(\d+)", path.name)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def newest_trace_dir(path: Path) -> Path:
|
||||
if path.is_file():
|
||||
return path.parent
|
||||
direct = list(path.glob("*.trace.json")) + list(path.glob("*.trace.json.gz"))
|
||||
if direct:
|
||||
return path
|
||||
child_candidates = [item for item in path.rglob("*") if item.is_dir()]
|
||||
trace_dirs = [
|
||||
candidate
|
||||
for candidate in child_candidates
|
||||
if list(candidate.glob("*.trace.json"))
|
||||
or list(candidate.glob("*.trace.json.gz"))
|
||||
]
|
||||
if not trace_dirs:
|
||||
raise FileNotFoundError(f"No trace files found under {path}")
|
||||
trace_dirs.sort(key=lambda item: item.stat().st_mtime)
|
||||
return trace_dirs[-1]
|
||||
|
||||
|
||||
def discover_trace_targets(
|
||||
path: Path, all_traces: bool
|
||||
) -> Tuple[List[Path], Optional[dict]]:
|
||||
if path.is_file():
|
||||
return [path], load_server_args(path)
|
||||
|
||||
trace_dir = newest_trace_dir(path)
|
||||
traces = sorted(
|
||||
list(trace_dir.glob("*.trace.json")) + list(trace_dir.glob("*.trace.json.gz")),
|
||||
key=lambda item: item.stat().st_mtime,
|
||||
)
|
||||
if not traces:
|
||||
raise FileNotFoundError(f"No trace files found under {trace_dir}")
|
||||
|
||||
non_merged = [trace for trace in traces if not trace.name.startswith("merged-")]
|
||||
selected = non_merged or traces
|
||||
if not all_traces:
|
||||
ranks = sorted(
|
||||
{
|
||||
rank
|
||||
for rank in (parse_tp_rank(trace) for trace in selected)
|
||||
if rank is not None
|
||||
}
|
||||
)
|
||||
if ranks:
|
||||
rank = 0 if 0 in ranks else ranks[0]
|
||||
selected = [trace for trace in selected if parse_tp_rank(trace) == rank]
|
||||
grouped: Dict[str, List[Path]] = defaultdict(list)
|
||||
for trace in selected:
|
||||
grouped[parse_stage(trace)].append(trace)
|
||||
selected = [
|
||||
sorted(group, key=lambda item: item.stat().st_mtime)[-1]
|
||||
for group in grouped.values()
|
||||
]
|
||||
|
||||
selected.sort(key=lambda item: (STAGE_ORDER.get(parse_stage(item), 99), item.name))
|
||||
return selected, load_server_args(trace_dir)
|
||||
|
||||
|
||||
def post_json(url: str, payload: dict, timeout: float = 60.0) -> Optional[dict]:
|
||||
req = request.Request(
|
||||
url=url,
|
||||
data=json.dumps(payload).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with request.urlopen(req, timeout=timeout) as response:
|
||||
raw = response.read()
|
||||
return json.loads(raw.decode("utf-8")) if raw else None
|
||||
|
||||
|
||||
def send_probe_request(
|
||||
url: str, prompt: str, max_new_tokens: int, sampling_seed: int
|
||||
) -> None:
|
||||
payload = {
|
||||
"text": prompt,
|
||||
"sampling_params": {
|
||||
"sampling_seed": sampling_seed,
|
||||
"temperature": 0.0,
|
||||
"max_new_tokens": max_new_tokens,
|
||||
},
|
||||
"stream": False,
|
||||
}
|
||||
post_json(url.rstrip("/") + "/generate", payload, timeout=300.0)
|
||||
|
||||
|
||||
def run_profiler(
|
||||
url: str,
|
||||
output_dir: Optional[str],
|
||||
num_steps: int,
|
||||
profile_by_stage: bool,
|
||||
merge_profiles: bool,
|
||||
profile_prefix: Optional[str],
|
||||
probe_requests: int,
|
||||
probe_prompt: str,
|
||||
probe_max_new_tokens: Optional[int],
|
||||
probe_delay: float,
|
||||
start_step: Optional[int] = None,
|
||||
) -> Path:
|
||||
if output_dir is None:
|
||||
output_dir = tempfile.mkdtemp(prefix="sglang-torch-profile-")
|
||||
output_path = Path(output_dir).resolve()
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cmd = [
|
||||
sys.executable,
|
||||
"-m",
|
||||
"sglang.profiler",
|
||||
"--url",
|
||||
url,
|
||||
"--output-dir",
|
||||
str(output_path),
|
||||
"--num-steps",
|
||||
str(num_steps),
|
||||
"--cpu",
|
||||
"--gpu",
|
||||
"--merge-profiles" if merge_profiles else "--no-merge-profiles",
|
||||
"--profile-by-stage" if profile_by_stage else "--no-profile-by-stage",
|
||||
]
|
||||
if profile_prefix:
|
||||
cmd.extend(["--profile-prefix", profile_prefix])
|
||||
if start_step is not None:
|
||||
cmd.extend(["--start-step", str(start_step)])
|
||||
|
||||
profiler_proc = subprocess.Popen(cmd)
|
||||
try:
|
||||
if probe_requests > 0:
|
||||
time.sleep(max(0.0, probe_delay))
|
||||
effective_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
|
||||
for request_idx in range(probe_requests):
|
||||
send_probe_request(
|
||||
url=url,
|
||||
prompt=probe_prompt,
|
||||
max_new_tokens=effective_max_new_tokens,
|
||||
sampling_seed=request_idx,
|
||||
)
|
||||
if profiler_proc.poll() is not None:
|
||||
break
|
||||
return_code = profiler_proc.wait()
|
||||
finally:
|
||||
if profiler_proc.poll() is None:
|
||||
profiler_proc.kill()
|
||||
|
||||
if return_code != 0:
|
||||
raise subprocess.CalledProcessError(return_code, cmd)
|
||||
|
||||
deadline = time.time() + 15.0
|
||||
while time.time() < deadline:
|
||||
child_dirs = [path for path in output_path.iterdir() if path.is_dir()]
|
||||
if child_dirs:
|
||||
child_dirs.sort(key=lambda path: path.stat().st_mtime)
|
||||
newest_child = child_dirs[-1]
|
||||
if any(newest_child.glob("*.trace.json*")):
|
||||
return newest_child
|
||||
time.sleep(0.5)
|
||||
|
||||
child_dirs = [path for path in output_path.iterdir() if path.is_dir()]
|
||||
if child_dirs:
|
||||
child_dirs.sort(key=lambda path: path.stat().st_mtime)
|
||||
return child_dirs[-1]
|
||||
return output_path
|
||||
|
||||
|
||||
def select_heaviest_pid(
|
||||
events: Sequence[dict],
|
||||
event_filter: Callable[[dict], bool],
|
||||
pid_substring: Optional[str] = None,
|
||||
preferred_substrings: Iterable[str] = (),
|
||||
) -> Optional[str]:
|
||||
durations: Counter = Counter()
|
||||
for event in events:
|
||||
if not event_filter(event):
|
||||
continue
|
||||
pid = str(event.get("pid"))
|
||||
if pid_substring and pid_substring not in pid:
|
||||
continue
|
||||
durations[pid] += float(event["dur"])
|
||||
if not durations:
|
||||
return None
|
||||
|
||||
for substring in preferred_substrings:
|
||||
preferred = [pid for pid in durations if substring in pid]
|
||||
if preferred:
|
||||
return max(preferred, key=lambda pid: durations[pid])
|
||||
return max(durations, key=lambda pid: durations[pid])
|
||||
@@ -66,6 +66,7 @@ Defined in `python/sglang/test/test_utils.py`:
|
||||
| `stage-b-test-2-gpu-large` | `2-gpu-h100` | Two-GPU correctness and parallelism (TP/PP) on H100 |
|
||||
| `stage-b-test-4-gpu-b200` | `4-gpu-b200` | Early Blackwell coverage (SM100+ paths) on four GPUs |
|
||||
| `stage-b-kernel-unit-1-gpu-large` | `1-gpu-h100` | JIT kernel correctness tests under `python/sglang/jit_kernel/tests/` |
|
||||
| `stage-b-kernel-unit-1-gpu-b200` | `4-gpu-b200` | JIT kernel correctness tests for Blackwell / SM100-specific paths |
|
||||
| `stage-b-kernel-unit-8-gpu-h200` | `8-gpu-h200` | Multi-GPU JIT kernel correctness tests under `python/sglang/jit_kernel/tests/` |
|
||||
| `stage-b-kernel-benchmark-1-gpu-large` | `1-gpu-h100` | JIT kernel benchmark files under `python/sglang/jit_kernel/benchmark/` |
|
||||
| `stage-c-test-4-gpu-h100` | `4-gpu-h100` | Large 4-GPU H100 integration and scaling tests |
|
||||
@@ -75,7 +76,8 @@ Defined in `python/sglang/test/test_utils.py`:
|
||||
| `stage-c-test-deepep-8-gpu-h200` | `8-gpu-h200` | DeepEP at 8-GPU H200 scale |
|
||||
| `stage-c-test-8-gpu-b200` | `8-gpu-b200` | 8-GPU B200 suite (registered but not yet wired to a workflow) |
|
||||
| `stage-c-test-4-gpu-b200` | `4-gpu-b200` | 4-GPU B200 suite for large models on Blackwell |
|
||||
| `stage-c-test-4-gpu-gb200` | `4-gpu-gb200` | 4-GPU GB200 suite for large models on Grace Blackwell |
|
||||
| `stage-c-test-4-gpu-b200-small` | `4-gpu-b200` | Smaller 4-GPU B200 suite split onto low-disk B200 runners |
|
||||
| `stage-c-test-4-gpu-gb200` | `4-gpu-gb200` | 4-GPU GB200 suite for Grace Blackwell; registered in `run_suite.py`, but the PR workflow is currently disabled until a runner is provisioned |
|
||||
|
||||
#### Per-commit (AMD)
|
||||
|
||||
@@ -107,7 +109,7 @@ Defined in `python/sglang/test/test_utils.py`:
|
||||
|
||||
#### Nightly
|
||||
|
||||
Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml` amd `nightly-test-npu.yml`, not `pr-test.yml`. Examples:
|
||||
Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. Examples:
|
||||
|
||||
- `nightly-1-gpu` (CUDA)
|
||||
- `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids)
|
||||
@@ -132,6 +134,7 @@ Use the lightest suite that meets your test's needs:
|
||||
- **Most small GPU tests** → `stage-b-test-1-gpu-small` (default choice)
|
||||
- **Need H100 memory or Hopper features** → `stage-b-test-1-gpu-large`
|
||||
- **JIT kernel correctness** → `stage-b-kernel-unit-1-gpu-large`
|
||||
- **JIT kernel correctness for B200 / SM100 paths** → `stage-b-kernel-unit-1-gpu-b200`
|
||||
- **JIT kernel benchmarks** → `stage-b-kernel-benchmark-1-gpu-large`
|
||||
- **Multi-GPU** → only when the test actually needs multiple GPUs
|
||||
|
||||
@@ -352,6 +355,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
# Correctness tests in python/sglang/jit_kernel/tests/
|
||||
register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-large")
|
||||
register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-b200")
|
||||
register_cuda_ci(est_time=120, suite="stage-b-kernel-unit-8-gpu-h200")
|
||||
|
||||
# Benchmarks in python/sglang/jit_kernel/benchmark/
|
||||
|
||||
@@ -246,10 +246,14 @@ The `PipelineConfig` holds static model configuration and defines callback metho
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.multimodal_gen.configs.models import DiTConfig, VAEConfig
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.base import (
|
||||
ImagePipelineConfig, # for image generation
|
||||
# SpatialImagePipelineConfig, # alternative base
|
||||
# VideoPipelineConfig, # for video generation
|
||||
ImagePipelineConfig,
|
||||
ModelTaskType,
|
||||
# PipelineConfig, # common base for many video pipelines
|
||||
# SpatialImagePipelineConfig, # alternative base for spatial image models
|
||||
)
|
||||
from sglang.multimodal_gen.configs.models.dits.mymodel import MyModelDitConfig
|
||||
from sglang.multimodal_gen.configs.models.vaes.mymodel import MyModelVAEConfig
|
||||
@@ -314,6 +318,11 @@ class MyModelPipelineConfig(ImagePipelineConfig):
|
||||
return frames
|
||||
```
|
||||
|
||||
There is no separate `VideoPipelineConfig` base class. For video models, choose
|
||||
`ModelTaskType.T2V`, `ModelTaskType.I2V`, or `ModelTaskType.TI2V`, and follow
|
||||
existing video configs such as Wan, LTX, Hunyuan, Helios, or MOVA when deciding
|
||||
whether to subclass `PipelineConfig` directly or use a model-specific base.
|
||||
|
||||
**Important**: The `prepare_pos_cond_kwargs` / `prepare_neg_cond_kwargs` methods define what the DiT receives at each denoising step. These must match the DiT's `forward()` signature.
|
||||
|
||||
### Step 6: Implement the BeforeDenoisingStage (Core Step)
|
||||
@@ -502,15 +511,21 @@ In `python/sglang/multimodal_gen/registry.py`, register your configs:
|
||||
|
||||
```python
|
||||
register_configs(
|
||||
model_family="my_model",
|
||||
sampling_param_cls=MyModelSamplingParams,
|
||||
pipeline_config_cls=MyModelPipelineConfig,
|
||||
hf_model_paths=[
|
||||
"org/my-model-name", # HuggingFace model ID(s)
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path: "my-model" in path.lower(),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
`register_configs()` does not take a `model_family` argument. It registers the
|
||||
sampling and pipeline config classes, then resolves models by exact
|
||||
`hf_model_paths` or optional detector predicates.
|
||||
|
||||
The `EntryClass` in your pipeline file is automatically discovered by the registry's `_discover_and_register_pipelines()` function -- no additional registration needed for the pipeline class itself.
|
||||
|
||||
### Step 9: Verify Output Quality
|
||||
@@ -590,4 +605,5 @@ After the model produces non-noise output, read
|
||||
[references/testing-and-accuracy.md](references/testing-and-accuracy.md) before
|
||||
adding GPU cases, component-accuracy skips/hooks, suite entries, or benchmark
|
||||
claims. That reference tracks the current `gpu_cases.py` / `testcase_configs.py`
|
||||
/ `run_suite.py` split and the component-accuracy decision rules.
|
||||
/ `accuracy_testcase_configs.py` / `run_suite.py` split and the component-accuracy
|
||||
decision rules.
|
||||
|
||||
+8
-2
@@ -7,6 +7,9 @@ produce a non-noise image or video.
|
||||
|
||||
- Add concrete GPU integration cases in `python/sglang/multimodal_gen/test/server/gpu_cases.py`.
|
||||
- Keep reusable dataclasses, constants, thresholds, and testcase factory helpers in `python/sglang/multimodal_gen/test/server/testcase_configs.py`.
|
||||
- Add the case id to `python/sglang/multimodal_gen/test/server/accuracy_testcase_configs.py`
|
||||
only when it should be part of component-accuracy coverage. Adding a GPU case
|
||||
alone does not enroll it there.
|
||||
- Let `python/sglang/multimodal_gen/test/run_suite.py` own suite selection, runtime-based partitioning, and standalone test files. Do not hard-code CI shard lists elsewhere.
|
||||
- If a new standalone test file is added to a suite, update `STANDALONE_FILE_EST_TIMES` after the first measured CI/runtime value is known.
|
||||
|
||||
@@ -22,8 +25,8 @@ PYTHONPATH=python python3 python/sglang/multimodal_gen/test/run_suite.py --suite
|
||||
|
||||
If you add a new entry to `ONE_GPU_CASES`, `TWO_GPU_CASES`, or a B200-specific
|
||||
case group in `gpu_cases.py`, treat component accuracy as part of the
|
||||
model-adding workflow. Do not assume the new testcase will automatically fit the
|
||||
existing component-accuracy harness.
|
||||
model-adding workflow. Do not assume the new testcase will automatically fit or
|
||||
enter the existing component-accuracy harness.
|
||||
|
||||
The component-accuracy harness compares SGLang components against Diffusers/HF
|
||||
reference components. This is stricter than pipeline-level inference. New GPU
|
||||
@@ -46,9 +49,12 @@ cases commonly fail here for one of three reasons:
|
||||
|
||||
When adding a new GPU case, make this decision explicitly:
|
||||
|
||||
- if the case should have component-accuracy coverage, add its case id to
|
||||
`accuracy_testcase_configs.py`
|
||||
- if the family needs minimal harness wiring, add the smallest possible change in `accuracy_hooks.py`
|
||||
- if the case is only a variant of an already covered source component and topology, add a skip in `accuracy_config.py`
|
||||
- if the HF/Diffusers reference component cannot be compared faithfully, add a skip in `accuracy_config.py`
|
||||
- if the case is intentionally GPU-smoke-only, leave it out of `accuracy_testcase_configs.py` and keep that choice explicit in the PR notes
|
||||
|
||||
Do not add a new GPU case and wait for CI to discover missing component-accuracy
|
||||
wiring.
|
||||
|
||||
@@ -32,6 +32,7 @@ First use [../sglang-diffusion-benchmark-profile/SKILL.md](../sglang-diffusion-b
|
||||
- collect the perf dump baseline
|
||||
- capture one representative `torch.profiler` trace
|
||||
- rule out existing mainline fast paths
|
||||
- prove the run stayed on the native SGLang diffusion backend, not a diffusers fallback
|
||||
|
||||
If a future specialized optimization skill matches the kernel family better than AKO4ALL, hand off there instead. The diagnosis contract stays the same.
|
||||
|
||||
@@ -127,5 +128,9 @@ See [references/ako-loop.md](references/ako-loop.md) for the checklist and commo
|
||||
|
||||
- Treat AKO4ALL repo hygiene as a gate, not a suggestion.
|
||||
- Prefer exact local snapshot validation over hand-wavy “remote tree is close enough”.
|
||||
- Do not start or justify kernel work from traces collected after
|
||||
`Falling back to diffusers backend`, `Using diffusers backend`, or
|
||||
`Loaded diffusers pipeline`; fix backend selection and rerun the
|
||||
benchmark/profile workflow first.
|
||||
- Keep model-level validation honest: if microbench improves but denoise does not, do not keep the AKO-only variant in the main code path.
|
||||
- When writing conclusions, explain the win in terms of measurable causes such as lower registers per thread, higher occupancy, fewer executed instructions, or better scheduler eligibility.
|
||||
|
||||
Reference in New Issue
Block a user