minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[SKILL] Upgrade sglang profile and auto_benchmark skills (#24250)

Browse Source
This commit is contained in:
Xiaoyu Zhang
2026-05-02 10:12:47 +08:00
committed by GitHub
parent 4c2ed9a254
commit 321298da75
105 changed files with 10996 additions and 3813 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/llm-torch-profiler-analysis/SKILL.md
+149 -26
View File
@@ -55,18 +55,39 @@ add one short note after the tables with exactly one of:
| Existing trace triage | yes | yes | yes |
| Single-trace live capture | yes | yes, if torch profiler is enabled on server | requires profiler control endpoints |
| Two-trace mapping+formal triage | yes | yes | yes |
| Stage-aware live capture | yes | no | no |
| Stage-separated live workload | yes | yes | yes, with a writable shared trace dir or per-stage host runner |
| `--profile-by-stage` capture | yes | no | no |
| `--profile-prefix` control | yes | usually ignored on HTTP profiler route | usually ignored on HTTP profiler route |
For TensorRT-LLM, live capture only works when the server exposes `/start_profile` and
`/stop_profile`, and when the deployment already provides a shared trace path plus the
required env vars.
## Validation Notes
## Real H100 Validation
This unified workflow has been validated with a `4x H100` matrix across SGLang,
vLLM, and TensorRT-LLM. Use these model shapes as representative coverage when
refreshing or extending the skill:
The current reference run is the `4x H100` matrix captured on `2026-04-23` on
`h100_sglang` under:
- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3`
Rendered markdown bundle:
- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md`
Validated model directories:
- `mixtral_8x7b_instruct`
- `qwen2_5_32b_instruct`
- `qwen3_32b`
Each model directory contains:
- `analysis_sglang.txt`
- `analysis_vllm.txt`
- `analysis_trtllm.txt`
- framework-specific trace roots and probe artifacts
Validated matrix:
| Model | SGLang | vLLM | TensorRT-LLM | Result |
| --- | --- | --- | --- | --- |
@@ -74,17 +95,49 @@ refreshing or extending the skill:
| `Qwen/Qwen2.5-32B-Instruct` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; benchmark probes returned direct, non-empty text |
| `Qwen/Qwen3-32B` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; vLLM and TensorRT-LLM chat probes often emitted `<think>` prefixes |
Use this run as the main H100 reference.
The older `2026-04-22` single-card Qwen3 matrix is still useful for bring-up, but it is
not the default reference anymore.
Stage-separated workload validation captured on `2026-05-01` on `h100_sglang`:
- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation`
- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation_large`
Validated models:
| Model | GPU | Workloads | Result |
| --- | --- | --- | --- |
| `Qwen/Qwen2.5-0.5B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate `prefill/*.trace.json.gz` and `decode/*.trace.json.gz`; kernel, overlap, and fuse tables rendered with separate `extend/prefill` and `decode` sections |
| `Qwen/Qwen2.5-1.5B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate `prefill/*.trace.json.gz` and `decode/*.trace.json.gz`; kernel, overlap, and fuse tables rendered with separate `extend/prefill` and `decode` sections |
| `Qwen/Qwen2.5-7B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate traces; prefill kernel table captured 28-layer GEMM/FA3/RMSNorm work, decode captured 5-step graph launches, and fuse rows were split by stage |
| `Qwen/Qwen2.5-14B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate traces; prefill kernel table captured 48-layer GEMM/FA3/RMSNorm work, decode captured 5-step graph launches, and fuse rows were split by stage |
| `Qwen/Qwen3-8B` | `2x H100`, TP=2 | prefill `4090->1`, decode `1->2048`, warmup 10/capture 5 | generated separate prefill/decode traces and all three tables; unique probe prompts avoided prefix-cache pollution in the prefill table |
| `mistralai/Mistral-7B-Instruct-v0.3` | `2x H100`, TP=2 | prefill `4090->1`, decode `1->2048`, warmup 10/capture 5 | generated separate prefill/decode traces and all three tables; server logs showed no repeated-prompt prefix-cache shortcut during the active prefill window |
This validation also covers the compatibility fix for older SGLang profiler
state machines: workload-separated live capture labels stages by output
directory and avoids nesting SGLang's internal `profile_by_stage` state machine
inside each workload. The helper
adds one internal scheduler guard step because SGLang increments `forward_ct`
before checking whether the profiler should stop; without that guard, a
`num_steps=1` prefill capture can stop just before the actual prefill forward.
The 2026-05-01 two-card validation artifacts for the additional models are:
- `/data/bbuf/validate/core_skill_validation_20260501/qwen3_8b/profiler`
- `/data/bbuf/validate/core_skill_validation_20260501/mistral_7b_instruct_v03/profiler`
To render a validated run into one markdown document:
```bash
python3 scripts/render_triage_markdown_bundle.py \
--analysis-root /path/to/analysis_root \
--output /path/to/analysis_bundle.md
--analysis-root /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3 \
--output /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md
```
The bundle groups by model and keeps the three tables for each framework.
Validation notes:
H100 notes:
- all three frameworks now render kernel, overlap, and fuse tables with separate `extend/prefill` and `decode` sections when the trace contains a clean stage split
- SGLang live capture is validated and calls the server profiler API directly instead of shelling out to `sglang.profiler`
@@ -92,7 +145,8 @@ Validation notes:
- SGLang kernel-site reconstruction keeps sampling disabled in the mapping path so the optimized parser does not perturb SGLang table output; equality rechecks matched for `Mixtral-8x7B-Instruct-v0.1`, `Qwen3-32B`, and `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`
- vLLM live capture requires `--output-dir` to match the server `torch_profiler_dir`; the validated H100 flow uses `--profiler-config {"profiler":"torch","torch_profiler_dir":"..."}` and then drives `/start_profile` and `/stop_profile`
- TensorRT-LLM validation stays on `--backend pytorch`; the H100 flow writes the trace with `TLLM_TORCH_PROFILE_TRACE` and then analyzes the saved trace
- the 2026-04-22 TensorRT-LLM 1.0.0 `py_executor.py` profiler setup still needed a `with_stack=True` override for table-quality Python locations; re-check this on TensorRT-LLM 1.2.1 or any 1.3.x release-candidate image before assuming the override is still required
- the 2026-04-22 TensorRT-LLM 1.0.0 `py_executor.py` profiler setup still needed a `with_stack=True` override for table-quality Python locations, and the matrix runner generated that override under `/data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm`; re-check this on TensorRT-LLM 1.2.1 or any 1.3.x release-candidate image before assuming the override is still required
- on this host, keep all trace roots under `/data/...`, not `/home/...`
## When To Use It
@@ -119,6 +173,43 @@ Handle it as a backend-selection issue, not as native-kernel profiler evidence.
## Main Flows
## Stage-Separated Live Capture Contract
Live capture must not use one mixed prompt as the default.
By default, `analyze_llm_torch_profile.py --url ...` captures two labeled
workloads and then renders the same three tables with separate stage sections:
- prefill: synthetic input length `4090`, output length `1`
- decode: synthetic input length `1`, output length `2048`
Every live profiler path warms up `10` steps before arming the profiler and then
captures `5` active steps by default. Keep this warmup/active split aligned
across SGLang, vLLM, and TensorRT-LLM before comparing kernel tables.
Use these options to override the contract when the benchmark workload is known:
```bash
--profile-workload both \
--warmup-steps 10 --num-steps 5 \
--prefill-input-len 4090 --prefill-output-len 1 \
--decode-input-len 1 --decode-output-len 2048
```
Allowed `--profile-workload` values:
- `both`: default; capture prefill and decode separately
- `prefill`: capture only the long-input / one-token workload
- `decode`: capture only the one-input / long-output workload
- `legacy`: keep the old `--probe-prompt` / `--probe-max-new-tokens` behavior
For `sglang-sota-performance`, do not use the defaults if the slow SGLang
benchmark scenario has a known input/output distribution.
Set the profiler lengths from that slow scenario instead: prefill uses the slow
input length with output `1`, and decode uses input `1` with the slow output
length. For a mixed dataset, profile the slowest representative bucket such as
the p50 or p95 input/output pair used in the benchmark report, and record the
bucket in the artifact notes.
### 1. Single-trace triage from an existing profile dir or trace
```bash
@@ -136,14 +227,24 @@ mapping/formal pair is needed.
python3 scripts/analyze_llm_torch_profile.py \
--framework sglang \
--url http://127.0.0.1:30000 \
--output-dir /tmp/llm-profiler/sglang_profile_live \
--output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/sglang_profile_live \
--num-steps 5 \
--profile-by-stage
--warmup-steps 10 \
--profile-by-stage \
--profile-workload both
```
The script sends `POST /start_profile` to the SGLang server directly.
The script writes `server_args.json`, sends the probe requests after profiling is armed,
and waits longer for trace flush than the earlier implementation.
Keep `--output-dir` under `/data/...` so later analysis and docs can see the trace.
The script writes `server_args.json`, warms up with the same workload shape,
sends the active probe requests after profiling is armed, captures separate
`prefill/` and `decode/` profile roots by default, and waits longer for trace
flush than the earlier implementation.
For the default workload-separated capture, the directory name labels the stage
and the SGLang internal `profile_by_stage` mode is not used inside each
workload. This avoids mixing a one-token prefill probe with a separate decode
profile. The helper still adds one internal guard step because older SGLang
profilers check the target counter before running the next forward.
### 3. Single-trace live capture from vLLM
@@ -151,7 +252,7 @@ Launch vLLM with torch profiler enabled, for example:
```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--profiler-config '{"profiler":"torch","torch_profiler_dir":"/tmp/llm-profiler/vllm_profile"}'
--profiler-config '{"profiler":"torch","torch_profiler_dir":"/data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile"}'
```
Then run:
@@ -160,14 +261,20 @@ Then run:
python3 scripts/analyze_llm_torch_profile.py \
--framework vllm \
--url http://127.0.0.1:8000 \
--output-dir /tmp/llm-profiler/vllm_profile \
--output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile \
--num-steps 5 \
--no-profile-by-stage
--warmup-steps 10 \
--no-profile-by-stage \
--profile-workload both
```
For vLLM, `--output-dir` must point to the same `torch_profiler_dir` the server uses.
The current vLLM profiler config already defaults `torch_profiler_with_stack=true`,
so the runner only needs to set `torch_profiler_dir`.
On `h100_sglang`, external vLLM containers should mount both:
- `/data/.cache/huggingface:/root/.cache/huggingface`
- `/data/bbuf/validate/unified_llm_profiler_skill:/data/bbuf/validate/unified_llm_profiler_skill`
### 4. Single-trace live capture from TensorRT-LLM
@@ -187,11 +294,15 @@ python3 scripts/analyze_llm_torch_profile.py \
--url http://127.0.0.1:8000 \
--output-dir /shared/path \
--num-steps 5 \
--no-profile-by-stage
--no-profile-by-stage \
--profile-workload both
```
If the deployment does not expose the profiler control endpoints, fall back to analyzing
an existing trace instead of trying live capture.
If the TensorRT-LLM trace output is configured as one fixed file path, use
`scripts/run_trtllm_pytorch_profile_host.sh --stage prefill` and `--stage decode`
instead of direct `--profile-workload both`, so each stage gets its own trace file.
On the current TensorRT-LLM mainline path, `py_executor.py` creates the torch profiler
with `record_shapes=True` and `with_modules=True` but not `with_stack=True`.
@@ -200,19 +311,21 @@ For table-quality validation, use the override generator:
```bash
python3 scripts/make_trtllm_py_executor_override.py \
--source /path/to/original/py_executor.py \
--output /tmp/llm-profiler/py_executor_with_stack.py
--output /data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm/py_executor_with_stack.py
```
The validated TensorRT-LLM flow is:
The matrix runner does this automatically on H100 before TensorRT-LLM capture starts.
1. launch `trtllm-serve` with `TLLM_TORCH_PROFILE_TRACE=/shared/path/trace.json`
This is the validated TensorRT-LLM flow on `h100_sglang`:
1. launch `trtllm-serve` with `TLLM_TORCH_PROFILE_TRACE=/data/.../trace.json`
2. run a few benchmark requests
3. analyze the emitted trace with `--input /shared/path/trace.json`
3. analyze the emitted trace with `--input /data/.../trace.json`
### 5. Two-trace triage from existing profile dirs or traces
```bash
python3 scripts/analyze_llm_torch_profile.py triage \
python3 scripts/analyze_llm_torch_profile.py \
--mapping-input /path/to/graph_off_profile_dir \
--formal-input /path/to/graph_on_profile_dir
```
@@ -222,7 +335,7 @@ Use this when you need stronger overlap attribution and kernel-to-source mapping
### 6. Two-trace triage from running servers
```bash
python3 scripts/analyze_llm_torch_profile.py triage \
python3 scripts/analyze_llm_torch_profile.py \
--framework sglang \
--mapping-url http://127.0.0.1:31025 \
--formal-url http://127.0.0.1:31026 \
@@ -241,7 +354,13 @@ For `vllm` or `TensorRT-LLM`, use the same shape but pass:
`--profile-by-stage` is only meaningful on the SGLang live-capture path.
- On ordinary non-PD SGLang serving, it is still useful because prefill and decode usually have very different bottlenecks.
- With `--profile-workload both` / `prefill` / `decode`, workload directories
are the stage labels; the live-capture helper disables SGLang's internal
stage profiler per workload, warms up first, and captures the requested
active step count for the selected workload.
- On legacy or hand-captured SGLang serving, internal `profile_by_stage` is
still useful because prefill and decode usually have very different
bottlenecks.
- On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
- PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary `profile_by_stage`.
- For `vllm` and `TensorRT-LLM`, disable it with `--no-profile-by-stage`.
@@ -279,8 +398,10 @@ It exists to recover `kernel -> cpu_op -> python scope`.
1. If the user only wants a diagnosis, one trace is enough.
2. Prefer one-rank traces over merged traces whenever the profiler emitted both.
3. For a live server, let the script drive the profiler only when the framework-specific prerequisites are already met.
4. Prefer SGLang `--profile-by-stage` unless the user explicitly wants an all-stage mixed trace.
5. Create or clean the target trace directory before live capture so the profiler can write artifacts without permission surprises.
4. Prefer `--profile-workload both`; use `legacy` only when reproducing an old trace contract.
5. Prefer workload-separated SGLang capture; use internal `--profile-by-stage`
mainly for `legacy` or manually collected traces.
6. When on `h100_sglang`, create or clean the target trace directory through `docker exec sglang_bbuf ...` so the path is definitely writable under `/data`.
### Two-trace workflow
@@ -312,6 +433,8 @@ Load these only when needed:
- overlap labels, dependency-risk interpretation, and limits
- [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md)
- mixed source-backed catalog of existing fuse and overlap patterns, including mainline rows plus PR-backed / in-flight rows
- [references/vllm-torch-compile-fusions.md](references/vllm-torch-compile-fusions.md)
- current vLLM torch.compile fusion passes and the source patterns they target
- [references/overlap-catalog.md](references/overlap-catalog.md)
- overlap-only lookup table across LLM, VLM, diffusion, disaggregation, HiSparse, and speculative scheduling