[SKILL] Upgrade sglang profile and auto_benchmark skills (#24250)

2026-05-02 10:12:47 +08:00
parent 4c2ed9a254
commit 321298da75
105 changed files with 10996 additions and 3813 deletions
@@ -55,18 +55,39 @@ add one short note after the tables with exactly one of:
 | Existing trace triage | yes | yes | yes |
 | Single-trace live capture | yes | yes, if torch profiler is enabled on server | requires profiler control endpoints |
 | Two-trace mapping+formal triage | yes | yes | yes |
-| Stage-aware live capture | yes | no | no |
+| Stage-separated live workload | yes | yes | yes, with a writable shared trace dir or per-stage host runner |
+| `--profile-by-stage` capture | yes | no | no |
 | `--profile-prefix` control | yes | usually ignored on HTTP profiler route | usually ignored on HTTP profiler route |

 For TensorRT-LLM, live capture only works when the server exposes `/start_profile` and
 `/stop_profile`, and when the deployment already provides a shared trace path plus the
 required env vars.

-## Validation Notes
+## Real H100 Validation

-This unified workflow has been validated with a `4x H100` matrix across SGLang,
-vLLM, and TensorRT-LLM. Use these model shapes as representative coverage when
-refreshing or extending the skill:
+The current reference run is the `4x H100` matrix captured on `2026-04-23` on
+`h100_sglang` under:
+
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3`
+
+Rendered markdown bundle:
+
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md`
+
+Validated model directories:
+
+- `mixtral_8x7b_instruct`
+- `qwen2_5_32b_instruct`
+- `qwen3_32b`
+
+Each model directory contains:
+
+- `analysis_sglang.txt`
+- `analysis_vllm.txt`
+- `analysis_trtllm.txt`
+- framework-specific trace roots and probe artifacts
+
+Validated matrix:

 | Model | SGLang | vLLM | TensorRT-LLM | Result |
 | --- | --- | --- | --- | --- |
@@ -74,17 +95,49 @@ refreshing or extending the skill:
 | `Qwen/Qwen2.5-32B-Instruct` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; benchmark probes returned direct, non-empty text |
 | `Qwen/Qwen3-32B` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; vLLM and TensorRT-LLM chat probes often emitted `<think>` prefixes |

+Use this run as the main H100 reference.
+The older `2026-04-22` single-card Qwen3 matrix is still useful for bring-up, but it is
+not the default reference anymore.
+
+Stage-separated workload validation captured on `2026-05-01` on `h100_sglang`:
+
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation`
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation_large`
+
+Validated models:
+
+| Model | GPU | Workloads | Result |
+| --- | --- | --- | --- |
+| `Qwen/Qwen2.5-0.5B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate `prefill/*.trace.json.gz` and `decode/*.trace.json.gz`; kernel, overlap, and fuse tables rendered with separate `extend/prefill` and `decode` sections |
+| `Qwen/Qwen2.5-1.5B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate `prefill/*.trace.json.gz` and `decode/*.trace.json.gz`; kernel, overlap, and fuse tables rendered with separate `extend/prefill` and `decode` sections |
+| `Qwen/Qwen2.5-7B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate traces; prefill kernel table captured 28-layer GEMM/FA3/RMSNorm work, decode captured 5-step graph launches, and fuse rows were split by stage |
+| `Qwen/Qwen2.5-14B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate traces; prefill kernel table captured 48-layer GEMM/FA3/RMSNorm work, decode captured 5-step graph launches, and fuse rows were split by stage |
+| `Qwen/Qwen3-8B` | `2x H100`, TP=2 | prefill `4090->1`, decode `1->2048`, warmup 10/capture 5 | generated separate prefill/decode traces and all three tables; unique probe prompts avoided prefix-cache pollution in the prefill table |
+| `mistralai/Mistral-7B-Instruct-v0.3` | `2x H100`, TP=2 | prefill `4090->1`, decode `1->2048`, warmup 10/capture 5 | generated separate prefill/decode traces and all three tables; server logs showed no repeated-prompt prefix-cache shortcut during the active prefill window |
+
+This validation also covers the compatibility fix for older SGLang profiler
+state machines: workload-separated live capture labels stages by output
+directory and avoids nesting SGLang's internal `profile_by_stage` state machine
+inside each workload. The helper
+adds one internal scheduler guard step because SGLang increments `forward_ct`
+before checking whether the profiler should stop; without that guard, a
+`num_steps=1` prefill capture can stop just before the actual prefill forward.
+The 2026-05-01 two-card validation artifacts for the additional models are:
+
+- `/data/bbuf/validate/core_skill_validation_20260501/qwen3_8b/profiler`
+- `/data/bbuf/validate/core_skill_validation_20260501/mistral_7b_instruct_v03/profiler`
+
 To render a validated run into one markdown document:

 ```bash
 python3 scripts/render_triage_markdown_bundle.py \
-  --analysis-root /path/to/analysis_root \
-  --output /path/to/analysis_bundle.md
+  --analysis-root /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3 \
+  --output /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md
 ```

 The bundle groups by model and keeps the three tables for each framework.

-Validation notes:
+H100 notes:

 - all three frameworks now render kernel, overlap, and fuse tables with separate `extend/prefill` and `decode` sections when the trace contains a clean stage split
 - SGLang live capture is validated and calls the server profiler API directly instead of shelling out to `sglang.profiler`
@@ -92,7 +145,8 @@ Validation notes:
 - SGLang kernel-site reconstruction keeps sampling disabled in the mapping path so the optimized parser does not perturb SGLang table output; equality rechecks matched for `Mixtral-8x7B-Instruct-v0.1`, `Qwen3-32B`, and `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`
 - vLLM live capture requires `--output-dir` to match the server `torch_profiler_dir`; the validated H100 flow uses `--profiler-config {"profiler":"torch","torch_profiler_dir":"..."}` and then drives `/start_profile` and `/stop_profile`
 - TensorRT-LLM validation stays on `--backend pytorch`; the H100 flow writes the trace with `TLLM_TORCH_PROFILE_TRACE` and then analyzes the saved trace
- the 2026-04-22 TensorRT-LLM 1.0.0 `py_executor.py` profiler setup still needed a `with_stack=True` override for table-quality Python locations; re-check this on TensorRT-LLM 1.2.1 or any 1.3.x release-candidate image before assuming the override is still required
+- the 2026-04-22 TensorRT-LLM 1.0.0 `py_executor.py` profiler setup still needed a `with_stack=True` override for table-quality Python locations, and the matrix runner generated that override under `/data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm`; re-check this on TensorRT-LLM 1.2.1 or any 1.3.x release-candidate image before assuming the override is still required
+- on this host, keep all trace roots under `/data/...`, not `/home/...`

 ## When To Use It

@@ -119,6 +173,43 @@ Handle it as a backend-selection issue, not as native-kernel profiler evidence.

 ## Main Flows

+## Stage-Separated Live Capture Contract
+
+Live capture must not use one mixed prompt as the default.
+By default, `analyze_llm_torch_profile.py --url ...` captures two labeled
+workloads and then renders the same three tables with separate stage sections:
+
+- prefill: synthetic input length `4090`, output length `1`
+- decode: synthetic input length `1`, output length `2048`
+
+Every live profiler path warms up `10` steps before arming the profiler and then
+captures `5` active steps by default. Keep this warmup/active split aligned
+across SGLang, vLLM, and TensorRT-LLM before comparing kernel tables.
+
+Use these options to override the contract when the benchmark workload is known:
+
+```bash
+--profile-workload both \
+--warmup-steps 10 --num-steps 5 \
+--prefill-input-len 4090 --prefill-output-len 1 \
+--decode-input-len 1 --decode-output-len 2048
+```
+
+Allowed `--profile-workload` values:
+
+- `both`: default; capture prefill and decode separately
+- `prefill`: capture only the long-input / one-token workload
+- `decode`: capture only the one-input / long-output workload
+- `legacy`: keep the old `--probe-prompt` / `--probe-max-new-tokens` behavior
+
+For `sglang-sota-performance`, do not use the defaults if the slow SGLang
+benchmark scenario has a known input/output distribution.
+Set the profiler lengths from that slow scenario instead: prefill uses the slow
+input length with output `1`, and decode uses input `1` with the slow output
+length. For a mixed dataset, profile the slowest representative bucket such as
+the p50 or p95 input/output pair used in the benchmark report, and record the
+bucket in the artifact notes.
+
 ### 1. Single-trace triage from an existing profile dir or trace

 ```bash
@@ -136,14 +227,24 @@ mapping/formal pair is needed.
 python3 scripts/analyze_llm_torch_profile.py \
  --framework sglang \
  --url http://127.0.0.1:30000 \
-  --output-dir /tmp/llm-profiler/sglang_profile_live \
+  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/sglang_profile_live \
  --num-steps 5 \
-  --profile-by-stage
+  --warmup-steps 10 \
+  --profile-by-stage \
+  --profile-workload both
 ```

 The script sends `POST /start_profile` to the SGLang server directly.
-The script writes `server_args.json`, sends the probe requests after profiling is armed,
-and waits longer for trace flush than the earlier implementation.
+Keep `--output-dir` under `/data/...` so later analysis and docs can see the trace.
+The script writes `server_args.json`, warms up with the same workload shape,
+sends the active probe requests after profiling is armed, captures separate
+`prefill/` and `decode/` profile roots by default, and waits longer for trace
+flush than the earlier implementation.
+For the default workload-separated capture, the directory name labels the stage
+and the SGLang internal `profile_by_stage` mode is not used inside each
+workload. This avoids mixing a one-token prefill probe with a separate decode
+profile. The helper still adds one internal guard step because older SGLang
+profilers check the target counter before running the next forward.

 ### 3. Single-trace live capture from vLLM

@@ -151,7 +252,7 @@ Launch vLLM with torch profiler enabled, for example:

 ```bash
 vllm serve meta-llama/Llama-3.1-8B-Instruct \
-  --profiler-config '{"profiler":"torch","torch_profiler_dir":"/tmp/llm-profiler/vllm_profile"}'
+  --profiler-config '{"profiler":"torch","torch_profiler_dir":"/data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile"}'
 ```

 Then run:
@@ -160,14 +261,20 @@ Then run:
 python3 scripts/analyze_llm_torch_profile.py \
  --framework vllm \
  --url http://127.0.0.1:8000 \
-  --output-dir /tmp/llm-profiler/vllm_profile \
+  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile \
  --num-steps 5 \
-  --no-profile-by-stage
+  --warmup-steps 10 \
+  --no-profile-by-stage \
+  --profile-workload both
 ```

 For vLLM, `--output-dir` must point to the same `torch_profiler_dir` the server uses.
 The current vLLM profiler config already defaults `torch_profiler_with_stack=true`,
 so the runner only needs to set `torch_profiler_dir`.
+On `h100_sglang`, external vLLM containers should mount both:
+
+- `/data/.cache/huggingface:/root/.cache/huggingface`
+- `/data/bbuf/validate/unified_llm_profiler_skill:/data/bbuf/validate/unified_llm_profiler_skill`

 ### 4. Single-trace live capture from TensorRT-LLM

@@ -187,11 +294,15 @@ python3 scripts/analyze_llm_torch_profile.py \
  --url http://127.0.0.1:8000 \
  --output-dir /shared/path \
  --num-steps 5 \
-  --no-profile-by-stage
+  --no-profile-by-stage \
+  --profile-workload both
 ```

 If the deployment does not expose the profiler control endpoints, fall back to analyzing
 an existing trace instead of trying live capture.
+If the TensorRT-LLM trace output is configured as one fixed file path, use
+`scripts/run_trtllm_pytorch_profile_host.sh --stage prefill` and `--stage decode`
+instead of direct `--profile-workload both`, so each stage gets its own trace file.

 On the current TensorRT-LLM mainline path, `py_executor.py` creates the torch profiler
 with `record_shapes=True` and `with_modules=True` but not `with_stack=True`.
@@ -200,19 +311,21 @@ For table-quality validation, use the override generator:
 ```bash
 python3 scripts/make_trtllm_py_executor_override.py \
  --source /path/to/original/py_executor.py \
-  --output /tmp/llm-profiler/py_executor_with_stack.py
+  --output /data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm/py_executor_with_stack.py
 ```

-The validated TensorRT-LLM flow is:
+The matrix runner does this automatically on H100 before TensorRT-LLM capture starts.

-1. launch `trtllm-serve` with `TLLM_TORCH_PROFILE_TRACE=/shared/path/trace.json`
+This is the validated TensorRT-LLM flow on `h100_sglang`:
+
+1. launch `trtllm-serve` with `TLLM_TORCH_PROFILE_TRACE=/data/.../trace.json`
 2. run a few benchmark requests
-3. analyze the emitted trace with `--input /shared/path/trace.json`
+3. analyze the emitted trace with `--input /data/.../trace.json`

 ### 5. Two-trace triage from existing profile dirs or traces

 ```bash
-python3 scripts/analyze_llm_torch_profile.py triage \
+python3 scripts/analyze_llm_torch_profile.py \
  --mapping-input /path/to/graph_off_profile_dir \
  --formal-input /path/to/graph_on_profile_dir
 ```
@@ -222,7 +335,7 @@ Use this when you need stronger overlap attribution and kernel-to-source mapping
 ### 6. Two-trace triage from running servers

 ```bash
-python3 scripts/analyze_llm_torch_profile.py triage \
+python3 scripts/analyze_llm_torch_profile.py \
  --framework sglang \
  --mapping-url http://127.0.0.1:31025 \
  --formal-url http://127.0.0.1:31026 \
@@ -241,7 +354,13 @@ For `vllm` or `TensorRT-LLM`, use the same shape but pass:

 `--profile-by-stage` is only meaningful on the SGLang live-capture path.

- On ordinary non-PD SGLang serving, it is still useful because prefill and decode usually have very different bottlenecks.
+- With `--profile-workload both` / `prefill` / `decode`, workload directories
+  are the stage labels; the live-capture helper disables SGLang's internal
+  stage profiler per workload, warms up first, and captures the requested
+  active step count for the selected workload.
+- On legacy or hand-captured SGLang serving, internal `profile_by_stage` is
+  still useful because prefill and decode usually have very different
+  bottlenecks.
 - On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
 - PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary `profile_by_stage`.
 - For `vllm` and `TensorRT-LLM`, disable it with `--no-profile-by-stage`.
@@ -279,8 +398,10 @@ It exists to recover `kernel -> cpu_op -> python scope`.
 1. If the user only wants a diagnosis, one trace is enough.
 2. Prefer one-rank traces over merged traces whenever the profiler emitted both.
 3. For a live server, let the script drive the profiler only when the framework-specific prerequisites are already met.
-4. Prefer SGLang `--profile-by-stage` unless the user explicitly wants an all-stage mixed trace.
-5. Create or clean the target trace directory before live capture so the profiler can write artifacts without permission surprises.
+4. Prefer `--profile-workload both`; use `legacy` only when reproducing an old trace contract.
+5. Prefer workload-separated SGLang capture; use internal `--profile-by-stage`
+   mainly for `legacy` or manually collected traces.
+6. When on `h100_sglang`, create or clean the target trace directory through `docker exec sglang_bbuf ...` so the path is definitely writable under `/data`.

 ### Two-trace workflow

@@ -312,6 +433,8 @@ Load these only when needed:
  - overlap labels, dependency-risk interpretation, and limits
 - [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md)
  - mixed source-backed catalog of existing fuse and overlap patterns, including mainline rows plus PR-backed / in-flight rows
+- [references/vllm-torch-compile-fusions.md](references/vllm-torch-compile-fusions.md)
+  - current vLLM torch.compile fusion passes and the source patterns they target
 - [references/overlap-catalog.md](references/overlap-catalog.md)
  - overlap-only lookup table across LLM, VLM, diffusion, disaggregation, HiSparse, and speculative scheduling

@@ -28,21 +28,14 @@ overlap opportunity as novel.

 The catalog is grouped by reusable optimization family, not by one specific model.

-Refresh note `2026-04-22`: rescanned current `sglang`, `flashinfer`,
-`TensorRT-LLM`, and `vllm` mainline plus rechecked referenced PR state via the
-GitHub API on `2026-04-22`. Stable current-code families such as Qwen-style
-shared-expert top-k append, TensorRT-LLM Triton fused add+RMSNorm+FP8 quant,
-and vLLM `merge_attn_states` attention-output quant are folded into the
-mainline rows below. Closed-unmerged SGLang
-[#22410](https://github.com/sgl-project/sglang/pull/22410) and FlashInfer
-[#2840](https://github.com/flashinfer-ai/flashinfer/pull/2840) were removed
-from the PR-backed sections. Keep FlashInfer
-[#3058](https://github.com/flashinfer-ai/flashinfer/pull/3058) /
-[#3079](https://github.com/flashinfer-ai/flashinfer/pull/3079) in mind because
-that branch was reverted, and keep vLLM
-[#40057](https://github.com/vllm-project/vllm/pull/40057) in mind when using
-B200 FP4 MoE test coverage as a signal: it disables some B200 FP4 MoE layer
-tests rather than proving the kernel family is absent.
+Refresh note `2026-05-01`: rescanned current `sglang` and vLLM mainline, then
+rechecked recent merged and open optimization PRs through the GitHub CLI/API.
+The vLLM torch.compile pass inventory is now split out in
+[`vllm-torch-compile-fusions.md`](vllm-torch-compile-fusions.md). Stable
+current-code families remain folded into the mainline rows below. New
+status-sensitive rows were added for DeepSeek-V4, GLM5 NSA / PDL, NVFP4 MoE,
+torch.compile decode, vLLM DSV4, vLLM ROCm WMMA, and vLLM GPU/CPU sync-removal
+work. Recheck PR state before treating an in-flight row as shipped.

 ## 1. LLM / SRT fused-kernel families

@@ -147,12 +140,21 @@ Stable entries should be folded into the mainline family rows above.
 | PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/jit_kernel/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/jit_kernel/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
 | PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
 | PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`sgl-kernel/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
+| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/jit_kernel/nvfp4.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
+| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
+| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
+| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
+| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
+| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |

 ## 7. PR-backed / in-flight kernel-overlap families

 | Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
 | --- | --- | --- | --- | --- |
 | PR `#21877` fused down-GEMM + combine superseding SBO | `enable_fused_grouped_gemm_combine`<br>`combine`<br>`down_gemm` | `PR #21877`<br>`python/sglang/srt/server_args.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | Fused combine eliminates the standalone combine window, so SBO is intentionally disabled when this path is on | If the trace discussion is about combine overlap, first classify it as this upstream fused-overlap family. |
+| PR `#23965` PDL for DSV32 / GLM5 kernels | `enable_pdl`<br>`TRTLLM_ENABLE_PDL`<br>`cudaGridDependencySynchronize`<br>`DSV32`<br>`GLM5` | `PR #23965`<br>`python/sglang/srt/layers`<br>`sgl-kernel` | Enables programmatic dependent launch on selected DeepSeek / GLM kernels so dependent decode kernels can overlap launch-to-start gaps | Treat tight same-stream decode windows around DSV32 / GLM5 as an in-flight PDL overlap family. |
+| PR `#21878` TTFT / TPOT torch.compile optimization | `enable_torch_compile`<br>`decode graph`<br>`piecewise cudagraph` | `PR #21878`<br>`python/sglang/srt` | Uses compiler and graph capture changes to shave TTFT / TPOT rather than adding one handwritten kernel | If the trace shows many small compiler-visible decode ops, compare against this compile-overlap / graph-capture family first. |
+| PR `#24168` batched GPU-to-CPU sync for logprobs / embeddings | `logprobs`<br>`embeddings`<br>`GPU->CPU sync`<br>`batch sync` | `PR #24168`<br>`python/sglang/srt` | Batches per-request synchronization work that can otherwise serialize decode progress around logprob or embedding outputs | Treat per-request CPU sync stalls in logprob / embedding traces as a concrete in-flight SGLang scheduler/data-movement family. |

 ## 8. FlashInfer mainline fused-kernel families

@@ -272,6 +274,15 @@ contain the same implementation.
 | PR `#37646` ROCm AITER fused allreduce + RMSNorm | `rocm_aiter_fused_allreduce_rmsnorm`<br>`custom_fused_ar_rms`<br>`RocmAiterAllReduceFusionPass` | `PR #37646`<br>`vllm/_aiter_ops.py`<br>`vllm/compilation/passes/pass_manager.py` | ROCm-specific compile-time path swaps the generic all-reduce fusion pass for an AITER fused allreduce-plus-RMSNorm kernel family | Treat ROCm TP all-reduce + RMSNorm ladders as an in-flight upstream fused-collective family first. |
 | PR `#36413` FlashInfer RMSNorm + FP4 quant fusion | `fuse_norm_quant`<br>`flashinfer`<br>`NVFP4`<br>`rmsnorm + fp4 quant` | `PR #36413`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/docs/design/fusions.md` | FlashInfer-backed norm-plus-FP4 quant fusion extends the existing RMSNorm+quant family to NVFP4 flows | Treat split RMSNorm + FP4 quant ladders as an upstream in-flight family, not a fresh idea. |
 | PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | Extends the specialized router GEMM family to GLM5 hidden size and uses PDL to overlap the router launch with the preceding fused allreduce-plus-RMS block | Treat this as an in-flight upstream router-kernel plus launch-overlap family before calling it novel. |
+| PR `#41455` ROCm WMMA paged prefill and split-K decode | `wmma`<br>`paged prefill`<br>`split-K decode`<br>`ROCm attention` | `PR #41455`<br>`vllm/v1/attention`<br>`vllm/_aiter_ops.py` | Adds ROCm WMMA attention kernels for paged prefill and split-K decode shapes | Treat split attention support kernels on AMD as an in-flight vLLM attention-kernel family before calling them novel. |
+| PR `#41263` DeepSeek-V4 fused norm / router low-latency path | `DSV4`<br>`fuse norm router`<br>`low latency`<br>`router` | `PR #41263`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/model_executor/layers/fused_moe/router` | Targets DeepSeek-V4 decode latency by fusing norm / router-adjacent work and low-latency model paths | Treat DSV4 norm-router ladders as a concrete in-flight upstream family. |
+| PR `#41428` DSV4 fused indexer Q quant kernel | `DSV4`<br>`fused Indexer Q quant`<br>`indexer q`<br>`fp4` | `PR #41428`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/csrc` | Improves the fused DeepSeek-V4 indexer Q quant kernel instead of materializing Q then quantizing separately | Treat DSV4 indexer-Q quant ladders as an in-flight upstream fused quant family. |
+| PR `#41255` DeepSeek-V4 Tile kernels / `head_compute_mix_kernel` | `head_compute_mix_kernel`<br>`Tile kernel`<br>`DSV4`<br>`MLA` | `PR #41255`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/csrc` | Adds DeepSeek-V4 Tile kernels that mix head compute work in one specialized kernel | Treat DSV4 MLA head-compute ladders as a known in-flight specialized-kernel family. |
+| PR `#41441` DSV4 all-reduce plus `mhc_post` fusion | `DSV4`<br>`AR+mhc_post`<br>`allreduce`<br>`mhc_post` | `PR #41441`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/compilation/passes/fusion` | Fuses or overlaps DSV4 all-reduce with post-MLA head-compute work | Treat all-reduce followed by `mhc_post` in DSV4 traces as an in-flight vLLM overlap/fusion family. |
+| PR `#41446` AMD GatedDeltaNet FLA prefill kernels | `GatedDeltaNet`<br>`FLA prefill`<br>`AMD`<br>`Qwen3-Next` | `PR #41446`<br>`vllm/model_executor/models/qwen3_next.py`<br>`vllm/v1/attention` | Optimizes GatedDeltaNet / FLA prefill kernels on AMD linear-attention models | Treat split GDN prefill kernels on ROCm as an in-flight upstream family. |
+| PR `#39748` dual-stream GDN input projection | `dual-stream`<br>`input projection`<br>`GatedDeltaNet`<br>`Qwen3.5` | `PR #39748`<br>`vllm/model_executor/models/qwen3_next.py` | Overlaps sibling input-projection branches for Qwen3 / Qwen3.5 GDN-style blocks | Treat serial GDN input projections as a known in-flight overlap opportunity. |
+| PRs `#41433` / `#41434` / `#41429` / `#40561` GPU/CPU sync removal | `GPU->CPU sync`<br>`cpu sync`<br>`item()`<br>`non_blocking` | `PR #41433`<br>`PR #41434`<br>`PR #41429`<br>`PR #40561` | Removes or gates accidental GPU-to-CPU synchronization points and adds sync-detection coverage | Treat CPU gaps next to small GPU kernels as an upstream vLLM sync-removal family before proposing a kernel-only fix. |
+| PR `#36823` vLLM IR `fused_add_rms_norm` overload | `vllm_ir`<br>`fused_add_rms_norm`<br>`maybe_inplace` | `PR #36823`<br>`vllm/compilation/passes/ir`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py` | Extends vLLM IR lowering so fused-add-RMSNorm variants remain visible to later compile-time fusions | Treat missing norm/quant compile fusion as potentially an IR-lowering visibility issue. |

 ## 17. Important toggles and caveats

@@ -306,8 +317,11 @@ contain the same implementation.
 | `PassConfig.fuse_norm_quant` | `vllm/config/compilation.py` | Enables vLLM's RMSNorm(+residual add) -> FP8 / FP4 quant compile-time fusion family. |
 | `PassConfig.fuse_act_quant` | `vllm/config/compilation.py` | Enables vLLM's `SiLU+Mul -> quant` fusion family, plus ROCm AITER variants where applicable. |
 | `PassConfig.fuse_attn_quant` | `vllm/config/compilation.py` | Enables attention-epilogue quant fusion; requires the right backend / graph visibility, so split kernels may still be expected. |
+| `PassConfig.fuse_mla_dual_rms_norm` | `vllm/config/compilation.py` | Enables the AITER-backed MLA paired-Q/KV RMSNorm fusion family on ROCm. |
 | `PassConfig.enable_qk_norm_rope_fusion` | `vllm/config/compilation.py` | Enables the compile-time QK RMSNorm + RoPE family on CUDA-like backends. |
 | `PassConfig.fuse_rope_kvcache` | `vllm/config/compilation.py` | Enables ROCm / AITER RoPE + KV-cache update fusion and is range-limited by token count. |
+| `PassConfig.fuse_minimax_qk_norm` | `vllm/config/compilation.py` | Enables the MiniMax decode Q/K allreduce-plus-RMSNorm compile-time fusion family. |
+| `PassConfig.fuse_act_padding` | `vllm/config/compilation.py` | Enables the ROCm AITER add-RMSNorm-plus-pad fusion family when AITER is available. |
 | `PassConfig.enable_sp` | `vllm/config/compilation.py` | Rewrites all-reduce into sequence-parallel staging; this is often a prerequisite for the overlap family, not just a pure fuse toggle. |
 | `PassConfig.fuse_gemm_comms` | `vllm/config/compilation.py` | Enables AsyncTP GEMM + collective overlap and auto-enables `enable_sp` when valid. |
 | `TRTLLM_ENABLE_PDL` | `vllm/csrc/dsv3_fused_a_gemm.cu`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | Enables programmatic dependent launch for the DSV3 specialized CUDA kernels, which can change launch grouping and trace shape for router / QKV-A paths. |
@@ -0,0 +1,66 @@
+# vLLM Torch Compile Fusion Patterns
+
+Refresh: `2026-05-01`.
+Source tree: vLLM `origin/main` at `7075df79b`.
+
+Use this file when the fuse-pattern table reports split kernels in a trace and
+you need to decide whether the shape is already covered by vLLM's
+`torch.compile` pattern matcher. Treat every row here as an upstream precedent
+before calling a similar SGLang opportunity novel.
+
+## Pass Registration
+
+vLLM registers these passes from
+`vllm/compilation/passes/pass_manager.py` through `PassConfig`.
+
+| Toggle | Pass | Target shape |
+| --- | --- | --- |
+| `enable_sp` | `SequenceParallelismPass` | all-reduce around residual/norm blocks becomes reduce-scatter, local work, and all-gather |
+| `fuse_gemm_comms` | `AsyncTPPass` | GEMM plus reduce-scatter / all-gather overlap through symmetric-memory collectives |
+| `fuse_allreduce_rms` | `AllReduceFusionPass` | all-reduce followed by RMSNorm, optional residual add, optional FP8 / NVFP4 quant |
+| `fuse_minimax_qk_norm` | `MiniMaxQKNormPass` | MiniMax Q/K all-reduce plus RMSNorm decode path |
+| `fuse_norm_quant` | `RMSNormQuantFusionPass` | RMSNorm or fused-add-RMSNorm followed by FP8 / FP4 quant |
+| `fuse_norm_quant` + AITER | `RocmAiterRMSNormQuantFusionPass` | ROCm AITER RMSNorm / fused-add-RMSNorm followed by AITER or vLLM quant |
+| `fuse_act_quant` | `ActivationQuantFusionPass` | SiLU-and-mul followed by FP8 / NVFP4 / block quant |
+| `fuse_act_quant` + AITER | `RocmAiterSiluMulFp8GroupQuantFusionPass` | AITER SiLU-and-mul followed by FP8 group quant |
+| `fuse_act_padding` + AITER | `RocmAiterTritonAddRMSNormPadFusionPass` | AITER fused-add-RMSNorm followed by padding into the next layout |
+| `fuse_mla_dual_rms_norm` + AITER | `MLADualRMSNormFusionPass` | MLA paired Q and KV RMSNorms become `fused_mla_dual_rms_norm` |
+| `fuse_rope_kvcache` | `RopeKVCacheFusionPass` | RoPE plus paged KV-cache update, after split cleanup passes |
+| `fuse_attn_quant` | `AttnQuantFusionPass` | attention output followed by FP8 / NVFP4 quant |
+| `fuse_attn_quant` | `MLAAttnQuantFusionPass` | MLA attention output followed by FP8 / NVFP4 / FP8 group quant |
+| `enable_qk_norm_rope_fusion` | `QKNormRoPEFusionPass` | Q/K RMSNorm plus RoPE on packed QKV tensors |
+
+## Pattern Inventory
+
+| Source file | Pattern classes | Trace clue | Replacement |
+| --- | --- | --- | --- |
+| `fusion/allreduce_rms_fusion.py` | `AllReduceRMSNormPattern`, `AllReduceFusedAddRMSNormPattern`, `AllReduceFusedRMSNormStaticQuantFP8Pattern`, `AllReduceFusedAddRMSNormStaticQuantFP8Pattern`, `AllReduceFusedRMSNormStaticQuantNVFP4Pattern`, `AllReduceFusedAddRMSNormStaticQuantNVFP4Pattern` | TP all-reduce directly before RMSNorm, residual-add RMSNorm, or quant | `flashinfer_trtllm_fused_allreduce_norm` with FlashInfer allreduce fusion pattern codes |
+| `fusion/rms_quant_fusion.py` | `RMSNormStaticQuantPattern`, `FusedAddRMSNormStaticQuantPattern`, `RMSNormDynamicQuantPattern`, `FusedAddRMSNormDynamicQuantPattern`, `RMSNormGroupQuantPattern`, `FusedAddRMSNormGroupQuantPattern` | RMSNorm or fused-add-RMSNorm followed by static FP8, dynamic per-token FP8, FP8 group quant, or NVFP4 quant | `_C.rms_norm_*_quant`, `_C.fused_add_rms_norm_*_quant`, or per-block quant custom op |
+| `fusion/rocm_aiter_fusion.py` | `AiterRMSNormDynamicQuantPattern`, `AiterFusedAddRMSNormDynamicQuantPattern`, `AiterRMSFp8GroupQuantPattern`, `AiterFusedAddRMSFp8GroupQuantPattern` | AITER RMSNorm/fused-add-RMSNorm followed by AITER or vLLM FP8 quant | AITER fused RMSNorm-quant custom ops |
+| `fusion/act_quant_fusion.py` | `SiluMulFp8StaticQuantPattern`, `SiluMulNvfp4QuantPattern`, `SiluMulBlockQuantPattern` | SiLU-and-mul activation output immediately quantized | fused activation-plus-quant custom op |
+| `fusion/rocm_aiter_fusion.py` | `AiterSiluMulFp8GroupQuantPattern` | AITER SiLU-and-mul followed by FP8 group quant | AITER `act_mul_fused_fp8_group_quant` |
+| `fusion/rocm_aiter_fusion.py` | `AddAiterRMSNormPadPattern` | AITER fused-add-RMSNorm output padded before the next op | AITER add-RMSNorm-pad op |
+| `fusion/rocm_aiter_fusion.py` | `MLADualRMSNormPattern` | MLA Q branch and KV branch each run RMSNorm | `torch.ops.vllm.fused_mla_dual_rms_norm` backed by AITER fused QK RMSNorm |
+| `fusion/qk_norm_rope_fusion.py` | `QkNormRopePattern` | Q/K RMSNorm, split/getitem reshapes, then RoPE | `_C.fused_qk_norm_rope` |
+| `fusion/rope_kvcache_fusion.py` | `RopeReshapeKVCachePattern` | RoPE output followed by reshape/cache update | `vllm.fused_rope_and_unified_kv_cache_update` |
+| `fusion/attn_quant_fusion.py` | `AttnFp8StaticQuantPattern`, `AttnNvfp4QuantPattern` | attention output followed by FP8 static quant or NVFP4 quant | backend attention op with fused output quant when supported |
+| `fusion/mla_attn_quant_fusion.py` | `MLAAttnFp8StaticQuantPattern`, `MLAAttnNvfp4QuantPattern`, `MLAAttnFp8GroupQuantPattern` | MLA attention output followed by static FP8, NVFP4, or FP8 group quant | MLA attention op with fused output quant when supported |
+| `fusion/minimax_qk_norm_fusion.py` | `MiniMaxQKNormPattern` | MiniMax `forward_qk`: Q/K variance all-reduce divided by TP world size, then RMS apply | `vllm.minimax_qk_norm_fused` / Lamport fused kernel |
+| `fusion/sequence_parallelism.py` | `FirstAllReduceRMSNormPattern`, `MiddleAllReduceRMSNormPattern`, `FirstAllReduceRMSNormStaticFP8Pattern`, `MiddleAllReduceRMSNormStaticFP8Pattern` | all-reduce plus norm block in a full-graph TP model | sequence-parallel reduce-scatter, local norm, all-gather staging |
+| `fusion/collective_fusion.py` | `GEMMReduceScatterPattern`, `AllGatherGEMMPattern`, `ScaledMMReduceScatterPattern`, `AllGatherScaledMMPattern`, `CutlassScaledMMReduceScatterPattern`, `AllGatherCutlassScaledMMPattern`, `FlashInferBMMFP8ReduceScatterPattern`, `FlashInferAllGatherBMMFP8Pattern` | matmul / scaled-mm / FlashInfer BMM adjacent to TP collectives | symmetric-memory fused matmul+reduce-scatter or all-gather+matmul |
+
+## Triage Rules
+
+- If the trace shows split norm/add/quant, compare first against
+  `RMSNormQuantFusionPass`, AITER variants, and `AllReduceFusionPass`.
+- If the trace shows attention output followed by quant kernels, compare against
+  `AttnQuantFusionPass` or `MLAAttnQuantFusionPass`, not only handwritten
+  attention kernels.
+- If the trace shows Q/K norm followed by RoPE or cache update, compare both
+  `QKNormRoPEFusionPass` and `RopeKVCacheFusionPass`; they are separate passes.
+- If the trace is a TP decode trace with visible collectives, check whether
+  `enable_sp` and `fuse_gemm_comms` would transform the same region into
+  sequence-parallel or AsyncTP overlap.
+- A missing vLLM compile fusion may be intentional when the graph range, backend
+  support check, dtype, token count, or AITER / FlashInfer availability does not
+  satisfy the pass-specific guard.
@@ -11,6 +11,12 @@ from typing import Dict, List, Optional, Sequence, Tuple
 import triage_kernel_helpers as kernel_helpers
 import triage_overlap_helpers as overlap_helpers
 from profile_common import (
+    DEFAULT_DECODE_INPUT_LEN,
+    DEFAULT_DECODE_OUTPUT_LEN,
+    DEFAULT_PREFILL_INPUT_LEN,
+    DEFAULT_PREFILL_OUTPUT_LEN,
+    DEFAULT_WARMUP_STEPS,
+    PROFILE_WORKLOAD_CHOICES,
    discover_trace_targets,
    framework_display_name,
    load_server_args,
@@ -57,8 +63,8 @@ def build_triage_parser() -> argparse.ArgumentParser:
        default=None,
        help=(
            "Running server URL for single-trace triage. SGLang supports direct "
-            "capture through its profiler HTTP API. vLLM and TensorRT-LLM require "
-            "a server-side torch-profiler output path exposed via --output-dir."
+            "capture via sglang.profiler. vLLM and TensorRT-LLM require a server-side "
+            "torch-profiler output path exposed via --output-dir."
        ),
    )
    parser.add_argument(
@@ -132,7 +138,13 @@ def build_triage_parser() -> argparse.ArgumentParser:
        "--num-steps",
        type=int,
        default=5,
-        help="Profiler steps when generating traces from URLs.",
+        help="Active profiler steps when generating traces from URLs.",
+    )
+    parser.add_argument(
+        "--warmup-steps",
+        type=int,
+        default=DEFAULT_WARMUP_STEPS,
+        help="Warmup steps to run before arming the profiler for URL capture.",
    )
    parser.add_argument(
        "--profile-by-stage", action=argparse.BooleanOptionalAction, default=True
@@ -151,11 +163,45 @@ def build_triage_parser() -> argparse.ArgumentParser:
    )
    parser.add_argument("--probe-max-new-tokens", type=int, default=None)
    parser.add_argument("--probe-delay", type=float, default=0.5)
+    parser.add_argument(
+        "--profile-workload",
+        choices=PROFILE_WORKLOAD_CHOICES,
+        default="both",
+        help=(
+            "Live-capture workload shape. Default 'both' captures separate "
+            "prefill and decode profiles instead of one mixed request. Use "
+            "'legacy' to keep the old --probe-prompt behavior."
+        ),
+    )
+    parser.add_argument(
+        "--prefill-input-len",
+        type=int,
+        default=DEFAULT_PREFILL_INPUT_LEN,
+        help="Synthetic input length for the prefill profile workload.",
+    )
+    parser.add_argument(
+        "--prefill-output-len",
+        type=int,
+        default=DEFAULT_PREFILL_OUTPUT_LEN,
+        help="Output length for the prefill profile workload.",
+    )
+    parser.add_argument(
+        "--decode-input-len",
+        type=int,
+        default=DEFAULT_DECODE_INPUT_LEN,
+        help="Synthetic input length for the decode profile workload.",
+    )
+    parser.add_argument(
+        "--decode-output-len",
+        type=int,
+        default=DEFAULT_DECODE_OUTPUT_LEN,
+        help="Output length for the decode profile workload.",
+    )
    parser.add_argument(
        "--start-step",
        type=int,
        default=None,
-        help="SGLang-only profiler start step when generating traces from URLs.",
+        help="Pass through to sglang.profiler when generating traces from URLs.",
    )
    parser.add_argument(
        "--pid-substring",
@@ -239,9 +285,15 @@ def resolve_profile_targets(
            probe_prompt=args.probe_prompt,
            probe_max_new_tokens=args.probe_max_new_tokens,
            probe_delay=args.probe_delay,
+            warmup_steps=args.warmup_steps,
            start_step=args.start_step,
            framework=framework,
            framework_hint_path=output_dir,
+            profile_workload=args.profile_workload,
+            prefill_input_len=args.prefill_input_len,
+            prefill_output_len=args.prefill_output_len,
+            decode_input_len=args.decode_input_len,
+            decode_output_len=args.decode_output_len,
        )
        traces, server_args = discover_trace_targets(target_dir, all_traces=False)
        resolved_framework = resolve_framework(
@@ -15,7 +15,7 @@ from urllib import request
 from profile_common import extract_openai_chat_text

 DEFAULT_PROMPTS = [
-    "用一句中文介绍上海。",
+    "Introduce Shanghai in one short sentence.",
    "What is 2+2? Answer briefly.",
    "Write one short haiku about GPUs.",
 ]
@@ -5,10 +5,12 @@ from __future__ import annotations
 import gzip
 import json
 import re
+import shutil
 import sys
 import tempfile
 import time
 from collections import Counter, defaultdict
+from dataclasses import dataclass
 from functools import lru_cache
 from pathlib import Path
 from typing import Callable, Dict, Iterable, List, Optional, Sequence, Tuple
@@ -42,6 +44,21 @@ TRACE_METADATA_NAMES = {
 }
 NON_KERNEL_TRACE_CATEGORIES = ("python_function", "cpu_op", "trace")
 PYTHON_SCOPE_NAME_PREFIXES = ("python/", "nn.module:")
+PROFILE_WORKLOAD_CHOICES = ("legacy", "prefill", "decode", "both")
+DEFAULT_PREFILL_INPUT_LEN = 4090
+DEFAULT_PREFILL_OUTPUT_LEN = 1
+DEFAULT_DECODE_INPUT_LEN = 1
+DEFAULT_DECODE_OUTPUT_LEN = 2048
+DEFAULT_WARMUP_STEPS = 10
+
+
+@dataclass(frozen=True)
+class ProbePlan:
+    prompt: str
+    capture_max_new_tokens: int
+    capture_requests: int
+    warmup_max_new_tokens: int
+    warmup_requests: int


@lru_cache(maxsize=65536)
@@ -416,10 +433,16 @@ def resolve_framework(


 def parse_stage(path: Path) -> str:
-    name = path.name.lower()
-    if "-extend" in name or "-prefill" in name:
+    parts = [part.lower() for part in path.parts[-6:]]
+    name = " ".join(parts)
+    segment_path = "/" + "/".join(parts) + "/"
+    if any(marker in name for marker in ("-extend", "-prefill", "_extend", "_prefill")):
        return "extend"
-    if "-decode" in name:
+    if any(f"/{segment}/" in segment_path for segment in ("extend", "prefill")):
+        return "extend"
+    if any(marker in name for marker in ("-decode", "_decode")):
+        return "decode"
+    if "/decode/" in segment_path:
        return "decode"
    return "all"

@@ -514,8 +537,19 @@ def discover_trace_targets(
    if path.is_file():
        return [path], load_server_args(path)

-    trace_dir = newest_trace_dir(path)
-    traces = discover_trace_files(trace_dir, recursive=False)
+    direct_traces = discover_trace_files(path, recursive=False)
+    recursive_traces = discover_trace_files(path, recursive=True)
+    recursive_stages = {parse_stage(trace) for trace in recursive_traces}
+    if (
+        not direct_traces
+        and recursive_traces
+        and any(stage != "all" for stage in recursive_stages)
+    ):
+        traces = recursive_traces
+        trace_dir = path
+    else:
+        trace_dir = newest_trace_dir(path)
+        traces = discover_trace_files(trace_dir, recursive=False)
    if not traces:
        raise FileNotFoundError(f"No trace files found under {trace_dir}")

@@ -606,6 +640,109 @@ def send_probe_request(
        )


+def unique_probe_prompt(prompt: str, probe_index: int) -> str:
+    marker = f"profile_probe_{max(0, int(probe_index))}"
+    parts = prompt.split(maxsplit=1)
+    suffix = parts[1] if len(parts) == 2 else prompt
+    return f"{marker} {suffix}".strip()
+
+
+def send_probe_requests(
+    *,
+    url: str,
+    prompt: str,
+    max_new_tokens: int,
+    request_count: int,
+    framework: str,
+    model: Optional[str] = None,
+    sampling_seed_offset: int = 0,
+) -> None:
+    request_count = max(0, int(request_count))
+    seed_offset = max(0, int(sampling_seed_offset))
+    for request_idx in range(request_count):
+        probe_index = seed_offset + request_idx
+        send_probe_request(
+            url=url,
+            prompt=unique_probe_prompt(prompt, probe_index),
+            max_new_tokens=max_new_tokens,
+            sampling_seed=probe_index,
+            framework=framework,
+            model=model,
+        )
+
+
+def synthetic_prompt(input_len: int) -> str:
+    token_count = max(1, int(input_len))
+    return " ".join(["profile"] * token_count)
+
+
+def workload_probe(
+    stage: str,
+    *,
+    prefill_input_len: int,
+    prefill_output_len: int,
+    decode_input_len: int,
+    decode_output_len: int,
+) -> Tuple[str, int]:
+    if stage == "prefill":
+        return synthetic_prompt(prefill_input_len), max(1, int(prefill_output_len))
+    if stage == "decode":
+        return synthetic_prompt(decode_input_len), max(1, int(decode_output_len))
+    raise ValueError(f"unknown profile workload stage: {stage}")
+
+
+def build_probe_plan(
+    stage: str,
+    *,
+    prompt: str,
+    max_new_tokens: int,
+    num_steps: int,
+    probe_requests: int,
+    warmup_steps: int,
+) -> ProbePlan:
+    active_steps = max(1, int(num_steps))
+    requested_probes = max(1, int(probe_requests))
+    warmup_steps = max(0, int(warmup_steps))
+    max_new_tokens = max(1, int(max_new_tokens))
+
+    if stage == "prefill":
+        return ProbePlan(
+            prompt=prompt,
+            capture_max_new_tokens=max_new_tokens,
+            capture_requests=max(requested_probes, active_steps),
+            warmup_max_new_tokens=max_new_tokens,
+            warmup_requests=warmup_steps,
+        )
+    if stage == "decode":
+        return ProbePlan(
+            prompt=prompt,
+            capture_max_new_tokens=max_new_tokens,
+            capture_requests=requested_probes,
+            warmup_max_new_tokens=max(1, warmup_steps),
+            warmup_requests=1 if warmup_steps else 0,
+        )
+    return ProbePlan(
+        prompt=prompt,
+        capture_max_new_tokens=max_new_tokens,
+        capture_requests=requested_probes,
+        warmup_max_new_tokens=max_new_tokens,
+        warmup_requests=warmup_steps,
+    )
+
+
+def expand_profile_workload(profile_workload: str) -> List[str]:
+    workload = normalize_text(profile_workload).lower()
+    if workload not in PROFILE_WORKLOAD_CHOICES:
+        raise ValueError(
+            f"--profile-workload must be one of {', '.join(PROFILE_WORKLOAD_CHOICES)}"
+        )
+    if workload == "both":
+        return ["prefill", "decode"]
+    if workload == "legacy":
+        return ["legacy"]
+    return [workload]
+
+
 def discover_openai_model(url: str) -> str:
    payload = try_get_json(url.rstrip("/") + "/v1/models", timeout=60.0)
    if not isinstance(payload, dict):
@@ -690,35 +827,50 @@ def run_remote_profiler(
    url: str,
    output_dir: Optional[str],
    framework: str,
-    probe_requests: int,
-    probe_prompt: str,
-    probe_max_new_tokens: Optional[int],
+    probe_plan: ProbePlan,
    probe_delay: float,
-    num_steps: int,
+    stage: Optional[str] = None,
 ) -> Path:
    framework = canonicalize_framework(framework)
    output_path = ensure_remote_profiler_output_path(output_dir, framework)
+    if stage and output_path.is_file():
+        raise ValueError(
+            "--profile-workload both requires a directory output path for "
+            f"{framework_display_name(framework)} so each stage trace can be labeled."
+        )
+    before_traces = (
+        set(discover_trace_files(output_path, recursive=True))
+        if output_path.exists()
+        else set()
+    )
+    model = discover_openai_model(url) if framework in {"vllm", "trtllm"} else None
+    if probe_plan.warmup_requests > 0:
+        send_probe_requests(
+            url=url,
+            prompt=probe_plan.prompt,
+            max_new_tokens=probe_plan.warmup_max_new_tokens,
+            request_count=probe_plan.warmup_requests,
+            framework=framework,
+            model=model,
+        )
+
    start_remote_profiler(url, framework)
    stop_error: Optional[BaseException] = None
    try:
-        if probe_requests > 0:
-            # Some profiler endpoints need a brief setup window after
+        if probe_plan.capture_requests > 0:
+            # `sglang.profiler` performs its own startup work before it reaches
            # POST /start_profile. A very short delay can send probes too early
            # and miss the profiling window entirely.
            time.sleep(max(5.0, probe_delay))
-            effective_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
-            model = (
-                discover_openai_model(url) if framework in {"vllm", "trtllm"} else None
+            send_probe_requests(
+                url=url,
+                prompt=probe_plan.prompt,
+                max_new_tokens=probe_plan.capture_max_new_tokens,
+                request_count=probe_plan.capture_requests,
+                framework=framework,
+                model=model,
+                sampling_seed_offset=probe_plan.warmup_requests,
            )
-            for request_idx in range(probe_requests):
-                send_probe_request(
-                    url=url,
-                    prompt=probe_prompt,
-                    max_new_tokens=effective_max_new_tokens,
-                    sampling_seed=request_idx,
-                    framework=framework,
-                    model=model,
-                )
    finally:
        try:
            stop_remote_profiler(url, framework)
@@ -726,7 +878,22 @@ def run_remote_profiler(
            stop_error = exc
    if stop_error is not None:
        raise stop_error
-    return wait_for_profiler_artifact(output_path)
+    artifact = wait_for_profiler_artifact(output_path)
+    if stage and output_path.is_dir():
+        after_traces = set(discover_trace_files(output_path, recursive=True))
+        new_traces = sorted(after_traces - before_traces, key=lambda item: item.name)
+        if new_traces:
+            stage_dir = output_path / stage
+            stage_dir.mkdir(parents=True, exist_ok=True)
+            for trace in new_traces:
+                if stage_dir in trace.parents:
+                    continue
+                target = stage_dir / trace.name
+                if target.exists():
+                    target = stage_dir / f"{time.time_ns()}-{trace.name}"
+                shutil.move(str(trace), str(target))
+            return stage_dir
+    return artifact


 def run_sglang_profiler(
@@ -736,9 +903,7 @@ def run_sglang_profiler(
    profile_by_stage: bool,
    merge_profiles: bool,
    profile_prefix: Optional[str],
-    probe_requests: int,
-    probe_prompt: str,
-    probe_max_new_tokens: Optional[int],
+    probe_plan: ProbePlan,
    probe_delay: float,
    start_step: Optional[int] = None,
 ) -> Path:
@@ -765,6 +930,15 @@ def run_sglang_profiler(
    if start_step is not None:
        payload["start_step"] = str(start_step)

+    if probe_plan.warmup_requests > 0:
+        send_probe_requests(
+            url=url,
+            prompt=probe_plan.prompt,
+            max_new_tokens=probe_plan.warmup_max_new_tokens,
+            request_count=probe_plan.warmup_requests,
+            framework="sglang",
+        )
+
    req = request.Request(
        url.rstrip("/") + "/start_profile",
        data=json.dumps(payload).encode("utf-8"),
@@ -773,17 +947,20 @@ def run_sglang_profiler(
    with request.urlopen(req, timeout=300.0):
        pass

-    if probe_requests > 0:
+    if probe_plan.capture_requests > 0:
        time.sleep(max(0.0, probe_delay))
-        effective_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
-        for request_idx in range(probe_requests):
-            send_probe_request(
-                url=url,
-                prompt=probe_prompt,
-                max_new_tokens=effective_max_new_tokens,
-                sampling_seed=request_idx,
-                framework="sglang",
-            )
+        send_probe_requests(
+            url=url,
+            prompt=probe_plan.prompt,
+            max_new_tokens=probe_plan.capture_max_new_tokens,
+            request_count=probe_plan.capture_requests,
+            framework="sglang",
+            sampling_seed_offset=probe_plan.warmup_requests,
+        )
+        try:
+            stop_remote_profiler(url, "sglang")
+        except RuntimeError:
+            pass

    return wait_for_profiler_artifact(output_path, timeout_s=180.0)

@@ -799,9 +976,15 @@ def run_profiler(
    probe_prompt: str,
    probe_max_new_tokens: Optional[int],
    probe_delay: float,
+    warmup_steps: int = DEFAULT_WARMUP_STEPS,
    start_step: Optional[int] = None,
    framework: str = "auto",
    framework_hint_path: Optional[str] = None,
+    profile_workload: str = "both",
+    prefill_input_len: int = DEFAULT_PREFILL_INPUT_LEN,
+    prefill_output_len: int = DEFAULT_PREFILL_OUTPUT_LEN,
+    decode_input_len: int = DEFAULT_DECODE_INPUT_LEN,
+    decode_output_len: int = DEFAULT_DECODE_OUTPUT_LEN,
 ) -> Path:
    resolved_framework = resolve_framework(
        framework,
@@ -813,6 +996,58 @@ def run_profiler(
        ),
    )
    if resolved_framework == "sglang":
+        stages = expand_profile_workload(profile_workload)
+        if stages != ["legacy"]:
+            output_root = (
+                Path(output_dir).expanduser().resolve()
+                if output_dir
+                else Path(tempfile.mkdtemp(prefix="sglang-torch-profile-"))
+            )
+            output_root.mkdir(parents=True, exist_ok=True)
+            for stage in stages:
+                prompt, max_new_tokens = workload_probe(
+                    stage,
+                    prefill_input_len=prefill_input_len,
+                    prefill_output_len=prefill_output_len,
+                    decode_input_len=decode_input_len,
+                    decode_output_len=decode_output_len,
+                )
+                probe_plan = build_probe_plan(
+                    stage,
+                    prompt=prompt,
+                    max_new_tokens=max_new_tokens,
+                    num_steps=num_steps,
+                    probe_requests=probe_requests,
+                    warmup_steps=warmup_steps,
+                )
+                # SGLang increments `forward_ct` before checking whether the
+                # profiler reached its target. Ask for one extra step so the
+                # requested stage forward is captured instead of stopping just
+                # before it runs.
+                stage_num_steps = max(1, int(num_steps)) + 1
+                run_sglang_profiler(
+                    url=url,
+                    output_dir=str(output_root / stage),
+                    num_steps=stage_num_steps,
+                    profile_by_stage=False,
+                    merge_profiles=merge_profiles,
+                    profile_prefix=(
+                        f"{profile_prefix}-{stage}" if profile_prefix else stage
+                    ),
+                    probe_plan=probe_plan,
+                    probe_delay=probe_delay,
+                    start_step=start_step,
+                )
+            return output_root
+        legacy_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
+        legacy_plan = build_probe_plan(
+            "legacy",
+            prompt=probe_prompt,
+            max_new_tokens=legacy_max_new_tokens,
+            num_steps=num_steps,
+            probe_requests=probe_requests,
+            warmup_steps=warmup_steps,
+        )
        return run_sglang_profiler(
            url=url,
            output_dir=output_dir,
@@ -820,9 +1055,7 @@ def run_profiler(
            profile_by_stage=profile_by_stage,
            merge_profiles=merge_profiles,
            profile_prefix=profile_prefix,
-            probe_requests=probe_requests,
-            probe_prompt=probe_prompt,
-            probe_max_new_tokens=probe_max_new_tokens,
+            probe_plan=legacy_plan,
            probe_delay=probe_delay,
            start_step=start_step,
        )
@@ -844,16 +1077,48 @@ def run_profiler(
            "--profile-prefix on the HTTP profiler control path.",
            file=sys.stderr,
        )
-    return run_remote_profiler(
-        url=url,
-        output_dir=output_dir,
-        framework=resolved_framework,
-        probe_requests=probe_requests,
-        probe_prompt=probe_prompt,
-        probe_max_new_tokens=probe_max_new_tokens,
-        probe_delay=probe_delay,
-        num_steps=num_steps,
-    )
+    stages = expand_profile_workload(profile_workload)
+    if stages == ["legacy"]:
+        legacy_max_new_tokens = probe_max_new_tokens or max(64, num_steps * 8)
+        return run_remote_profiler(
+            url=url,
+            output_dir=output_dir,
+            framework=resolved_framework,
+            probe_plan=build_probe_plan(
+                "legacy",
+                prompt=probe_prompt,
+                max_new_tokens=legacy_max_new_tokens,
+                num_steps=num_steps,
+                probe_requests=probe_requests,
+                warmup_steps=warmup_steps,
+            ),
+            probe_delay=probe_delay,
+        )
+    output_root = ensure_remote_profiler_output_path(output_dir, resolved_framework)
+    for stage in stages:
+        prompt, max_new_tokens = workload_probe(
+            stage,
+            prefill_input_len=prefill_input_len,
+            prefill_output_len=prefill_output_len,
+            decode_input_len=decode_input_len,
+            decode_output_len=decode_output_len,
+        )
+        run_remote_profiler(
+            url=url,
+            output_dir=str(output_root),
+            framework=resolved_framework,
+            probe_plan=build_probe_plan(
+                stage,
+                prompt=prompt,
+                max_new_tokens=max_new_tokens,
+                num_steps=num_steps,
+                probe_requests=probe_requests,
+                warmup_steps=warmup_steps,
+            ),
+            probe_delay=probe_delay,
+            stage=stage,
+        )
+    return output_root


 def select_heaviest_pid(
@@ -0,0 +1,274 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_llm_single_model_matrix_host.sh \
+    --model-id gpt_oss_20b \
+    --model openai/gpt-oss-20b \
+    --root /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix \
+    --gpus 2,3,4,5 \
+    --sglang-port 30098 \
+    --vllm-formal-port 31098 \
+    --vllm-mapping-port 31099 \
+    --trt-formal-prefill-port 32098 \
+    --trt-formal-decode-port 32099 \
+    --trt-mapping-prefill-port 32198 \
+    --trt-mapping-decode-port 32199
+
+This script is intended to run on the H100 host. It:
+1. captures SGLang live profiling and writes `analysis_sglang.txt`
+2. captures vLLM formal + eager mapping traces and writes `analysis_vllm.txt`
+3. captures TensorRT-LLM formal + graph-off mapping traces and writes `analysis_trtllm.txt`
+4. stores one benchmark JSON per framework under the model run directory
+
+Default profiler workloads are stage-separated:
+  prefill: input 4090, output 1
+  decode:  input 1, output 2048
+
+Environment:
+  Export `HF_TOKEN` and `HUGGINGFACE_HUB_TOKEN` before running.
+EOF
+}
+
+MODEL_ID=""
+MODEL=""
+ROOT=""
+GPUS=""
+TP_SIZE=""
+SGLANG_PORT=""
+VLLM_FORMAL_PORT=""
+VLLM_MAPPING_PORT=""
+TRT_FORMAL_PREFILL_PORT=""
+TRT_FORMAL_DECODE_PORT=""
+TRT_MAPPING_PREFILL_PORT=""
+TRT_MAPPING_DECODE_PORT=""
+SGLANG_MEM_FRACTION="0.85"
+MAX_MODEL_LEN="4096"
+KV_FRACTION="0.85"
+SGLANG_SERVER_EXTRA=""
+PROFILE_WORKLOAD="both"
+PREFILL_INPUT_LEN=4090
+PREFILL_OUTPUT_LEN=1
+DECODE_INPUT_LEN=1
+DECODE_OUTPUT_LEN=2048
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TRT_IMAGE="nvcr.io/nvidia/tensorrt-llm/release:latest"
+TRT_OVERRIDE_ROOT="/data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm"
+TRT_OVERRIDE_SOURCE="$TRT_OVERRIDE_ROOT/py_executor.original.py"
+TRT_OVERRIDE_PATH="$TRT_OVERRIDE_ROOT/py_executor_with_stack.py"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model-id) MODEL_ID="$2"; shift 2 ;;
+    --model) MODEL="$2"; shift 2 ;;
+    --root) ROOT="$2"; shift 2 ;;
+    --gpus) GPUS="$2"; shift 2 ;;
+    --tp-size) TP_SIZE="$2"; shift 2 ;;
+    --sglang-port) SGLANG_PORT="$2"; shift 2 ;;
+    --vllm-formal-port) VLLM_FORMAL_PORT="$2"; shift 2 ;;
+    --vllm-mapping-port) VLLM_MAPPING_PORT="$2"; shift 2 ;;
+    --trt-formal-prefill-port) TRT_FORMAL_PREFILL_PORT="$2"; shift 2 ;;
+    --trt-formal-decode-port) TRT_FORMAL_DECODE_PORT="$2"; shift 2 ;;
+    --trt-mapping-prefill-port) TRT_MAPPING_PREFILL_PORT="$2"; shift 2 ;;
+    --trt-mapping-decode-port) TRT_MAPPING_DECODE_PORT="$2"; shift 2 ;;
+    --sglang-mem-fraction) SGLANG_MEM_FRACTION="$2"; shift 2 ;;
+    --sglang-server-extra) SGLANG_SERVER_EXTRA="$2"; shift 2 ;;
+    --max-model-len) MAX_MODEL_LEN="$2"; shift 2 ;;
+    --kv-fraction) KV_FRACTION="$2"; shift 2 ;;
+    --profile-workload) PROFILE_WORKLOAD="$2"; shift 2 ;;
+    --prefill-input-len) PREFILL_INPUT_LEN="$2"; shift 2 ;;
+    --prefill-output-len) PREFILL_OUTPUT_LEN="$2"; shift 2 ;;
+    --decode-input-len) DECODE_INPUT_LEN="$2"; shift 2 ;;
+    --decode-output-len) DECODE_OUTPUT_LEN="$2"; shift 2 ;;
+    --help|-h) usage; exit 0 ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "${HF_TOKEN:-}" && -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  echo "Set HF_TOKEN or HUGGINGFACE_HUB_TOKEN before running." >&2
+  exit 2
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  HF_TOKEN="$HUGGINGFACE_HUB_TOKEN"
+fi
+if [[ -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  HUGGINGFACE_HUB_TOKEN="$HF_TOKEN"
+fi
+
+for value in \
+  MODEL_ID MODEL ROOT GPUS \
+  SGLANG_PORT VLLM_FORMAL_PORT VLLM_MAPPING_PORT \
+  TRT_FORMAL_PREFILL_PORT TRT_FORMAL_DECODE_PORT \
+  TRT_MAPPING_PREFILL_PORT TRT_MAPPING_DECODE_PORT; do
+  if [[ -z "${!value}" ]]; then
+    echo "Missing required argument: $value" >&2
+    usage >&2
+    exit 2
+  fi
+done
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tp-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+
+MODEL_ROOT="$ROOT/$MODEL_ID"
+SGLANG_ANALYSIS="$MODEL_ROOT/analysis_sglang.txt"
+VLLM_FORMAL_DIR="$MODEL_ROOT/vllm_formal"
+VLLM_MAPPING_DIR="$MODEL_ROOT/vllm_mapping"
+VLLM_ANALYSIS="$MODEL_ROOT/analysis_vllm.txt"
+TRT_FORMAL_DIR="$MODEL_ROOT/trtllm_formal"
+TRT_MAPPING_DIR="$MODEL_ROOT/trtllm_mapping"
+TRT_ANALYSIS="$MODEL_ROOT/analysis_trtllm.txt"
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$MODEL_ROOT'"
+
+if [[ ! -s "$TRT_OVERRIDE_SOURCE" ]]; then
+  echo "[bootstrap] TensorRT-LLM py_executor source snapshot"
+  docker exec sglang_bbuf bash -lc "mkdir -p '$TRT_OVERRIDE_ROOT'"
+  docker run --rm --entrypoint cat "$TRT_IMAGE" \
+    /usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py \
+    | docker exec -i sglang_bbuf bash -lc "cat > '$TRT_OVERRIDE_SOURCE'"
+fi
+echo "[bootstrap] TensorRT-LLM py_executor override with with_stack=True and rank0-only trace export"
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 make_trtllm_py_executor_override.py --source '$TRT_OVERRIDE_SOURCE' --output '$TRT_OVERRIDE_PATH'"
+
+sglang_args=(
+  --model "$MODEL"
+  --run-dir "$MODEL_ROOT"
+  --port "$SGLANG_PORT"
+  --gpus "$GPUS"
+  --tp-size "$TP_SIZE"
+  --mem-fraction "$SGLANG_MEM_FRACTION"
+  --profile-workload "$PROFILE_WORKLOAD"
+  --prefill-input-len "$PREFILL_INPUT_LEN"
+  --prefill-output-len "$PREFILL_OUTPUT_LEN"
+  --decode-input-len "$DECODE_INPUT_LEN"
+  --decode-output-len "$DECODE_OUTPUT_LEN"
+  --trust-remote-code
+)
+if [[ -n "$SGLANG_SERVER_EXTRA" ]]; then
+  sglang_args+=(--server-extra "$SGLANG_SERVER_EXTRA")
+fi
+
+echo "[1/6] SGLang server + live triage"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_sglang_torch_profile_host.sh" \
+  "${sglang_args[@]}"
+
+echo "[2/6] vLLM formal"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_vllm_torch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$VLLM_FORMAL_DIR" \
+  --port "$VLLM_FORMAL_PORT" \
+  --gpus "$GPUS" \
+  --tensor-parallel-size "$TP_SIZE" \
+  --max-model-len "$MAX_MODEL_LEN" \
+  --profile-workload "$PROFILE_WORKLOAD" \
+  --prefill-input-len "$PREFILL_INPUT_LEN" \
+  --prefill-output-len "$PREFILL_OUTPUT_LEN" \
+  --decode-input-len "$DECODE_INPUT_LEN" \
+  --decode-output-len "$DECODE_OUTPUT_LEN" \
+  --trust-remote-code
+
+echo "[3/6] vLLM mapping"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_vllm_torch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$VLLM_MAPPING_DIR" \
+  --port "$VLLM_MAPPING_PORT" \
+  --gpus "$GPUS" \
+  --tensor-parallel-size "$TP_SIZE" \
+  --profiler-active-iterations 2 \
+  --max-model-len "$MAX_MODEL_LEN" \
+  --profile-workload "$PROFILE_WORKLOAD" \
+  --prefill-input-len "$PREFILL_INPUT_LEN" \
+  --prefill-output-len "$PREFILL_OUTPUT_LEN" \
+  --decode-input-len "$DECODE_INPUT_LEN" \
+  --decode-output-len "$DECODE_OUTPUT_LEN" \
+  --trust-remote-code \
+  --enforce-eager
+
+echo "[4/6] vLLM mapping-formal analysis"
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 analyze_llm_torch_profile.py --framework vllm --mapping-input '$VLLM_MAPPING_DIR' --formal-input '$VLLM_FORMAL_DIR' > '$VLLM_ANALYSIS'"
+
+echo "[5/6] TensorRT-LLM formal + mapping captures"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_FORMAL_DIR" \
+  --stage prefill \
+  --port "$TRT_FORMAL_PREFILL_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$PREFILL_INPUT_LEN" \
+  --output-len "$PREFILL_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --trust-remote-code
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_FORMAL_DIR" \
+  --stage decode \
+  --port "$TRT_FORMAL_DECODE_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$DECODE_INPUT_LEN" \
+  --output-len "$DECODE_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --trust-remote-code
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_MAPPING_DIR" \
+  --stage prefill \
+  --port "$TRT_MAPPING_PREFILL_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$PREFILL_INPUT_LEN" \
+  --output-len "$PREFILL_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --disable-cudagraph \
+  --trust-remote-code
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_MAPPING_DIR" \
+  --stage decode \
+  --port "$TRT_MAPPING_DECODE_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$DECODE_INPUT_LEN" \
+  --output-len "$DECODE_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --disable-cudagraph \
+  --trust-remote-code
+
+echo "[6/6] TensorRT-LLM mapping-formal analysis"
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 analyze_llm_torch_profile.py --framework trtllm --mapping-input '$TRT_MAPPING_DIR' --formal-input '$TRT_FORMAL_DIR' > '$TRT_ANALYSIS'"
+
+echo "MODEL_ROOT=$MODEL_ROOT"
+echo "ANALYSIS_SGLANG=$SGLANG_ANALYSIS"
+echo "ANALYSIS_VLLM=$VLLM_ANALYSIS"
+echo "ANALYSIS_TRTLLM=$TRT_ANALYSIS"
@@ -0,0 +1,241 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_sglang_torch_profile_host.sh \
+    --model Qwen/Qwen3-8B \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_sglang \
+    --port 30088 \
+    --gpus 0
+
+  run_sglang_torch_profile_host.sh \
+    --model openai/gpt-oss-20b \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_sglang_4gpu \
+    --port 30088 \
+    --gpus 2,3,4,5 \
+    --tp-size 4
+
+Options:
+  --model TEXT                  Model id or local path for SGLang.
+  --run-dir PATH               Shared /data directory for logs and traces.
+  --port INT                   Server port.
+  --gpus TEXT                  CUDA_VISIBLE_DEVICES value, for example 0 or 2,3,4,5.
+  --gpu TEXT                   Alias for --gpus.
+  --tp-size INT                Tensor parallel size. Defaults to the visible GPU count.
+  --trust-remote-code          Pass --trust-remote-code.
+  --mem-fraction FLOAT         SGLang static memory fraction.
+  --request-max-tokens INT     Generation length for the probe request.
+  --prompt TEXT                Probe prompt.
+  --warmup-steps INT           Warmup steps before profiling. Defaults to 10.
+  --profile-workload TEXT      legacy|prefill|decode|both. Defaults to both.
+  --prefill-input-len INT      Synthetic prefill prompt length. Defaults to 4090.
+  --prefill-output-len INT     Synthetic prefill output length. Defaults to 1.
+  --decode-input-len INT       Synthetic decode prompt length. Defaults to 1.
+  --decode-output-len INT      Synthetic decode output length. Defaults to 2048.
+  --repo-dir PATH              SGLang repo path inside `sglang_bbuf`.
+  --server-extra TEXT          Extra args appended to launch_server.
+  --help                       Show this message.
+
+Notes:
+  - Run this on the H100 host. It uses `docker exec sglang_bbuf`.
+  - The server is launched first, then the profiler capture runs with
+    stage-separated prefill/decode workloads and `--profile-by-stage`.
+  - A small benchmark summary is written after profiling.
+EOF
+}
+
+MODEL=""
+RUN_DIR=""
+PORT=""
+GPUS=""
+TP_SIZE=""
+TRUST_REMOTE_CODE=0
+MEM_FRACTION=0.85
+REQUEST_MAX_TOKENS=12
+PROMPT="Explain the difference between CUDA graph mode and eager mode in two sentences."
+WARMUP_STEPS=10
+PROFILE_WORKLOAD="both"
+PREFILL_INPUT_LEN=4090
+PREFILL_OUTPUT_LEN=1
+DECODE_INPUT_LEN=1
+DECODE_OUTPUT_LEN=2048
+SGLANG_REPO_DIR="${SGLANG_REPO_DIR:-/data/bbuf/repos/sglang}"
+SERVER_EXTRA=""
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model)
+      MODEL="$2"
+      shift 2
+      ;;
+    --run-dir)
+      RUN_DIR="$2"
+      shift 2
+      ;;
+    --port)
+      PORT="$2"
+      shift 2
+      ;;
+    --gpu)
+      GPUS="$2"
+      shift 2
+      ;;
+    --gpus)
+      GPUS="$2"
+      shift 2
+      ;;
+    --tp-size)
+      TP_SIZE="$2"
+      shift 2
+      ;;
+    --trust-remote-code)
+      TRUST_REMOTE_CODE=1
+      shift
+      ;;
+    --mem-fraction)
+      MEM_FRACTION="$2"
+      shift 2
+      ;;
+    --request-max-tokens)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --prompt)
+      PROMPT="$2"
+      shift 2
+      ;;
+    --warmup-steps)
+      WARMUP_STEPS="$2"
+      shift 2
+      ;;
+    --profile-workload)
+      PROFILE_WORKLOAD="$2"
+      shift 2
+      ;;
+    --prefill-input-len)
+      PREFILL_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --prefill-output-len)
+      PREFILL_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-input-len)
+      DECODE_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-output-len)
+      DECODE_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --repo-dir)
+      SGLANG_REPO_DIR="$2"
+      shift 2
+      ;;
+    --server-extra)
+      SERVER_EXTRA="$2"
+      shift 2
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "$MODEL" || -z "$RUN_DIR" || -z "$PORT" || -z "$GPUS" ]]; then
+  usage >&2
+  exit 2
+fi
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tp-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+
+LOG_PATH="$RUN_DIR/sglang_server.log"
+ANALYSIS_PATH="$RUN_DIR/analysis_sglang.txt"
+PROFILE_ROOT="$RUN_DIR/sglang_profile_live"
+BENCHMARK_PATH="$RUN_DIR/benchmark_sglang.json"
+PID_PATH="$RUN_DIR/sglang_server.pid"
+LAUNCH_PATTERN="[s]glang.launch_server.*--port $PORT"
+SERVER_ARGS="python3 -m sglang.launch_server --model-path \"$MODEL\" --port \"$PORT\" --tp-size \"$TP_SIZE\" --mem-fraction-static \"$MEM_FRACTION\""
+
+if [[ "$TRUST_REMOTE_CODE" -eq 1 ]]; then
+  SERVER_ARGS="$SERVER_ARGS --trust-remote-code"
+fi
+if [[ -n "$SERVER_EXTRA" ]]; then
+  SERVER_ARGS="$SERVER_ARGS $SERVER_EXTRA"
+fi
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$RUN_DIR' '$PROFILE_ROOT'"
+docker exec sglang_bbuf bash -lc "pkill -f '$LAUNCH_PATTERN' >/dev/null 2>&1 || true"
+docker exec sglang_bbuf bash -lc "mkdir -p '$RUN_DIR' '$PROFILE_ROOT' && cd '$SGLANG_REPO_DIR' && rm -f '$PID_PATH' && (CUDA_VISIBLE_DEVICES=$GPUS PYTHONPATH=python nohup $SERVER_ARGS > '$LOG_PATH' 2>&1 < /dev/null & echo \$! > '$PID_PATH')"
+
+cleanup() {
+  docker exec sglang_bbuf bash -lc "pkill -f '$LAUNCH_PATTERN' >/dev/null 2>&1 || true" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+ready=0
+for _ in $(seq 1 180); do
+  if curl -sf "http://127.0.0.1:${PORT}/v1/models" >/dev/null; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$ready" -ne 1 ]]; then
+  echo "SGLang server did not become ready on port ${PORT}. Recent logs:" >&2
+  ssh_log=$(docker exec sglang_bbuf bash -lc "tail -n 120 '$LOG_PATH'" 2>/dev/null || true)
+  printf '%s\n' "$ssh_log" >&2
+  exit 1
+fi
+
+python3 - <<PY
+import json
+import urllib.request
+
+payload = {
+    "text": ${PROMPT@Q},
+    "sampling_params": {
+        "temperature": 0.0,
+        "max_new_tokens": int(${REQUEST_MAX_TOKENS@Q}),
+    },
+    "stream": False,
+}
+req = urllib.request.Request(
+    "http://127.0.0.1:${PORT}/generate",
+    data=json.dumps(payload).encode(),
+    headers={"Content-Type": "application/json"},
+)
+with urllib.request.urlopen(req, timeout=600) as resp:
+    body = json.loads(resp.read().decode())
+text = body.get("text", "")
+print(text[:400])
+PY
+
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 analyze_llm_torch_profile.py --framework sglang --url http://127.0.0.1:${PORT} --output-dir '$PROFILE_ROOT' --num-steps 5 --warmup-steps '$WARMUP_STEPS' --probe-requests 1 --profile-by-stage --profile-workload '$PROFILE_WORKLOAD' --prefill-input-len '$PREFILL_INPUT_LEN' --prefill-output-len '$PREFILL_OUTPUT_LEN' --decode-input-len '$DECODE_INPUT_LEN' --decode-output-len '$DECODE_OUTPUT_LEN' > '$ANALYSIS_PATH'"
+python3 "$SCRIPT_DIR/probe_llm_server.py" \
+  --framework sglang \
+  --url "http://127.0.0.1:${PORT}" \
+  | docker exec -i sglang_bbuf bash -lc "cat > '$BENCHMARK_PATH'" >/dev/null
+docker exec sglang_bbuf bash -lc "sed -n '1,240p' '$ANALYSIS_PATH'"
+echo "BENCHMARK_PATH=$BENCHMARK_PATH"
@@ -0,0 +1,407 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_trtllm_pytorch_profile_host.sh \
+    --model Qwen/Qwen3-8B \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example \
+    --stage prefill \
+    --port 32188 \
+    --gpus 0
+
+  run_trtllm_pytorch_profile_host.sh \
+    --model openai/gpt-oss-20b \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_4gpu \
+    --stage prefill \
+    --port 32188 \
+    --gpus 2,3,4,5 \
+    --tp-size 4
+
+Options:
+  --model TEXT                     Hugging Face model id.
+  --run-dir PATH                  Shared /data run directory for logs and traces.
+  --stage prefill|decode          Capture window. Prefill profiles 4090->1 by
+                                  default; decode profiles 1->2048 by default.
+  --port INT                      Host port for trtllm-serve.
+  --gpus TEXT                     CUDA_VISIBLE_DEVICES value, for example 0 or 2,3,4,5.
+  --gpu TEXT                      Alias for --gpus.
+  --tp-size INT                   Tensor parallel size. Defaults to the visible GPU count.
+  --image TEXT                    Container image.
+  --shared-root PATH              Shared validation root mounted into the container.
+  --hf-cache PATH                 Host Hugging Face cache path.
+  --override-py-executor PATH     Optional py_executor.py override path.
+  --disable-cudagraph             Generate/use a YAML override with cuda_graph_config: null.
+  --input-len INT                 Synthetic prompt length for this stage.
+                                  Defaults: prefill 4090, decode 1.
+  --request-max-tokens INT        Generation length for this stage.
+                                  Defaults: prefill 1, decode 2048.
+  --output-len INT                Alias for --request-max-tokens.
+  --prompt TEXT                   Probe prompt. Defaults to a synthetic prompt
+                                  sized by --input-len.
+  --warmup-steps INT              Warmup steps before the profiler window. Defaults to 10.
+  --active-steps INT              Active profiler steps to capture. Defaults to 5.
+  --max-seq-len INT               Serve max sequence length.
+  --kv-fraction FLOAT             KV cache free GPU memory fraction.
+  --container-name TEXT           Override container name.
+  --trust-remote-code             Pass --trust_remote_code to trtllm-serve.
+  --help                          Show this message.
+
+Environment:
+  HF_TOKEN or HUGGINGFACE_HUB_TOKEN must be set.
+
+Notes:
+  - Run this on the H100 host, not inside `sglang_bbuf`.
+  - It always pins TensorRT-LLM to `--backend pytorch`.
+  - The default image tag is floating; record the resolved TensorRT-LLM version
+    in the run manifest and pass --image for reproducible validation.
+  - Profiling uses `TLLM_PROFILE_START_STOP` and `TLLM_TORCH_PROFILE_TRACE`.
+  - For Python-location recovery, prefer a `py_executor.py` override with `with_stack=True`.
+  - A small benchmark summary is written after the trace is emitted.
+EOF
+}
+
+IMAGE="nvcr.io/nvidia/tensorrt-llm/release:latest"
+SHARED_ROOT="/data/bbuf/validate/unified_llm_profiler_skill"
+HF_CACHE="/data/.cache/huggingface"
+OVERRIDE_PY_EXECUTOR=""
+DISABLE_CUDAGRAPH=0
+REQUEST_MAX_TOKENS=""
+INPUT_LEN=""
+PROMPT=""
+WARMUP_STEPS=10
+ACTIVE_STEPS=5
+MAX_SEQ_LEN=4096
+KV_FRACTION=0.85
+CONTAINER_NAME=""
+TRUST_REMOTE_CODE=0
+TP_SIZE=""
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+MODEL=""
+RUN_DIR=""
+STAGE=""
+PORT=""
+GPUS=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model)
+      MODEL="$2"
+      shift 2
+      ;;
+    --run-dir)
+      RUN_DIR="$2"
+      shift 2
+      ;;
+    --stage)
+      STAGE="$2"
+      shift 2
+      ;;
+    --port)
+      PORT="$2"
+      shift 2
+      ;;
+    --gpu)
+      GPUS="$2"
+      shift 2
+      ;;
+    --gpus)
+      GPUS="$2"
+      shift 2
+      ;;
+    --tp-size)
+      TP_SIZE="$2"
+      shift 2
+      ;;
+    --image)
+      IMAGE="$2"
+      shift 2
+      ;;
+    --shared-root)
+      SHARED_ROOT="$2"
+      shift 2
+      ;;
+    --hf-cache)
+      HF_CACHE="$2"
+      shift 2
+      ;;
+    --override-py-executor)
+      OVERRIDE_PY_EXECUTOR="$2"
+      shift 2
+      ;;
+    --disable-cudagraph)
+      DISABLE_CUDAGRAPH=1
+      shift
+      ;;
+    --input-len)
+      INPUT_LEN="$2"
+      shift 2
+      ;;
+    --request-max-tokens)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --output-len)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --prompt)
+      PROMPT="$2"
+      shift 2
+      ;;
+    --warmup-steps)
+      WARMUP_STEPS="$2"
+      shift 2
+      ;;
+    --active-steps)
+      ACTIVE_STEPS="$2"
+      shift 2
+      ;;
+    --max-seq-len)
+      MAX_SEQ_LEN="$2"
+      shift 2
+      ;;
+    --kv-fraction)
+      KV_FRACTION="$2"
+      shift 2
+      ;;
+    --container-name)
+      CONTAINER_NAME="$2"
+      shift 2
+      ;;
+    --trust-remote-code)
+      TRUST_REMOTE_CODE=1
+      shift
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "${HF_TOKEN:-}" && -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  echo "Set HF_TOKEN or HUGGINGFACE_HUB_TOKEN before running." >&2
+  exit 2
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  HF_TOKEN="$HUGGINGFACE_HUB_TOKEN"
+fi
+if [[ -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  HUGGINGFACE_HUB_TOKEN="$HF_TOKEN"
+fi
+
+if [[ -z "$MODEL" || -z "$RUN_DIR" || -z "$STAGE" || -z "$PORT" || -z "$GPUS" ]]; then
+  usage >&2
+  exit 2
+fi
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tp-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+
+case "$STAGE" in
+  prefill)
+    TRACE_PATH="$RUN_DIR/trace-prefill.json"
+    LOG_PATH="$RUN_DIR/server-prefill.log"
+    BENCHMARK_PATH="$RUN_DIR/benchmark-prefill.json"
+    if [[ -z "$INPUT_LEN" ]]; then
+      INPUT_LEN=4090
+    fi
+    if [[ -z "$REQUEST_MAX_TOKENS" ]]; then
+      REQUEST_MAX_TOKENS=1
+    fi
+    ;;
+  decode)
+    TRACE_PATH="$RUN_DIR/trace-decode.json"
+    LOG_PATH="$RUN_DIR/server-decode.log"
+    BENCHMARK_PATH="$RUN_DIR/benchmark-decode.json"
+    if [[ -z "$INPUT_LEN" ]]; then
+      INPUT_LEN=1
+    fi
+    if [[ -z "$REQUEST_MAX_TOKENS" ]]; then
+      REQUEST_MAX_TOKENS=2048
+    fi
+    ;;
+  *)
+    echo "--stage must be prefill or decode." >&2
+    exit 2
+    ;;
+esac
+
+if (( WARMUP_STEPS < 0 || ACTIVE_STEPS < 1 )); then
+  echo "--warmup-steps must be >= 0 and --active-steps must be >= 1." >&2
+  exit 2
+fi
+
+case "$STAGE" in
+  prefill)
+    profile_start=$((WARMUP_STEPS + 1))
+    ;;
+  decode)
+    profile_start=$((WARMUP_STEPS + 2))
+    ;;
+esac
+profile_stop=$((profile_start + ACTIVE_STEPS - 1))
+PROFILE_START_STOP="${profile_start}-${profile_stop}"
+
+if [[ -z "$CONTAINER_NAME" ]]; then
+  model_slug="${MODEL##*/}"
+  model_slug="${model_slug//\//-}"
+  model_slug="${model_slug//./-}"
+  model_slug="${model_slug//_/-}"
+  model_slug="${model_slug// /-}"
+  gpu_slug="${GPUS//,/-}"
+  CONTAINER_NAME="trtllm-${model_slug}-${STAGE}-g${gpu_slug}-p${PORT}"
+fi
+
+EXTRA_LLM_OPTIONS=""
+if [[ "$DISABLE_CUDAGRAPH" -eq 1 ]]; then
+  EXTRA_CFG_PATH="$SHARED_ROOT/tmp/trt_no_cudagraph.yaml"
+  docker exec sglang_bbuf bash -lc "mkdir -p '$(dirname "$EXTRA_CFG_PATH")' && printf 'cuda_graph_config: null\n' > '$EXTRA_CFG_PATH'"
+  EXTRA_LLM_OPTIONS="--extra_llm_api_options $EXTRA_CFG_PATH"
+fi
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$RUN_DIR'"
+docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+
+docker_args=(
+  run -d --rm
+  --name "$CONTAINER_NAME"
+  --gpus all
+  --ipc=host
+  --network host
+  --entrypoint bash
+  -e "CUDA_VISIBLE_DEVICES=$GPUS"
+  -e "HF_TOKEN=$HF_TOKEN"
+  -e "HUGGINGFACE_HUB_TOKEN=$HUGGINGFACE_HUB_TOKEN"
+  -e "TLLM_PROFILE_START_STOP=$PROFILE_START_STOP"
+  -e "TLLM_LLMAPI_ENABLE_NVTX=1"
+  -e "TLLM_TORCH_PROFILE_TRACE=$TRACE_PATH"
+  -e "RUN_DIR=$RUN_DIR"
+  -e "LOG_PATH=$LOG_PATH"
+  -e "MODEL_ID=$MODEL"
+  -e "SERVE_PORT=$PORT"
+  -v "$HF_CACHE:/root/.cache/huggingface"
+  -v "$SHARED_ROOT:$SHARED_ROOT"
+)
+
+if [[ -n "$OVERRIDE_PY_EXECUTOR" ]]; then
+  docker_args+=(
+    -v "$OVERRIDE_PY_EXECUTOR:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py:ro"
+  )
+fi
+
+trust_remote_code_arg=""
+if [[ "$TRUST_REMOTE_CODE" -eq 1 ]]; then
+  trust_remote_code_arg="--trust_remote_code"
+fi
+
+container_cmd=$(
+  cat <<EOF
+mkdir -p "$RUN_DIR" && trtllm-serve serve "$MODEL" \
+  --backend pytorch \
+  --tp_size "$TP_SIZE" \
+  --gpus_per_node "$GPU_COUNT" \
+  --host 0.0.0.0 \
+  --port "$PORT" \
+  --max_seq_len "$MAX_SEQ_LEN" \
+  --kv_cache_free_gpu_memory_fraction "$KV_FRACTION" \
+  $trust_remote_code_arg \
+  $EXTRA_LLM_OPTIONS \
+  > "$LOG_PATH" 2>&1
+EOF
+)
+
+docker_args+=("$IMAGE" -lc "$container_cmd")
+docker "${docker_args[@]}" >/dev/null
+
+cleanup() {
+  docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+ready=0
+for _ in $(seq 1 180); do
+  if curl -sf "http://127.0.0.1:${PORT}/v1/models" >/dev/null; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$ready" -ne 1 ]]; then
+  echo "Server did not become ready on port ${PORT}. Recent logs:" >&2
+  docker logs "$CONTAINER_NAME" 2>&1 | tail -n 120 >&2 || true
+  exit 1
+fi
+
+python3 - <<PY
+import json
+import sys
+import urllib.request
+
+sys.path.insert(0, ${SCRIPT_DIR@Q})
+from profile_common import extract_openai_chat_text, synthetic_prompt
+
+prompt = ${PROMPT@Q} or synthetic_prompt(int(${INPUT_LEN@Q}))
+stage = ${STAGE@Q}
+warmup_steps = int(${WARMUP_STEPS@Q})
+active_steps = int(${ACTIVE_STEPS@Q})
+request_count = warmup_steps + active_steps if stage == "prefill" else 1
+
+payload = {
+    "model": ${MODEL@Q},
+    "messages": [{"role": "user", "content": prompt}],
+    "temperature": 0,
+    "max_tokens": int(${REQUEST_MAX_TOKENS@Q}),
+}
+for request_idx in range(request_count):
+    req = urllib.request.Request(
+        "http://127.0.0.1:${PORT}/v1/chat/completions",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(req, timeout=600) as resp:
+        body = json.loads(resp.read().decode())
+text, source = extract_openai_chat_text(body)
+print(text[:400] if text else f"[empty completion; source={source}]")
+PY
+
+for _ in $(seq 1 120); do
+  if [[ -s "$TRACE_PATH" ]]; then
+    break
+  fi
+  sleep 2
+done
+
+if [[ ! -s "$TRACE_PATH" ]]; then
+  echo "Trace was not written: $TRACE_PATH" >&2
+  exit 1
+fi
+
+python3 "$SCRIPT_DIR/probe_llm_server.py" \
+  --framework trtllm \
+  --url "http://127.0.0.1:${PORT}" \
+  --model "$MODEL" \
+  | docker exec -i sglang_bbuf bash -lc "cat > '$BENCHMARK_PATH'" >/dev/null
+
+echo "TRACE_PATH=$TRACE_PATH"
+echo "LOG_PATH=$LOG_PATH"
+echo "BENCHMARK_PATH=$BENCHMARK_PATH"
@@ -0,0 +1,343 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_vllm_torch_profile_host.sh \
+    --model Qwen/Qwen3-8B \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_vllm_formal \
+    --port 31088 \
+    --gpus 1
+
+  run_vllm_torch_profile_host.sh \
+    --model openai/gpt-oss-20b \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_vllm_4gpu \
+    --port 31088 \
+    --gpus 2,3,4,5 \
+    --tensor-parallel-size 4
+
+Options:
+  --model TEXT                   Hugging Face model id.
+  --run-dir PATH                Shared /data directory for logs and traces.
+  --port INT                    Host port for vllm serve.
+  --gpus TEXT                   CUDA_VISIBLE_DEVICES value, for example 1 or 2,3,4,5.
+  --gpu TEXT                    Alias for --gpus.
+  --image TEXT                  Container image.
+  --hf-cache PATH               Host Hugging Face cache path.
+  --gpu-memory-util FLOAT       vLLM --gpu-memory-utilization.
+  --max-model-len INT           vLLM --max-model-len.
+    --tensor-parallel-size INT    vLLM --tensor-parallel-size. Defaults to the visible GPU count.
+    --profiler-active-iterations INT
+                                 Torch-profiler active iterations.
+    --enforce-eager               Launch vLLM with --enforce-eager for mapping traces.
+  --trust-remote-code           Pass --trust-remote-code.
+  --request-max-tokens INT      Generation length for the probe request.
+  --prompt TEXT                 Probe prompt.
+  --warmup-steps INT            Warmup steps before profiling. Defaults to 10.
+  --profile-workload TEXT       legacy|prefill|decode|both. Defaults to both.
+  --prefill-input-len INT       Synthetic prefill prompt length. Defaults to 4090.
+  --prefill-output-len INT      Synthetic prefill output length. Defaults to 1.
+  --decode-input-len INT        Synthetic decode prompt length. Defaults to 1.
+  --decode-output-len INT       Synthetic decode output length. Defaults to 2048.
+  --container-name TEXT         Override container name.
+  --help                        Show this message.
+
+Environment:
+  HF_TOKEN or HUGGINGFACE_HUB_TOKEN must be set.
+
+Notes:
+  - Run this on the H100 host, not inside `sglang_bbuf`.
+  - This uses the vLLM torch-profiler flow: `--profiler-config`, then POST
+    `/start_profile` and `/stop_profile`.
+  - Default capture is two labeled profiles: prefill 4090->1 and decode 1->2048.
+  - Current vLLM profiler config already defaults `torch_profiler_with_stack=true`.
+  - A small benchmark summary is written after profiling.
+EOF
+}
+
+IMAGE="vllm/vllm-openai:latest"
+HF_CACHE="/data/.cache/huggingface"
+GPU_MEMORY_UTIL=0.90
+MAX_MODEL_LEN=4096
+TP_SIZE=""
+ENFORCE_EAGER=0
+TRUST_REMOTE_CODE=0
+REQUEST_MAX_TOKENS=12
+PROFILER_ACTIVE_ITERATIONS=5
+PROMPT="Explain the difference between CUDA graph mode and eager mode in two sentences."
+WARMUP_STEPS=10
+PROFILE_WORKLOAD="both"
+PREFILL_INPUT_LEN=4090
+PREFILL_OUTPUT_LEN=1
+DECODE_INPUT_LEN=1
+DECODE_OUTPUT_LEN=2048
+CONTAINER_NAME=""
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+MODEL=""
+RUN_DIR=""
+PORT=""
+GPUS=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model)
+      MODEL="$2"
+      shift 2
+      ;;
+    --run-dir)
+      RUN_DIR="$2"
+      shift 2
+      ;;
+    --port)
+      PORT="$2"
+      shift 2
+      ;;
+    --gpu)
+      GPUS="$2"
+      shift 2
+      ;;
+    --gpus)
+      GPUS="$2"
+      shift 2
+      ;;
+    --image)
+      IMAGE="$2"
+      shift 2
+      ;;
+    --hf-cache)
+      HF_CACHE="$2"
+      shift 2
+      ;;
+    --gpu-memory-util)
+      GPU_MEMORY_UTIL="$2"
+      shift 2
+      ;;
+    --max-model-len)
+      MAX_MODEL_LEN="$2"
+      shift 2
+      ;;
+    --tensor-parallel-size)
+      TP_SIZE="$2"
+      shift 2
+      ;;
+    --profiler-active-iterations)
+      PROFILER_ACTIVE_ITERATIONS="$2"
+      shift 2
+      ;;
+    --enforce-eager)
+      ENFORCE_EAGER=1
+      shift
+      ;;
+    --trust-remote-code)
+      TRUST_REMOTE_CODE=1
+      shift
+      ;;
+    --request-max-tokens)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --prompt)
+      PROMPT="$2"
+      shift 2
+      ;;
+    --warmup-steps)
+      WARMUP_STEPS="$2"
+      shift 2
+      ;;
+    --profile-workload)
+      PROFILE_WORKLOAD="$2"
+      shift 2
+      ;;
+    --prefill-input-len)
+      PREFILL_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --prefill-output-len)
+      PREFILL_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-input-len)
+      DECODE_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-output-len)
+      DECODE_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --container-name)
+      CONTAINER_NAME="$2"
+      shift 2
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "${HF_TOKEN:-}" && -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  echo "Set HF_TOKEN or HUGGINGFACE_HUB_TOKEN before running." >&2
+  exit 2
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  HF_TOKEN="$HUGGINGFACE_HUB_TOKEN"
+fi
+if [[ -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  HUGGINGFACE_HUB_TOKEN="$HF_TOKEN"
+fi
+
+if [[ -z "$MODEL" || -z "$RUN_DIR" || -z "$PORT" || -z "$GPUS" ]]; then
+  usage >&2
+  exit 2
+fi
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tensor-parallel-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+if (( PROFILER_ACTIVE_ITERATIONS < 1 )); then
+  echo "--profiler-active-iterations must be >= 1." >&2
+  exit 2
+fi
+
+PROFILE_DIR="$RUN_DIR/vllm_profile"
+LOG_PATH="$RUN_DIR/server.log"
+ANALYSIS_PATH="$RUN_DIR/analysis_vllm_live.txt"
+BENCHMARK_PATH="$RUN_DIR/benchmark_vllm.json"
+
+if [[ -z "$CONTAINER_NAME" ]]; then
+  model_slug="${MODEL##*/}"
+  model_slug="${model_slug//\//-}"
+  model_slug="${model_slug//./-}"
+  model_slug="${model_slug//_/-}"
+  gpu_slug="${GPUS//,/-}"
+  CONTAINER_NAME="vllm-${model_slug}-g${gpu_slug}-p${PORT}"
+  if [[ "$ENFORCE_EAGER" -eq 1 ]]; then
+    CONTAINER_NAME="${CONTAINER_NAME}-eager"
+  fi
+fi
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$PROFILE_DIR'"
+docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+
+profiler_config=$(python3 - <<PY
+import json
+print(json.dumps({
+    "profiler": "torch",
+    "torch_profiler_dir": ${PROFILE_DIR@Q},
+    "active_iterations": int(${PROFILER_ACTIVE_ITERATIONS@Q}),
+}))
+PY
+)
+
+docker_args=(
+  run -d --rm
+  --name "$CONTAINER_NAME"
+  --gpus all
+  --ipc=host
+  --network host
+  -e "CUDA_VISIBLE_DEVICES=$GPUS"
+  -e "HF_TOKEN=$HF_TOKEN"
+  -e "HUGGINGFACE_HUB_TOKEN=$HUGGINGFACE_HUB_TOKEN"
+  -e "VLLM_RPC_TIMEOUT=1800000"
+  -v "$HF_CACHE:/root/.cache/huggingface"
+  -v "$RUN_DIR:$RUN_DIR"
+)
+
+docker_cmd=(
+  "$IMAGE"
+  "$MODEL"
+  --host 0.0.0.0
+  --port "$PORT"
+  --tensor-parallel-size "$TP_SIZE"
+  --max-model-len "$MAX_MODEL_LEN"
+  --gpu-memory-utilization "$GPU_MEMORY_UTIL"
+  --profiler-config "$profiler_config"
+)
+
+if [[ "$ENFORCE_EAGER" -eq 1 ]]; then
+  docker_cmd+=(--enforce-eager)
+fi
+if [[ "$TRUST_REMOTE_CODE" -eq 1 ]]; then
+  docker_cmd+=(--trust-remote-code)
+fi
+
+docker "${docker_args[@]}" "${docker_cmd[@]}" >/dev/null
+cleanup() {
+  docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+ready=0
+for _ in $(seq 1 180); do
+  if curl -sf "http://127.0.0.1:${PORT}/v1/models" >/dev/null; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$ready" -ne 1 ]]; then
+  echo "Server did not become ready on port ${PORT}. Recent logs:" >&2
+  docker logs "$CONTAINER_NAME" 2>&1 | tail -n 120 >&2 || true
+  exit 1
+fi
+
+python3 "$SCRIPT_DIR/analyze_llm_torch_profile.py" \
+  --framework vllm \
+  --url "http://127.0.0.1:${PORT}" \
+  --output-dir "$PROFILE_DIR" \
+  --num-steps "$PROFILER_ACTIVE_ITERATIONS" \
+  --warmup-steps "$WARMUP_STEPS" \
+  --probe-requests 1 \
+  --no-profile-by-stage \
+  --profile-workload "$PROFILE_WORKLOAD" \
+  --probe-prompt "$PROMPT" \
+  --probe-max-new-tokens "$REQUEST_MAX_TOKENS" \
+  --prefill-input-len "$PREFILL_INPUT_LEN" \
+  --prefill-output-len "$PREFILL_OUTPUT_LEN" \
+  --decode-input-len "$DECODE_INPUT_LEN" \
+  --decode-output-len "$DECODE_OUTPUT_LEN" \
+  > "$ANALYSIS_PATH"
+
+profile_found=0
+for _ in $(seq 1 240); do
+  if find "$PROFILE_DIR" -type f \( -name '*.pt.trace.json' -o -name '*.pt.trace.json.gz' -o -name '*.trace.json' -o -name '*.trace.json.gz' \) | grep -q .; then
+    profile_found=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$profile_found" -ne 1 ]]; then
+  echo "No vLLM profiler traces appeared under $PROFILE_DIR" >&2
+  docker logs "$CONTAINER_NAME" 2>&1 | tail -n 120 >&2 || true
+  exit 1
+fi
+
+python3 "$SCRIPT_DIR/probe_llm_server.py" \
+  --framework vllm \
+  --url "http://127.0.0.1:${PORT}" \
+  --model "$MODEL" \
+  | docker exec -i sglang_bbuf bash -lc "cat > '$BENCHMARK_PATH'" >/dev/null
+
+docker logs "$CONTAINER_NAME" 2>&1 | docker exec -i sglang_bbuf bash -lc "cat > '$LOG_PATH'" || true
+sed -n '1,240p' "$ANALYSIS_PATH"
+echo "PROFILE_DIR=$PROFILE_DIR"
+echo "LOG_PATH=$LOG_PATH"
+echo "ANALYSIS_PATH=$ANALYSIS_PATH"
+echo "BENCHMARK_PATH=$BENCHMARK_PATH"