[skill] Remove outdated llm-serving-auto-benchmark skill (#29487)
This commit is contained in:
Xiaoyu Zhang
@@ -1,527 +0,0 @@
|
||||
---
|
||||
name: llm-serving-auto-benchmark
|
||||
description: Framework-independent LLM serving benchmark skill for comparing SGLang, vLLM, TensorRT-LLM, or another serving framework. Use when a user wants to find the best deployment command for one model across multiple serving frameworks under the same workload, GPU budget, and latency SLA.
|
||||
---
|
||||
|
||||
# LLM Serving Auto Benchmark
|
||||
|
||||
## Overview
|
||||
|
||||
Use this skill to compare LLM serving frameworks such as SGLang, vLLM, and
|
||||
TensorRT-LLM for the same model and workload.
|
||||
|
||||
Use a config-driven workflow:
|
||||
|
||||
- keep launch-only capacity choices in each framework's `base_server_flags`
|
||||
- put the search knobs in `search_space`
|
||||
- run the same dataset scenarios for every framework
|
||||
- generate a bounded candidate list from `search_space`, with the baseline
|
||||
candidate included first
|
||||
- keep failed candidates in the result file
|
||||
- pick the best SLA-passing candidate after normalizing the results
|
||||
|
||||
For model-specific starting points, prefer the shipped configs in
|
||||
`configs/cookbook-llm/`. They define a framework-neutral LLM serving cookbook
|
||||
model set and translate each entry into framework-native SGLang, vLLM, and
|
||||
TensorRT-LLM server flags. Validate those configs before a real run:
|
||||
|
||||
```bash
|
||||
python .claude/skills/llm-serving-auto-benchmark/scripts/validate_cookbook_configs.py \
|
||||
.claude/skills/llm-serving-auto-benchmark/configs/cookbook-llm
|
||||
```
|
||||
|
||||
If you have captured target-environment `--help` files, add
|
||||
`--help-dir <artifact-help-dir>`. That check only loads configs, verifies the
|
||||
server flag names, and renders candidate commands; it does not launch model
|
||||
servers.
|
||||
|
||||
Prefer native tooling when it gives better coverage:
|
||||
|
||||
- SGLang: `python -m sglang.auto_benchmark` when available, otherwise
|
||||
`python -m sglang.bench_serving`
|
||||
- vLLM: `vllm bench sweep serve` for server-parameter sweeps, otherwise
|
||||
`vllm serve` plus `vllm bench serve`
|
||||
- TensorRT-LLM: `trtllm-serve` for the OpenAI-compatible server plus the
|
||||
TensorRT-LLM serving benchmark client or a common OpenAI-compatible benchmark
|
||||
client
|
||||
|
||||
TensorRT-LLM has one hard scope rule in this skill: the server backend is fixed
|
||||
to `trtllm-serve serve --backend pytorch`. Do not search TensorRT-LLM backend
|
||||
choice. If a request, config, or candidate asks for `trt`, an engine backend, or
|
||||
any other non-PyTorch TensorRT-LLM server backend, reject that candidate as
|
||||
unsupported for this skill and record the reason. This does not change the
|
||||
benchmark client backend; the TensorRT-LLM benchmark client still uses
|
||||
OpenAI-compatible modes such as `--backend openai` or `--backend openai-chat`.
|
||||
|
||||
Only pick a winner after each requested framework has had its main serving knobs
|
||||
tuned.
|
||||
|
||||
The parameter lists in this skill are not a compatibility contract. They are
|
||||
version-sensitive candidate knob families. Before every real run, record the
|
||||
exact framework version or git commit and verify the concrete CLI flag names
|
||||
with `--help` in the target environment.
|
||||
|
||||
The default search style is framework-neutral: start from a mostly pure-TP
|
||||
baseline, sweep a small set of high-impact runtime knobs, and cap the first
|
||||
pass around 10 candidates per framework. Do not search memory fractions by
|
||||
default.
|
||||
|
||||
## Validation Environment
|
||||
|
||||
This skill is target-agnostic. It assumes any one of the following is
|
||||
available, and nothing more:
|
||||
|
||||
- a local GPU host with Docker/Podman and the target framework images pulled;
|
||||
- a remote GPU host reached via `ssh <host>` with the framework images already
|
||||
running in a container there;
|
||||
- a CI runner that can exec into a pre-built image for each framework.
|
||||
|
||||
Do not assume a specific operator host name (`h100_sglang`, `b200_*`,
|
||||
`radixark*`, `rtx5090_*`, etc.) inside this skill's own workflow. The concrete
|
||||
SSH wiring, container names, workspace paths, and HF token plumbing for a given
|
||||
box live in the operator-side per-host skills (for example `h100`,
|
||||
`h100-sglang-diffusion`, `b200`, `rtx5090`, `radixark02`, `radixark03`); this
|
||||
skill only requires that the caller can reach a shell inside a container with
|
||||
`sglang`, `vllm`, or `tensorrt_llm` installed.
|
||||
|
||||
Reference files are optional and version-sensitive. Treat historical flag notes
|
||||
as evidence from one image, not as a compatibility guarantee for the next run.
|
||||
|
||||
Additional H100 validation on `2026-05-01` used two 2-card models with a
|
||||
bounded search of two SGLang memory-fraction candidates and two vLLM
|
||||
memory-utilization candidates. The workload was random input `512`, output
|
||||
`64`, 8 prompts, and 2 warmup requests, only to prove the search and summary
|
||||
path can finish quickly.
|
||||
|
||||
| Model | GPUs | Best SGLang | Best vLLM | Artifact root |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `Qwen/Qwen3-8B` | 2x H100, TP=2 | `sglang_mem086`, 21.64 req/s, 1385.05 output tok/s, mean TTFT 70.54 ms | `vllm_mem080`, 22.88 req/s, 1464.25 output tok/s, mean TTFT 60.56 ms | `/data/bbuf/validate/core_skill_validation_20260501/qwen3_8b/auto_benchmark` |
|
||||
| `mistralai/Mistral-7B-Instruct-v0.3` | 2x H100, TP=2 | `sglang_mem080`, 24.09 req/s, 1541.92 output tok/s, mean TTFT 61.47 ms | `vllm_mem090`, 24.76 req/s, 1584.54 output tok/s, mean TTFT 58.63 ms | `/data/bbuf/validate/core_skill_validation_20260501/mistral_7b_instruct_v03/auto_benchmark` |
|
||||
|
||||
## Skill Scope
|
||||
|
||||
This skill is a playbook plus a config+validator toolchain, not a turn-key
|
||||
orchestrator. The operator still launches servers, drives workloads, and writes
|
||||
one normalized JSONL row per candidate.
|
||||
|
||||
The `scripts/` directory contains exactly two tools:
|
||||
|
||||
- `validate_cookbook_configs.py`: load cookbook YAML, render bounded candidate
|
||||
server commands, and check flag names against captured `--help` snapshots
|
||||
without launching servers.
|
||||
- `compare_benchmark_results.py`: turn normalized per-candidate JSONL into the
|
||||
markdown and optional CSV tables described in the Output Contract.
|
||||
|
||||
Cookbook configs under `configs/cookbook-llm/` must pass the validator. The
|
||||
shorter [references/example-plan.yaml](references/example-plan.yaml) is a
|
||||
one-off runtime-plan skeleton and is not expected to pass as-is. Use
|
||||
[references/result-schema.md](references/result-schema.md) as the single source
|
||||
of truth for SLA key names.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
Collect these before a long run:
|
||||
|
||||
- model and tokenizer path, target frameworks, GPU model/count, multi-node
|
||||
allowance, precision, and quantization constraints
|
||||
- endpoint shape, workload source, dataset scenarios, SLA target, search budget,
|
||||
and artifact output directory
|
||||
- version manifest: framework package version or git commit, container/Python
|
||||
environment, `--help` snapshots, and whether each search parameter was
|
||||
accepted by that exact CLI
|
||||
|
||||
If real production traffic is the goal, use the real request distribution. A
|
||||
synthetic workload is fine for bring-up and first-pass comparison, but it is not
|
||||
enough for a production choice.
|
||||
|
||||
Record each scenario's input/output length distribution in the normalized
|
||||
result rows. This is now part of the profiler handoff contract: if SGLang is
|
||||
slower and `sglang-sota-performance` invokes `llm-torch-profiler-analysis`,
|
||||
the profiler workload must reuse the slow SGLang benchmark scenario lengths
|
||||
instead of falling back to its generic prefill `4090->1` and decode `1->2048`
|
||||
defaults.
|
||||
|
||||
## Known Gotchas
|
||||
|
||||
Short list of failure modes that have bitten past validation runs. Check these
|
||||
before starting a long sweep.
|
||||
|
||||
- SGLang `fa3` attention backends need Hopper or newer. On A100, L40S, RTX
|
||||
5090, and older GPUs, drop `fa3` from the SGLang `search_space` and keep
|
||||
`flashinfer` (or `triton` when FlashInfer is unavailable).
|
||||
- SGLang `bench_serving` has two SGLang-facing backends: `--backend sglang` for
|
||||
the native `/generate` endpoint and `--backend sglang-oai` for the
|
||||
OpenAI-compatible endpoint. For cross-framework comparisons, prefer
|
||||
`sglang-oai` so every framework is measured on the same request path.
|
||||
- vLLM `--enable-dbo` only works when the target vLLM image is built with a
|
||||
supported all2all backend. Keep DBO out of the default candidate list unless
|
||||
the operator has verified the image.
|
||||
- vLLM `--max-num-partial-prefills > 1` is model- and runtime-gated. Keep `1`
|
||||
in the default pass; raise only after a preflight with the actual model.
|
||||
- The historical TensorRT-LLM 1.0.0 validation image accepted
|
||||
`--kv_cache_free_gpu_memory_fraction`; the older `--free_gpu_memory_fraction`
|
||||
exited with a CLI error. TensorRT-LLM was refreshed to 1.2.1 stable and
|
||||
1.3.0 release candidates by 2026-04-28, so re-check the accepted flag name
|
||||
via `--help` on the target image before a real run.
|
||||
- The historical TensorRT-LLM 1.0.0 multi-GPU PyTorch-backend validation used
|
||||
`--ipc=host`, `--ulimit memlock=-1`, `--ulimit stack=67108864`,
|
||||
`--shm-size=16g`, and `NCCL_IB_DISABLE=1` (for single-node) or an equivalent
|
||||
NCCL setup. Keep these as a starting point, not as a version-independent
|
||||
requirement.
|
||||
- The historical TensorRT-LLM 1.0.0 benchmark client took `--backend openai` or
|
||||
`--backend openai-chat`; `--backend trtllm` was rejected. This is separate
|
||||
from the server backend, which is pinned to `pytorch` by this skill.
|
||||
- `trtllm` `benchmark_serving --dataset-name random` silently falls back to
|
||||
ShareGPT sampling without `--random-ids` (or `--download-path`).
|
||||
- `max_seq_len` / `max_model_len` / `context_length` candidates must cover
|
||||
`max(input_len + output_len)` across every scenario, including values inside
|
||||
`search_space`, not just the baseline. The validator checks this; do not
|
||||
bypass it.
|
||||
|
||||
## Secrets Hygiene
|
||||
|
||||
- Never print `HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, or any upstream API key into
|
||||
a saved artifact. Pass them through container `-e VAR` (unquoted on the right
|
||||
side so the host value is inherited) and keep them out of `server_command`
|
||||
and `benchmark_command` fields written to the result JSONL.
|
||||
- When a framework echoes the full argv at startup, scrub the log or redact
|
||||
token-shaped substrings before uploading the artifact.
|
||||
|
||||
## Fairness Rules
|
||||
|
||||
Use these rules throughout the benchmark:
|
||||
|
||||
- Run every framework on the same GPU type, GPU count, model weights, tokenizer,
|
||||
precision, quantization policy, prompt distribution, output length target, and
|
||||
sampling settings.
|
||||
- Record framework version, git commit, container image, CUDA/NCCL versions, GPU
|
||||
driver, visible GPU ids, launch command, and benchmark command.
|
||||
- Warm the server before measuring. Restart or clear state between candidate
|
||||
configurations when cache effects would bias the comparison.
|
||||
- Compare steady-state fixed-QPS runs separately from burst throughput runs.
|
||||
- Keep failed candidates in the final results with their failure reason.
|
||||
- Report both raw throughput and SLA-passing throughput. The fastest failing
|
||||
candidate is not the best deployment command.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Preflight
|
||||
|
||||
Verify all requested frameworks before starting a search:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server --help
|
||||
python -m sglang.bench_serving --help
|
||||
vllm serve --help
|
||||
vllm serve --help=all
|
||||
vllm bench serve --help
|
||||
vllm bench serve --help=all
|
||||
vllm bench sweep serve --help=all
|
||||
trtllm-serve serve --help
|
||||
python -m tensorrt_llm.serve.scripts.benchmark_serving --help
|
||||
```
|
||||
|
||||
Use the framework-specific `--help` output in the target environment as the
|
||||
source of truth. Do not keep a stale launch flag just because it appears in an
|
||||
old note.
|
||||
|
||||
vLLM 0.19 and newer use grouped help. Plain `vllm serve --help` only shows the
|
||||
groups, so capture `--help=all` before deciding whether a search knob exists.
|
||||
|
||||
Save these `--help` outputs into the run artifact directory. If a listed search
|
||||
knob is missing from the current CLI, remove or translate that knob before
|
||||
running the benchmark. Do not silently pass unknown flags.
|
||||
|
||||
For TensorRT-LLM, also confirm that `trtllm-serve serve --help` accepts
|
||||
`--backend pytorch`. If it does not, mark TensorRT-LLM unsupported in that
|
||||
environment rather than falling back to a different server backend.
|
||||
|
||||
For each framework, launch a minimal server, confirm `/v1/models` or the native
|
||||
model-info endpoint, send one streaming request, run one tiny benchmark with at
|
||||
least 5 requests, then save the launch command, benchmark command, server log,
|
||||
and benchmark output.
|
||||
|
||||
Before any GPU-backed smoke run, check the requested GPU ids directly with
|
||||
`nvidia-smi`. If a requested GPU is already in use, stop and record that fact.
|
||||
Do not silently borrow a different GPU count for a performance comparison. It is
|
||||
fine to run a smaller one-GPU smoke only when the result is clearly labeled as a
|
||||
flow check rather than a fair throughput comparison.
|
||||
|
||||
If the target environment runs through containers, follow
|
||||
[references/container-runbook.md](references/container-runbook.md) and save image
|
||||
tags, pull commands, launch/benchmark logs, and cleanup commands.
|
||||
|
||||
### 2. Normalize The Workload
|
||||
|
||||
Use one canonical workload for all frameworks. Recommended JSONL row shape:
|
||||
|
||||
```json
|
||||
{"prompt": [{"role": "user", "content": "Summarize this text."}], "output_len": 256}
|
||||
{"prompt": "Write a short explanation of CUDA graphs.", "output_len": 128}
|
||||
```
|
||||
|
||||
Optional fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": [{"role": "user", "content": "Use low temperature."}],
|
||||
"output_len": 256,
|
||||
"extra_request_body": {"temperature": 0.0, "top_p": 0.95},
|
||||
"metadata": {"source": "prod-sample"}
|
||||
}
|
||||
```
|
||||
|
||||
When converting user data:
|
||||
|
||||
- inspect at least 3 rows before conversion
|
||||
- preserve request-level sampling options in `extra_request_body`
|
||||
- do not include the final assistant answer in the prompt when that answer is
|
||||
the target completion
|
||||
- keep multimodal or tool-call payloads only if all requested frameworks support
|
||||
the chosen endpoint shape
|
||||
|
||||
For synthetic bring-up, use the shipped two-scenario shape:
|
||||
|
||||
```yaml
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names: [chat, summarization]
|
||||
input_len: [1000, 8000]
|
||||
output_len: [1000, 1000]
|
||||
```
|
||||
|
||||
Each aligned `input_len` / `output_len` pair is one scenario. Do not take the
|
||||
cartesian product unless the user asks for that.
|
||||
Name each scenario and keep the aligned pair in the artifacts. For custom
|
||||
datasets, compute or record representative `input_len` and `output_len`
|
||||
buckets, at least p50 and p95 when possible, so later profiler runs can match
|
||||
the slow bucket rather than profiling an unrelated synthetic shape.
|
||||
|
||||
Before searching any sequence-length limit, compute the largest
|
||||
`input_len + output_len` in the dataset. SGLang `context_length`, vLLM
|
||||
`max_model_len`, and TensorRT-LLM `max_seq_len` must be at least that value for
|
||||
every candidate that is expected to run all scenarios.
|
||||
|
||||
### 3. Pick A Search Tier
|
||||
|
||||
Use the smallest tier that can answer the user's question:
|
||||
|
||||
- Tier 1: smoke and sanity. One baseline plus a few high-impact knobs.
|
||||
- Tier 2: default. A bounded sweep over the most likely server settings.
|
||||
- Tier 3: exhaustive. Only when the search space is already tight and the user
|
||||
accepts a long run.
|
||||
|
||||
Default budget:
|
||||
|
||||
- `num_prompts: 80` for the default cross-framework comparison; `num_prompts:
|
||||
20` per scenario is acceptable for a smoke/flow check and must be labeled as
|
||||
such in the artifact (not as a performance result).
|
||||
- `search.max_candidates_per_framework: 10` for the first useful pass
|
||||
- candidate generation: baseline first, then a bounded product or ordered
|
||||
candidate list from `search_space`
|
||||
- at most 5 QPS search rounds unless the user asks for more
|
||||
- stop early when every candidate in one framework is clearly OOM or fails the
|
||||
basic health check
|
||||
|
||||
Keep these in `base_server_flags` unless the user specifically wants a capacity
|
||||
or memory study:
|
||||
|
||||
- SGLang `mem_fraction_static`
|
||||
- SGLang `schedule_policy`
|
||||
- vLLM `gpu_memory_utilization`
|
||||
- TensorRT-LLM `kv_cache_free_gpu_memory_fraction`
|
||||
|
||||
These are real knobs, but they widen the search quickly and often turn a serving
|
||||
comparison into a memory-limit study.
|
||||
|
||||
### 4. Tune SGLang
|
||||
|
||||
Prefer the SGLang auto-benchmark runner when the target checkout supports it:
|
||||
|
||||
```bash
|
||||
python -m sglang.auto_benchmark run --config /path/to/sglang.yaml
|
||||
```
|
||||
|
||||
Otherwise launch the server manually and benchmark with:
|
||||
|
||||
```bash
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 256 \
|
||||
--num-prompts 80 \
|
||||
--request-rate 8 \
|
||||
--output-file /path/to/sglang/results.json \
|
||||
--output-details
|
||||
```
|
||||
|
||||
Version-sensitive SGLang knob families to verify:
|
||||
|
||||
- `tp_size`, `pp_size`, `dp_size`, `ep_size`
|
||||
- `attention_backend`, `prefill_attention_backend`, `decode_attention_backend`
|
||||
- `sampling_backend`
|
||||
- `max_running_requests`, `max_queued_requests`
|
||||
- `chunked_prefill_size`, `prefill_max_requests`, `max_prefill_tokens`
|
||||
- `max_total_tokens`, `page_size`
|
||||
- CUDA graph and piecewise CUDA graph settings
|
||||
- speculative or EAGLE settings only after the non-speculative baseline is tuned
|
||||
|
||||
Keep `mem_fraction_static` and `schedule_policy` pinned in the default pass,
|
||||
matching the shared cookbook config style.
|
||||
|
||||
For quick smoke tests, it is reasonable to disable CUDA graph and piecewise CUDA
|
||||
graph startup work if the goal is only to prove the framework flow. Record those
|
||||
flags in the artifact. Do not carry that smoke setting into a performance winner
|
||||
unless the user asked to tune eager-mode serving.
|
||||
|
||||
### 5. Tune vLLM
|
||||
|
||||
Use vLLM's sweep runner when available:
|
||||
|
||||
```bash
|
||||
vllm bench sweep serve \
|
||||
--serve-cmd 'vllm serve <model> --port 8000' \
|
||||
--bench-cmd 'vllm bench serve --backend vllm --model <model> --port 8000 --dataset-name random --num-prompts 80' \
|
||||
--serve-params /path/to/vllm_serve_params.json \
|
||||
--bench-params /path/to/vllm_bench_params.json \
|
||||
--output-dir /path/to/vllm_results
|
||||
```
|
||||
|
||||
If sweep support is unavailable, run `vllm serve` for each candidate and measure
|
||||
with `vllm bench serve`.
|
||||
|
||||
Version-sensitive vLLM knob families to verify:
|
||||
|
||||
- tensor, pipeline, data, decode-context, and expert parallelism
|
||||
- `gpu_memory_utilization`
|
||||
- `max_num_seqs`
|
||||
- `max_num_batched_tokens`
|
||||
- `max_model_len`
|
||||
- `enable_chunked_prefill`, partial prefill limits, and DBO thresholds
|
||||
- KV cache dtype and block size
|
||||
- dtype and quantization settings
|
||||
- CUDA graph capture sizes or eager-mode toggles when relevant
|
||||
- prefix cache and speculative decoding settings only when the workload needs
|
||||
those features
|
||||
|
||||
vLLM should get a normal sweep, not one baseline command. See
|
||||
[references/framework-reference.md](references/framework-reference.md) for
|
||||
native command templates and cross-framework knob families. Confirm each flag on
|
||||
the target image's `--help` before a run.
|
||||
|
||||
Keep `gpu_memory_utilization` in the baseline for the default pass. Search it
|
||||
only when the question is explicitly about fitting the model or trading capacity
|
||||
against throughput.
|
||||
|
||||
Keep DBO and all2all backend settings out of the default pass unless the target
|
||||
vLLM environment is already set up for them. They are real tuning knobs, but a
|
||||
candidate can fail at startup if the required all2all backend is not available.
|
||||
Also preflight concurrent partial prefill before raising
|
||||
`max_num_partial_prefills` above 1; some model/runtime combinations reject it at
|
||||
startup.
|
||||
|
||||
### 6. Tune TensorRT-LLM
|
||||
|
||||
Use `trtllm-serve serve` as the server entrypoint when the target environment
|
||||
supports it:
|
||||
|
||||
```bash
|
||||
trtllm-serve serve <model> \
|
||||
--backend pytorch \
|
||||
--tp_size <tp> \
|
||||
--pp_size <pp> \
|
||||
--kv_cache_free_gpu_memory_fraction 0.75 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
Then benchmark the OpenAI-compatible endpoint with the TensorRT-LLM serving
|
||||
benchmark client or with the same OpenAI-compatible client used for the other
|
||||
frameworks.
|
||||
|
||||
In the historical TensorRT-LLM 1.0.0 validation image,
|
||||
`benchmark_serving --dataset-name random` sampled from ShareGPT unless either
|
||||
`--download-path` or `--random-ids` was passed. For a fast synthetic smoke test,
|
||||
pass `--random-ids`, then confirm the behavior on the target TensorRT-LLM image.
|
||||
|
||||
TensorRT-LLM flag names are especially version-sensitive. In the validated
|
||||
TensorRT-LLM 1.0.0 image, the KV-cache memory flag accepted by
|
||||
`trtllm-serve serve` was `--kv_cache_free_gpu_memory_fraction`, not
|
||||
`--free_gpu_memory_fraction`. TensorRT-LLM 1.2.1 is the latest stable GitHub
|
||||
release as of 2026-04-28, with 1.3.0 release candidates also published; verify
|
||||
the current flag with `trtllm-serve serve --help` before running a search on any
|
||||
GPU target.
|
||||
|
||||
TensorRT-LLM backend policy for this skill:
|
||||
|
||||
- launch the server with `--backend pytorch`
|
||||
- keep `backend: pytorch` in `base_server_flags`
|
||||
- do not add `backend` to `search_space`
|
||||
- reject `trt`, engine-backed serving, or any other non-PyTorch TensorRT-LLM
|
||||
server backend as unsupported for this skill
|
||||
|
||||
Version-sensitive TensorRT-LLM knob families to verify:
|
||||
|
||||
- `tp_size`, `pp_size`, and `ep_size`
|
||||
- max batch size, max sequence length, max number of tokens, and KV-cache budget
|
||||
- inflight batching and scheduler options
|
||||
- extra LLM API options YAML used by `trtllm-serve` with the PyTorch backend
|
||||
|
||||
The `trtllm-serve serve` CLI exposes fewer direct runtime knobs than SGLang or
|
||||
vLLM. Use direct flags when they exist, then use `--extra_llm_api_options` for
|
||||
PyTorch-backend settings that are not top-level CLI flags. Keep unsupported
|
||||
backend or engine requests in the failure table instead of translating them.
|
||||
|
||||
Keep `kv_cache_free_gpu_memory_fraction` in the baseline for the default pass.
|
||||
Search `max_batch_size`, `max_num_tokens`, `max_seq_len`, and validated
|
||||
PyTorch-backend config options first. The server backend remains fixed to
|
||||
`pytorch`.
|
||||
|
||||
### 7. Normalize Results
|
||||
|
||||
Write one JSONL row per candidate using the schema in
|
||||
[references/result-schema.md](references/result-schema.md). Then run:
|
||||
|
||||
```bash
|
||||
python .claude/skills/llm-serving-auto-benchmark/scripts/compare_benchmark_results.py \
|
||||
--input /path/to/candidates.jsonl \
|
||||
--output /path/to/summary.md
|
||||
```
|
||||
|
||||
Rank candidates in this order:
|
||||
|
||||
1. SLA passed
|
||||
2. highest request throughput or goodput
|
||||
3. highest output token throughput
|
||||
4. lower mean TTFT
|
||||
5. lower mean TPOT/ITL
|
||||
6. lower GPU count or simpler deployment if performance is close
|
||||
|
||||
Keep the SLA gate itself unchanged. In the cookbook configs and normalized
|
||||
result schema, TTFT SLA still uses `max_p99_ttft_ms` and TPOT SLA still uses
|
||||
`max_p99_tpot_ms`; only the default cross-candidate comparison order switches
|
||||
to mean TTFT and mean TPOT.
|
||||
|
||||
## Output Contract
|
||||
|
||||
Return a compact report with workload/SLA, hardware and framework versions, best
|
||||
deployment-command tables per framework/scenario, one cross-framework comparison
|
||||
table, exact launch and benchmark commands for winners, and artifact paths for
|
||||
workload, raw/normalized results, CSV or markdown summary, and server logs.
|
||||
|
||||
When SGLang is not the winner, include a profiler handoff note with the slow
|
||||
SGLang scenario name and the exact input/output lengths or percentile bucket to
|
||||
pass to `llm-torch-profiler-analysis`.
|
||||
|
||||
Include failed or excluded candidates with reasons. Explain that this table is a
|
||||
record of tried configs that were not selected: candidates that failed, were
|
||||
skipped by policy, or completed but missed the SLA. Add caveats for synthetic
|
||||
workloads, incomplete fair searches, or framework-specific parameter
|
||||
substitutions.
|
||||
|
||||
Use [references/framework-reference.md](references/framework-reference.md) when
|
||||
you need command templates, source links, or knob-family mappings. Use
|
||||
[references/example-plan.yaml](references/example-plan.yaml) as the starting
|
||||
point for a full cross-framework run plan.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Cookbook LLM Configs
|
||||
|
||||
These configs define a framework-neutral LLM serving cookbook model set and translate each model into a three-framework run plan for SGLang, vLLM, and TensorRT-LLM.
|
||||
|
||||
Scope:
|
||||
- SGLang can preserve source-recipe `base_flags` and `search_space` where applicable; if a sequence limit is smaller than the default synthetic scenario, the config raises that limit so the shipped workload can run.
|
||||
- vLLM uses framework-native `vllm serve` flags. The translation keeps the same model, tokenizer, dataset shape, GPU count, and high-impact batching/prefix-cache knobs; it does not copy SGLang-only parser or scheduler flags.
|
||||
- TensorRT-LLM uses `trtllm-serve serve` with `backend: pytorch` fixed in `base_server_flags`. Backend choice is never searched.
|
||||
- The two default random scenarios remain aligned pairs: `chat` uses `1000 -> 1000`, and `summarization` uses `8000 -> 1000`.
|
||||
|
||||
Before a real run, capture the target framework `--help` output and validate the configs:
|
||||
|
||||
```bash
|
||||
python .claude/skills/llm-serving-auto-benchmark/scripts/validate_cookbook_configs.py .claude/skills/llm-serving-auto-benchmark/configs/cookbook-llm
|
||||
```
|
||||
|
||||
With captured help files, add `--help-dir <artifact-help-dir>` to check the concrete flag names against that environment. This check only loads configs and renders candidate commands; it does not launch model servers.
|
||||
@@ -1,130 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: deepseek-math-v2.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: deepseek-ai/DeepSeek-Math-V2
|
||||
tokenizer: deepseek-ai/DeepSeek-Math-V2
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: deepseek-ai/DeepSeek-Math-V2
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/deepseek-math-v2
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: deepseek-ai/DeepSeek-Math-V2
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,133 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: deepseek-r1-0528.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: deepseek-ai/DeepSeek-R1-0528
|
||||
tokenizer: deepseek-ai/DeepSeek-R1-0528
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: deepseek-ai/DeepSeek-R1-0528
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/deepseek-r1-0528
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
enable_symm_mem: true
|
||||
model_path: deepseek-ai/DeepSeek-R1-0528
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,132 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: deepseek-v3.1.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: deepseek-ai/DeepSeek-V3.1
|
||||
tokenizer: deepseek-ai/DeepSeek-V3.1
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: deepseek-ai/DeepSeek-V3.1
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/deepseek-v3.1
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: deepseek-ai/DeepSeek-V3.1
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,132 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: deepseek-v3.2.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: deepseek-ai/DeepSeek-V3.2
|
||||
tokenizer: deepseek-ai/DeepSeek-V3.2
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: deepseek-ai/DeepSeek-V3.2
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/deepseek-v3.2
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: deepseek-ai/DeepSeek-V3.2
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,133 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: deepseek-v3.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: deepseek-ai/DeepSeek-V3
|
||||
tokenizer: deepseek-ai/DeepSeek-V3
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: deepseek-ai/DeepSeek-V3
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/deepseek-v3
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
enable_symm_mem: true
|
||||
model_path: deepseek-ai/DeepSeek-V3
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
-123
@@ -1,123 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: devstral-small-2-24b-instruct-2512.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: mistralai/Devstral-Small-2-24B-Instruct-2512
|
||||
tokenizer: mistralai/Devstral-Small-2-24B-Instruct-2512
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: mistralai/Devstral-Small-2-24B-Instruct-2512
|
||||
max_concurrency:
|
||||
- null
|
||||
- 16
|
||||
- 32
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 16.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/devstral-small-2-24b-instruct-2512
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
model_path: mistralai/Devstral-Small-2-24B-Instruct-2512
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-117
@@ -1,117 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: ernie-4.5-21b-a3b-pt.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: baidu/ERNIE-4.5-21B-A3B-PT
|
||||
tokenizer: baidu/ERNIE-4.5-21B-A3B-PT
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: baidu/ERNIE-4.5-21B-A3B-PT
|
||||
max_concurrency:
|
||||
- null
|
||||
- 16
|
||||
- 32
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 16.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/ernie-4.5-21b-a3b-pt
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
model_path: baidu/ERNIE-4.5-21B-A3B-PT
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,122 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: glm-4.5.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: zai-org/GLM-4.5
|
||||
tokenizer: zai-org/GLM-4.5
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: zai-org/GLM-4.5
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 8.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/glm-4.5
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
context_length: 9000
|
||||
model_path: zai-org/GLM-4.5
|
||||
trust_remote_code: true
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
trust_remote_code: true
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 9000
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
trust_remote_code: true
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 9000
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 9000
|
||||
- 16384
|
||||
@@ -1,135 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: glm-4.6.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: zai-org/GLM-4.6
|
||||
tokenizer: zai-org/GLM-4.6
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: zai-org/GLM-4.6
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/glm-4.6
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: zai-org/GLM-4.6
|
||||
trust_remote_code: true
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
trust_remote_code: true
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
trust_remote_code: true
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,126 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: glm-4.7-flash.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: zai-org/GLM-4.7-Flash
|
||||
tokenizer: zai-org/GLM-4.7-Flash
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: zai-org/GLM-4.7-Flash
|
||||
max_concurrency:
|
||||
- null
|
||||
- 16
|
||||
- 32
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 16.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/glm-4.7-flash
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
model_path: zai-org/GLM-4.7-Flash
|
||||
trust_remote_code: true
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
trust_remote_code: true
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
trust_remote_code: true
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,130 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: glm-4.7.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: zai-org/GLM-4.7
|
||||
tokenizer: zai-org/GLM-4.7
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: zai-org/GLM-4.7
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 8.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/glm-4.7
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
context_length: 9000
|
||||
model_path: zai-org/GLM-4.7
|
||||
trust_remote_code: true
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
ep_size:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
trust_remote_code: true
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 9000
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
trust_remote_code: true
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 9000
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 9000
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
@@ -1,132 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: glm-5-fp8.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: zai-org/GLM-5-FP8
|
||||
tokenizer: zai-org/GLM-5-FP8
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: zai-org/GLM-5-FP8
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/glm-5-fp8
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: zai-org/GLM-5-FP8
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,126 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: glyph.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: zai-org/Glyph
|
||||
tokenizer: zai-org/Glyph
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: zai-org/Glyph
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 8.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/glyph
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
reasoning_parser: glm45
|
||||
tool_call_parser: glm45
|
||||
model_path: zai-org/Glyph
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,132 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: gpt-oss-120b.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: openai/gpt-oss-120b
|
||||
tokenizer: openai/gpt-oss-120b
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: openai/gpt-oss-120b
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/gpt-oss-120b
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: openai/gpt-oss-120b
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,135 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: intern-s1.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: internlm/Intern-S1
|
||||
tokenizer: internlm/Intern-S1
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: internlm/Intern-S1
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/intern-s1
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
trust_remote_code: true
|
||||
model_path: internlm/Intern-S1
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,133 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: kimi-k2-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: moonshotai/Kimi-K2-Instruct
|
||||
tokenizer: moonshotai/Kimi-K2-Instruct
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: moonshotai/Kimi-K2-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/kimi-k2-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
trust_remote_code: true
|
||||
model_path: moonshotai/Kimi-K2-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
@@ -1,127 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: kimi-k2.5.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: moonshotai/Kimi-K2.5
|
||||
tokenizer: moonshotai/Kimi-K2.5
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: moonshotai/Kimi-K2.5
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/kimi-k2.5
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
trust_remote_code: true
|
||||
model_path: moonshotai/Kimi-K2.5
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-121
@@ -1,121 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: kimi-linear-48b-a3b-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: moonshotai/Kimi-Linear-48B-A3B-Instruct
|
||||
tokenizer: moonshotai/Kimi-Linear-48B-A3B-Instruct
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: moonshotai/Kimi-Linear-48B-A3B-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 8.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/kimi-linear-48b-a3b-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
trust_remote_code: true
|
||||
model_path: moonshotai/Kimi-Linear-48B-A3B-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,134 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: ling-2.5-1t.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: inclusionAI/Ling-2.5-1T
|
||||
tokenizer: inclusionAI/Ling-2.5-1T
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: true
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: inclusionAI/Ling-2.5-1T
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 2.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/ling-2.5-1t
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
pp_size: 2
|
||||
nnodes: 2
|
||||
trust_remote_code: true
|
||||
tool_call_parser: qwen
|
||||
model_path: inclusionAI/Ling-2.5-1T
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
pp_size:
|
||||
- 1
|
||||
- 2
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
pipeline_parallel_size: 2
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 2
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,130 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: llada2-1-mini.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: inclusionAI/LLaDA2.1-mini
|
||||
tokenizer: inclusionAI/LLaDA2.1-mini
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: inclusionAI/LLaDA2.1-mini
|
||||
max_concurrency:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/llada2-1-mini
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 1
|
||||
dllm_algorithm: JointThreshold
|
||||
trust_remote_code: true
|
||||
max_running_requests: 1
|
||||
attention_backend: flashinfer
|
||||
model_path: inclusionAI/LLaDA2.1-mini
|
||||
mem_fraction_static: 0.77
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 1
|
||||
- 2
|
||||
- 4
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-124
@@ -1,124 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: llama-3.1-70b-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: meta-llama/Llama-3.1-70B-Instruct
|
||||
tokenizer: meta-llama/Llama-3.1-70B-Instruct
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: meta-llama/Llama-3.1-70B-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 12.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/llama-3.1-70b-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
model_path: meta-llama/Llama-3.1-70B-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-118
@@ -1,118 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: llama-3.3-70b-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: meta-llama/Llama-3.3-70B-Instruct
|
||||
tokenizer: meta-llama/Llama-3.3-70B-Instruct
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: meta-llama/Llama-3.3-70B-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 16
|
||||
- 32
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 16.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/llama-3.3-70b-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tool_call_parser: llama3
|
||||
model_path: meta-llama/Llama-3.3-70B-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-122
@@ -1,122 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: llama-4-maverick-17b-128e-instruct-fp8.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
||||
tokenizer: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
||||
max_concurrency:
|
||||
- null
|
||||
- 2
|
||||
- 4
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 2.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/llama-4-maverick-17b-128e-instruct-fp8
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
context_length: 1000000
|
||||
trust_remote_code: true
|
||||
enable_multimodal: true
|
||||
model_path: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 4
|
||||
- 8
|
||||
- 12
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 1000000
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 4
|
||||
- 8
|
||||
- 12
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 1000000
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 4
|
||||
- 8
|
||||
- 12
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 1000000
|
||||
-129
@@ -1,129 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: llama-4-scout-17b-16e-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
||||
tokenizer: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
||||
precision: bfloat16
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/llama-4-scout-17b-16e-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
enable_multimodal: true
|
||||
context_length: 65536
|
||||
dtype: bfloat16
|
||||
trust_remote_code: true
|
||||
model_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 8
|
||||
- 16
|
||||
- 24
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 65536
|
||||
dtype: bfloat16
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 8
|
||||
- 16
|
||||
- 24
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 65536
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 8
|
||||
- 16
|
||||
- 24
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 65536
|
||||
@@ -1,133 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: mimo-v2-flash.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: XiaomiMiMo/MiMo-V2-Flash
|
||||
tokenizer: XiaomiMiMo/MiMo-V2-Flash
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: XiaomiMiMo/MiMo-V2-Flash
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/mimo-v2-flash
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
trust_remote_code: true
|
||||
max_running_requests: 128
|
||||
chunked_prefill_size: 16384
|
||||
model_loader_extra_config: '{"enable_multithread_load": "true","num_threads": 64}'
|
||||
attention_backend: fa3
|
||||
reasoning_parser: qwen3
|
||||
tool_call_parser: mimo
|
||||
model_path: XiaomiMiMo/MiMo-V2-Flash
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,121 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: minimax-m2.1.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: MiniMaxAI/MiniMax-M2.1
|
||||
tokenizer: MiniMaxAI/MiniMax-M2.1
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: MiniMaxAI/MiniMax-M2.1
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 8.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/minimax-m2.1
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
trust_remote_code: true
|
||||
model_path: MiniMaxAI/MiniMax-M2.1
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,133 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: minimax-m2.5.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: MiniMaxAI/MiniMax-M2.5
|
||||
tokenizer: MiniMaxAI/MiniMax-M2.5
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: MiniMaxAI/MiniMax-M2.5
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/minimax-m2.5
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
trust_remote_code: true
|
||||
model_path: MiniMaxAI/MiniMax-M2.5
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
-121
@@ -1,121 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: ministral-3-8b-instruct-2512.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: mistralai/Ministral-3-8B-Instruct-2512
|
||||
tokenizer: mistralai/Ministral-3-8B-Instruct-2512
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: mistralai/Ministral-3-8B-Instruct-2512
|
||||
max_concurrency:
|
||||
- null
|
||||
- 16
|
||||
- 32
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 16.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/ministral-3-8b-instruct-2512
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
trust_remote_code: true
|
||||
tool_call_parser: mistral
|
||||
model_path: mistralai/Ministral-3-8B-Instruct-2512
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-124
@@ -1,124 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: mistral-small-4-119b-2603.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: mistralai/Mistral-Small-4-119B-2603
|
||||
tokenizer: mistralai/Mistral-Small-4-119B-2603
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 2
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: mistralai/Mistral-Small-4-119B-2603
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 6.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/mistral-small-4-119b-2603
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 2
|
||||
model_path: mistralai/Mistral-Small-4-119B-2603
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 2
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 2
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-128
@@ -1,128 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: nemotron-3-nano-30b-a3b-bf16.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
|
||||
tokenizer: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 1
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
|
||||
max_concurrency:
|
||||
- null
|
||||
- 16
|
||||
- 32
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 16.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/nemotron-3-nano-30b-a3b-bf16
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 1
|
||||
trust_remote_code: true
|
||||
kv_cache_dtype: fp8_e4m3
|
||||
model_path: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: fp8_e4m3
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 1
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-128
@@ -1,128 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: nemotron-3-super-120b-a12b-bf16.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
|
||||
tokenizer: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 6.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/nemotron-3-super-120b-a12b-bf16
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
trust_remote_code: true
|
||||
kv_cache_dtype: fp8_e4m3
|
||||
model_path: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: fp8_e4m3
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,132 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: qwen3-235b-a22b.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: Qwen/Qwen3-235B-A22B
|
||||
tokenizer: Qwen/Qwen3-235B-A22B
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: Qwen/Qwen3-235B-A22B
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/qwen3-235b-a22b
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: Qwen/Qwen3-235B-A22B
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
-131
@@ -1,131 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: qwen3-coder-480b-a35b-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: Qwen/Qwen3-Coder-480B-A35B-Instruct
|
||||
tokenizer: Qwen/Qwen3-Coder-480B-A35B-Instruct
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: Qwen/Qwen3-Coder-480B-A35B-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/qwen3-coder-480b-a35b-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
ep_size: 2
|
||||
moe_runner_backend: triton
|
||||
model_path: Qwen/Qwen3-Coder-480B-A35B-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 2
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
ep_size: 2
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 2
|
||||
@@ -1,124 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: qwen3-coder-next.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: Qwen/Qwen3-Coder-Next
|
||||
tokenizer: Qwen/Qwen3-Coder-Next
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 2
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: Qwen/Qwen3-Coder-Next
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 12.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/qwen3-coder-next
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 2
|
||||
model_path: Qwen/Qwen3-Coder-Next
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 2
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 2
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
-130
@@ -1,130 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: qwen3-next-80b-a3b-instruct.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: Qwen/Qwen3-Next-80B-A3B-Instruct
|
||||
tokenizer: Qwen/Qwen3-Next-80B-A3B-Instruct
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 2
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: Qwen/Qwen3-Next-80B-A3B-Instruct
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 12.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/qwen3-next-80b-a3b-instruct
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 2
|
||||
model_path: Qwen/Qwen3-Next-80B-A3B-Instruct
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
ep_size:
|
||||
- 1
|
||||
- 2
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 2
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 2
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 2
|
||||
-132
@@ -1,132 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: qwen35-397b-a17b-fp8.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: Qwen/Qwen3.5-397B-A17B-FP8
|
||||
tokenizer: Qwen/Qwen3.5-397B-A17B-FP8
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: Qwen/Qwen3.5-397B-A17B-FP8
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 4.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/qwen35-397b-a17b-fp8
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
model_path: Qwen/Qwen3.5-397B-A17B-FP8
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
- 8
|
||||
@@ -1,124 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: ring-2.5-1t.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: inclusionAI/Ring-2.5-1T
|
||||
tokenizer: inclusionAI/Ring-2.5-1T
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 8
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: inclusionAI/Ring-2.5-1T
|
||||
max_concurrency:
|
||||
- null
|
||||
- 4
|
||||
- 8
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 2.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/ring-2.5-1t
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 8
|
||||
model_path: inclusionAI/Ring-2.5-1T
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 8
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 8
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 32
|
||||
- 48
|
||||
- 64
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
@@ -1,133 +0,0 @@
|
||||
schema_version: 1
|
||||
source:
|
||||
kind: llm_serving_cookbook
|
||||
source_recipe_file: step-3.5-flash.yaml
|
||||
translation: SGLang flags preserve the source recipe where applicable, with sequence limits raised when needed for the default dataset; vLLM and TensorRT-LLM use framework-native serving flags for the same model and dataset shape.
|
||||
model:
|
||||
name: stepfun-ai/Step-3.5-Flash
|
||||
tokenizer: stepfun-ai/Step-3.5-Flash
|
||||
precision: auto
|
||||
quantization: model default
|
||||
hardware:
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names:
|
||||
- chat
|
||||
- summarization
|
||||
input_len:
|
||||
- 1000
|
||||
- 8000
|
||||
output_len:
|
||||
- 1000
|
||||
- 1000
|
||||
benchmark:
|
||||
endpoint: /v1/completions
|
||||
backend: openai-compatible
|
||||
tokenizer: stepfun-ai/Step-3.5-Flash
|
||||
max_concurrency:
|
||||
- null
|
||||
- 8
|
||||
- 16
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
qps:
|
||||
lower: 0.25
|
||||
upper: 8.0
|
||||
tolerance: 0.1
|
||||
sla:
|
||||
max_p99_ttft_ms: 1500
|
||||
max_p99_tpot_ms: 30
|
||||
min_success_rate: 0.99
|
||||
output_dir: ./auto_benchmark_results/cookbook-llm/step-3.5-flash
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 8
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
server_command: python -m sglang.launch_server
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
trust_remote_code: true
|
||||
model_path: stepfun-ai/Step-3.5-Flash
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
search_space:
|
||||
prefill_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
decode_attention_backend:
|
||||
- fa3
|
||||
- flashinfer
|
||||
chunked_prefill_size:
|
||||
- 4096
|
||||
- 8192
|
||||
max_running_requests:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
vllm:
|
||||
enabled: true
|
||||
server_command: vllm serve
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
enable_chunked_prefill: true
|
||||
kv_cache_dtype: auto
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_num_seqs:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_batched_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_num_partial_prefills:
|
||||
- 1
|
||||
max_long_partial_prefills:
|
||||
- 1
|
||||
long_prefill_token_threshold:
|
||||
- 0
|
||||
- 4096
|
||||
enable_prefix_caching:
|
||||
- true
|
||||
block_size:
|
||||
- 16
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
server_command: trtllm-serve serve
|
||||
backend_policy: fixed_pytorch
|
||||
config_source: framework_generic_translation
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
max_seq_len: 12288
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
max_batch_size:
|
||||
- 64
|
||||
- 96
|
||||
- 128
|
||||
max_num_tokens:
|
||||
- 8192
|
||||
- 16384
|
||||
max_seq_len:
|
||||
- 12288
|
||||
- 16384
|
||||
ep_size:
|
||||
- 1
|
||||
- 4
|
||||
@@ -1,321 +0,0 @@
|
||||
# Container Runbook
|
||||
|
||||
Use this runbook when the benchmark environment is container-based. It records
|
||||
the exact image, command, help output, server log, benchmark log, and cleanup
|
||||
step for each framework.
|
||||
|
||||
This runbook is target-agnostic. Every `docker run` / `docker exec` command
|
||||
works on a local box, an SSH-reachable remote GPU host, or a CI runner; the
|
||||
per-host skills (for example `h100`, `b200`, `rtx5090`, `radixark02`,
|
||||
`radixark03`) only add the SSH wrapper, container name, and workspace path
|
||||
for a specific operator box. Substitute those values where you see
|
||||
`$SGLANG_CONTAINER`, `$SGLANG_WORKSPACE`, and similar; nothing below assumes
|
||||
an H100.
|
||||
|
||||
## Common Setup
|
||||
|
||||
Pull the images that will be used:
|
||||
|
||||
```bash
|
||||
docker pull lmsysorg/sglang:dev
|
||||
docker pull vllm/vllm-openai:latest
|
||||
docker pull nvcr.io/nvidia/tensorrt-llm/release:latest
|
||||
```
|
||||
|
||||
Use quoted Docker GPU device lists:
|
||||
|
||||
```bash
|
||||
GPU_ARG='"device=6,7"'
|
||||
docker run --gpus "$GPU_ARG" ...
|
||||
```
|
||||
|
||||
The unquoted form `--gpus device=6,7` can be parsed incorrectly by Docker.
|
||||
|
||||
Mount the shared Hugging Face cache and pass tokens through environment variables
|
||||
when gated models are used:
|
||||
|
||||
```bash
|
||||
-v /data/.cache:/root/.cache \
|
||||
-e HF_TOKEN \
|
||||
-e HUGGINGFACE_HUB_TOKEN
|
||||
```
|
||||
|
||||
Do not print token values into logs.
|
||||
|
||||
Set the run variables once and pass them into containers that need them:
|
||||
|
||||
```bash
|
||||
export MODEL=TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
export TP=1
|
||||
export PP=1
|
||||
export PORT=8000
|
||||
export RUN_DIR=/tmp/llm-serving-auto-benchmark
|
||||
mkdir -p "$RUN_DIR"
|
||||
```
|
||||
|
||||
For synthetic validation, use two aligned scenarios rather than one tiny request
|
||||
shape:
|
||||
|
||||
```bash
|
||||
# chat-like
|
||||
RANDOM_INPUT_LEN=1000
|
||||
RANDOM_OUTPUT_LEN=1000
|
||||
|
||||
# summarization-like
|
||||
RANDOM_INPUT_LEN=8000
|
||||
RANDOM_OUTPUT_LEN=1000
|
||||
```
|
||||
|
||||
For a fast smoke on larger models, 20 prompts per scenario is a reasonable
|
||||
minimum. Do not treat that as a performance result.
|
||||
|
||||
Set each framework's sequence-length limit to cover the largest scenario. For
|
||||
the example above, use at least 9000 tokens for SGLang `--context-length`, vLLM
|
||||
`--max-model-len`, and TensorRT-LLM `--max_seq_len`.
|
||||
|
||||
Before launching a server, save the help output:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server --help > artifacts/help/sglang_launch_server.txt
|
||||
python -m sglang.bench_serving --help > artifacts/help/sglang_bench_serving.txt
|
||||
vllm serve --help=all > artifacts/help/vllm_serve_all.txt
|
||||
vllm bench serve --help=all > artifacts/help/vllm_bench_serve_all.txt
|
||||
vllm bench sweep serve --help=all > artifacts/help/vllm_bench_sweep_serve_all.txt
|
||||
trtllm-serve serve --help > artifacts/help/trtllm_serve.txt
|
||||
python -m tensorrt_llm.serve.scripts.benchmark_serving --help \
|
||||
> artifacts/help/trtllm_benchmark_serving.txt
|
||||
```
|
||||
|
||||
## SGLang
|
||||
|
||||
If a prepared GPU host already has a long-running SGLang container (local or
|
||||
reached via ssh; name is operator-specific), reuse it via `docker exec`
|
||||
instead of creating a new container. The per-host skills — `h100`,
|
||||
`h100-sglang-diffusion`, `b200`, `rtx5090`, `radixark02`, `radixark03`,
|
||||
and similar — provide the concrete container name and workspace path for
|
||||
that box; this runbook assumes the operator substitutes them:
|
||||
|
||||
```bash
|
||||
docker exec \
|
||||
-e MODEL \
|
||||
-e TP \
|
||||
-e PORT \
|
||||
"$SGLANG_CONTAINER" bash -lc "
|
||||
cd \"\$SGLANG_WORKSPACE\"
|
||||
python -m sglang.launch_server \\
|
||||
--model-path \"\$MODEL\" \\
|
||||
--tp-size \"\$TP\" \\
|
||||
--host 0.0.0.0 \\
|
||||
--port \"\$PORT\"
|
||||
"
|
||||
```
|
||||
|
||||
For a fresh container:
|
||||
|
||||
```bash
|
||||
docker run -d --name llmbench-sglang \
|
||||
--gpus "$GPU_ARG" \
|
||||
--network host \
|
||||
--ipc=host \
|
||||
-v /data/.cache:/root/.cache \
|
||||
-e MODEL \
|
||||
-e TP \
|
||||
-e PORT \
|
||||
-e HF_TOKEN \
|
||||
-e HUGGINGFACE_HUB_TOKEN \
|
||||
--entrypoint bash \
|
||||
lmsysorg/sglang:dev -lc '
|
||||
python -m sglang.launch_server \
|
||||
--model-path "$MODEL" \
|
||||
--tp-size "$TP" \
|
||||
--host 0.0.0.0 \
|
||||
--port "$PORT"
|
||||
'
|
||||
```
|
||||
|
||||
Then run either SGLang auto benchmark:
|
||||
|
||||
```bash
|
||||
python -m sglang.auto_benchmark run --config /path/to/sglang.yaml
|
||||
```
|
||||
|
||||
or a tiny OpenAI-compatible smoke benchmark:
|
||||
|
||||
```bash
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang-oai \
|
||||
--host 127.0.0.1 \
|
||||
--port "$PORT" \
|
||||
--dataset-name random \
|
||||
--random-input-len 32 \
|
||||
--random-output-len 8 \
|
||||
--num-prompts 4 \
|
||||
--request-rate 1 \
|
||||
--max-concurrency 2 \
|
||||
--output-file "$RUN_DIR/sglang/results.json" \
|
||||
--output-details
|
||||
```
|
||||
|
||||
## vLLM
|
||||
|
||||
Server template:
|
||||
|
||||
```bash
|
||||
docker run -d --name llmbench-vllm \
|
||||
--gpus "$GPU_ARG" \
|
||||
--network host \
|
||||
--ipc=host \
|
||||
-v /data/.cache:/root/.cache \
|
||||
-e MODEL \
|
||||
-e TP \
|
||||
-e PORT \
|
||||
-e HF_TOKEN \
|
||||
-e HUGGINGFACE_HUB_TOKEN \
|
||||
--entrypoint bash \
|
||||
vllm/vllm-openai:latest -lc '
|
||||
vllm serve "$MODEL" \
|
||||
--host 0.0.0.0 \
|
||||
--port "$PORT" \
|
||||
--tensor-parallel-size "$TP" \
|
||||
--dtype auto \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--max-model-len 4096 \
|
||||
--max-num-seqs 64 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--enable-chunked-prefill \
|
||||
--kv-cache-dtype auto \
|
||||
--enable-prefix-caching \
|
||||
--trust-remote-code
|
||||
'
|
||||
```
|
||||
|
||||
Benchmark template:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
--network host \
|
||||
-v /data/.cache:/root/.cache \
|
||||
-v "$RUN_DIR:/artifacts" \
|
||||
-e MODEL \
|
||||
-e PORT \
|
||||
--entrypoint bash \
|
||||
vllm/vllm-openai:latest -lc '
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--base-url "http://127.0.0.1:$PORT" \
|
||||
--model "$MODEL" \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 256 \
|
||||
--num-prompts 80 \
|
||||
--request-rate 8 \
|
||||
--max-concurrency 64 \
|
||||
--save-result \
|
||||
--result-dir /artifacts/vllm \
|
||||
--result-filename results.json
|
||||
'
|
||||
```
|
||||
|
||||
Use `vllm bench sweep serve` when the target image supports it and the search
|
||||
can be described with serve/bench parameter JSON files.
|
||||
|
||||
## TensorRT-LLM
|
||||
|
||||
This skill only supports the TensorRT-LLM PyTorch server backend. Keep
|
||||
`--backend pytorch` in every `trtllm-serve serve` command. Do not switch the
|
||||
server to `--backend trt`, an engine path, or any other backend; mark that
|
||||
candidate unsupported instead.
|
||||
|
||||
For single-node multi-GPU TensorRT-LLM containers, keep the IPC, ulimit, shared
|
||||
memory, and NCCL settings below. In a multi-GPU PyTorch-backend validation
|
||||
run (captured on an H100 host; the rule is not H100-specific), the server
|
||||
entered `PyTorchConfig` but failed NCCL allreduce without these container
|
||||
options; the same model and candidate list passed after adding them. Expect
|
||||
the same requirement on any single-node multi-GPU target.
|
||||
|
||||
Server template:
|
||||
|
||||
```bash
|
||||
docker run -d --name llmbench-trtllm \
|
||||
--gpus "$GPU_ARG" \
|
||||
--ipc=host \
|
||||
--ulimit memlock=-1 \
|
||||
--ulimit stack=67108864 \
|
||||
--shm-size=16g \
|
||||
--network host \
|
||||
-v /data/.cache:/root/.cache \
|
||||
-e MODEL \
|
||||
-e TP \
|
||||
-e PP \
|
||||
-e PORT \
|
||||
-e HF_TOKEN \
|
||||
-e HUGGINGFACE_HUB_TOKEN \
|
||||
-e NCCL_IB_DISABLE=1 \
|
||||
--entrypoint bash \
|
||||
nvcr.io/nvidia/tensorrt-llm/release:latest -lc '
|
||||
trtllm-serve serve "$MODEL" \
|
||||
--host 0.0.0.0 \
|
||||
--port "$PORT" \
|
||||
--backend pytorch \
|
||||
--tp_size "$TP" \
|
||||
--pp_size "$PP" \
|
||||
--max_batch_size 64 \
|
||||
--max_num_tokens 8192 \
|
||||
--max_seq_len 4096 \
|
||||
--kv_cache_free_gpu_memory_fraction 0.75 \
|
||||
--trust_remote_code
|
||||
'
|
||||
```
|
||||
|
||||
Benchmark template:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
--network host \
|
||||
-v /data/.cache:/root/.cache \
|
||||
-v "$RUN_DIR:/artifacts" \
|
||||
-e MODEL \
|
||||
-e PORT \
|
||||
--entrypoint bash \
|
||||
nvcr.io/nvidia/tensorrt-llm/release:latest -lc '
|
||||
python -m tensorrt_llm.serve.scripts.benchmark_serving \
|
||||
--backend openai \
|
||||
--host 127.0.0.1 \
|
||||
--port "$PORT" \
|
||||
--endpoint /v1/completions \
|
||||
--model "$MODEL" \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 256 \
|
||||
--random-ids \
|
||||
--num-prompts 80 \
|
||||
--request-rate 8 \
|
||||
--max-concurrency 64 \
|
||||
--save-result \
|
||||
--result-dir /artifacts/trtllm \
|
||||
--result-filename results.json
|
||||
'
|
||||
```
|
||||
|
||||
For TensorRT-LLM 1.0.0, the serving benchmark client `--backend` choices are
|
||||
`openai` and `openai-chat`. Do not pass `--backend trtllm`. This client flag is
|
||||
separate from the server backend pinned above.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Use unique container names per run and clean up by name:
|
||||
|
||||
```bash
|
||||
docker rm -f llmbench-sglang llmbench-vllm llmbench-trtllm
|
||||
```
|
||||
|
||||
If a port remains bound after container cleanup, inspect it before killing
|
||||
anything:
|
||||
|
||||
```bash
|
||||
ss -ltnp | grep ':8000'
|
||||
ps -eo pid,ppid,user,etime,cmd | grep '<model-or-port>'
|
||||
```
|
||||
|
||||
Only kill raw PIDs when the command line proves they belong to the current
|
||||
validation run.
|
||||
@@ -1,133 +0,0 @@
|
||||
# Example run plan for the llm-serving-auto-benchmark skill. Baseline flags stay
|
||||
# in base_server_flags, search knobs stay in search_space, and aligned dataset
|
||||
# length pairs define scenarios.
|
||||
#
|
||||
# Note: this is the runtime plan shape (top-level `sla`, no `schema_version` or
|
||||
# `server_command`). Cookbook configs in configs/cookbook-llm/ use the extended
|
||||
# schema enforced by scripts/validate_cookbook_configs.py; do not run the
|
||||
# validator against this file as-is.
|
||||
|
||||
model:
|
||||
name: Qwen/Qwen3-32B
|
||||
tokenizer: Qwen/Qwen3-32B
|
||||
precision: bf16
|
||||
quantization: none
|
||||
|
||||
version_manifest:
|
||||
sglang:
|
||||
container_image: lmsysorg/sglang:dev
|
||||
package_version: null
|
||||
git_commit: null
|
||||
server_help: artifacts/help/sglang_launch_server.txt
|
||||
benchmark_help: artifacts/help/sglang_bench_serving.txt
|
||||
vllm:
|
||||
container_image: vllm/vllm-openai:latest
|
||||
package_version: null
|
||||
git_commit: null
|
||||
server_help: artifacts/help/vllm_serve_all.txt
|
||||
benchmark_help: artifacts/help/vllm_bench_serve_all.txt
|
||||
sweep_help: artifacts/help/vllm_bench_sweep_serve_all.txt
|
||||
tensorrt_llm:
|
||||
container_image: nvcr.io/nvidia/tensorrt-llm/release:latest
|
||||
package_version: null
|
||||
git_commit: null
|
||||
server_help: artifacts/help/trtllm_serve.txt
|
||||
benchmark_help: artifacts/help/trtllm_benchmark_serving.txt
|
||||
|
||||
hardware:
|
||||
# Example values; replace with the actual target GPU (A100, H100, H200,
|
||||
# B200, MI300, RTX 5090, etc.). gpu_model is recorded for fairness audit,
|
||||
# not used as a scheduling hint.
|
||||
gpu_model: NVIDIA H100 80GB HBM3
|
||||
gpu_count: 4
|
||||
multi_node: false
|
||||
|
||||
dataset:
|
||||
kind: random
|
||||
num_prompts: 80
|
||||
scenario_names: [chat, summarization]
|
||||
input_len: [1000, 8000]
|
||||
output_len: [1000, 1000]
|
||||
canonical_jsonl: null
|
||||
|
||||
benchmark:
|
||||
endpoint: /v1/chat/completions
|
||||
backend: auto
|
||||
request_rates: null
|
||||
max_concurrency: [null, 16, 32]
|
||||
qps:
|
||||
lower: 1.0
|
||||
upper: 12.0
|
||||
tolerance: 0.1
|
||||
max_rounds: 5
|
||||
extra_request_body:
|
||||
temperature: 0.0
|
||||
|
||||
sla:
|
||||
max_p99_ttft_ms: 2000
|
||||
max_p99_tpot_ms: 80
|
||||
min_success_rate: 0.99
|
||||
|
||||
search:
|
||||
tier: 2
|
||||
max_candidates_per_framework: 10
|
||||
candidate_generation: baseline_first_bounded_product
|
||||
resume: true
|
||||
output_dir: /bench/results/llm-serving-auto-benchmark
|
||||
|
||||
frameworks:
|
||||
sglang:
|
||||
enabled: true
|
||||
base_server_flags:
|
||||
tp_size: 4
|
||||
trust_remote_code: true
|
||||
mem_fraction_static: 0.82
|
||||
schedule_policy: lpm
|
||||
context_length: 12288
|
||||
search_space:
|
||||
# Verify these names against `python -m sglang.launch_server --help`.
|
||||
prefill_attention_backend: [fa3, flashinfer]
|
||||
decode_attention_backend: [fa3, flashinfer]
|
||||
chunked_prefill_size: [8192, 16384]
|
||||
max_running_requests: [64, 128]
|
||||
|
||||
vllm:
|
||||
enabled: true
|
||||
base_server_flags:
|
||||
tensor_parallel_size: 4
|
||||
trust_remote_code: true
|
||||
gpu_memory_utilization: 0.90
|
||||
max_model_len: 12288
|
||||
dtype: auto
|
||||
search_space:
|
||||
# Verify these names against `vllm serve --help=all`.
|
||||
max_num_seqs: [64, 128]
|
||||
max_num_batched_tokens: [8192, 16384]
|
||||
enable_chunked_prefill: [true]
|
||||
# Raise above 1 only after the target model/runtime supports concurrent partial prefill.
|
||||
max_num_partial_prefills: [1]
|
||||
max_long_partial_prefills: [1]
|
||||
long_prefill_token_threshold: [0, 4096]
|
||||
enable_prefix_caching: [true]
|
||||
kv_cache_dtype: [auto]
|
||||
block_size: [16]
|
||||
|
||||
tensorrt_llm:
|
||||
enabled: true
|
||||
backend_policy: fixed_pytorch
|
||||
base_server_flags:
|
||||
backend: pytorch
|
||||
tp_size: 4
|
||||
pp_size: 1
|
||||
kv_cache_free_gpu_memory_fraction: 0.75
|
||||
trust_remote_code: true
|
||||
search_space:
|
||||
# Verify these names against `trtllm-serve serve --help`.
|
||||
# Do not add backend choices here; TensorRT-LLM is fixed to the PyTorch backend.
|
||||
max_batch_size: [64, 128]
|
||||
max_num_tokens: [8192, 16384]
|
||||
max_seq_len: [12288, 16384]
|
||||
# Uncomment and point at concrete config files to sweep PyTorch-backend
|
||||
# options via --extra_llm_api_options. A single [null] value contributes
|
||||
# no dimension to the search.
|
||||
# extra_llm_api_options: [null, /path/to/trt_llm_config_A.yaml]
|
||||
@@ -1,113 +0,0 @@
|
||||
# Framework Reference
|
||||
|
||||
Use this file when choosing native framework commands or translating tuning
|
||||
knobs across SGLang, vLLM, and TensorRT-LLM. Always verify the concrete CLI in
|
||||
the target container with `--help` before a long run.
|
||||
|
||||
## Native Entry Points
|
||||
|
||||
| Framework | Server | Benchmark | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| SGLang | `python -m sglang.launch_server` | `python -m sglang.auto_benchmark` or `python -m sglang.bench_serving` | Use `auto_benchmark` when available for server-flag search. Use `bench_serving` for direct native or OpenAI-compatible endpoint checks. |
|
||||
| vLLM | `vllm serve` | `vllm bench sweep serve` or `vllm bench serve` | Prefer `bench sweep serve` when sweeping server and benchmark parameter JSON files. |
|
||||
| TensorRT-LLM | `trtllm-serve serve --backend pytorch` | TensorRT-LLM serving benchmark client or a common OpenAI-compatible client | This skill does not cover engine-backed serving or non-PyTorch server backends. |
|
||||
|
||||
Common source docs:
|
||||
|
||||
- SGLang bench serving: <https://docs.sglang.ai/developer_guide/bench_serving.html>
|
||||
- vLLM benchmark sweeps: <https://docs.vllm.ai/en/latest/benchmarking/sweeps/>
|
||||
- vLLM `bench sweep serve`: <https://docs.vllm.ai/en/latest/cli/bench/sweep/serve.html>
|
||||
- TensorRT-LLM `trtllm-serve`: <https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve/trtllm-serve.html>
|
||||
- TensorRT-LLM deployment guide: <https://nvidia.github.io/TensorRT-LLM/deployment-guide/index.html>
|
||||
|
||||
## Command Templates
|
||||
|
||||
### SGLang
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path <model> \
|
||||
--tp-size <tp> \
|
||||
--port 30000
|
||||
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang-oai \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 256 \
|
||||
--num-prompts 80 \
|
||||
--request-rate 8
|
||||
```
|
||||
|
||||
Use `--backend sglang` for SGLang-native `/generate` checks. Use
|
||||
`--backend sglang-oai` when comparing against vLLM or TensorRT-LLM through an
|
||||
OpenAI-compatible path.
|
||||
|
||||
### vLLM
|
||||
|
||||
```bash
|
||||
vllm serve <model> \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--tensor-parallel-size <tp> \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--max-model-len 4096 \
|
||||
--max-num-seqs 64 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--enable-chunked-prefill
|
||||
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--base-url http://127.0.0.1:8000 \
|
||||
--model <model> \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 256 \
|
||||
--num-prompts 80
|
||||
```
|
||||
|
||||
### TensorRT-LLM
|
||||
|
||||
```bash
|
||||
trtllm-serve serve <model> \
|
||||
--backend pytorch \
|
||||
--tp_size <tp> \
|
||||
--kv_cache_free_gpu_memory_fraction 0.75 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
Benchmark the OpenAI-compatible endpoint with the TensorRT-LLM serving benchmark
|
||||
client or the same OpenAI-compatible client used for the other frameworks. Keep
|
||||
server backend choice fixed to `pytorch`.
|
||||
|
||||
## Knob Family Mapping
|
||||
|
||||
Do not copy flag names across frameworks. Compare knob families, then translate
|
||||
to the target CLI.
|
||||
|
||||
| Family | SGLang | vLLM | TensorRT-LLM |
|
||||
| --- | --- | --- | --- |
|
||||
| Parallelism | `--tp-size`, `--pp-size`, `--dp-size`, `--ep-size`, `--expert-parallel-size` | `--tensor-parallel-size`, `--pipeline-parallel-size`, `--data-parallel-size`, `--enable-expert-parallel` | `--tp_size`, `--pp_size`, `--ep_size`, `--gpus_per_node`, `--cluster_size` |
|
||||
| Memory and KV cache | `--mem-fraction-static`, `--max-total-tokens`, `--kv-cache-dtype`, `--page-size`, `--cpu-offload-gb` | `--gpu-memory-utilization`, `--kv-cache-memory-bytes`, `--kv-cache-dtype`, `--block-size`, `--cpu-offload-gb` | `--kv_cache_free_gpu_memory_fraction`, plus `--max_num_tokens`, `--max_seq_len`, `--max_batch_size` |
|
||||
| Batching and scheduler | `--max-running-requests`, `--schedule-policy`, `--chunked-prefill-size`, `--max-prefill-tokens`, `--prefill-max-requests` | `--max-num-seqs`, `--max-num-batched-tokens`, `--enable-chunked-prefill`, partial-prefill and DBO flags | `--max_batch_size`, `--max_num_tokens`, `--max_seq_len`; extra scheduler knobs may require `--extra_llm_api_options` |
|
||||
| Attention/backend | `--attention-backend`, `--prefill-attention-backend`, `--decode-attention-backend`, `--sampling-backend` | `--attention-backend`, `--gdn-prefill-backend`, `--mm-encoder-attn-backend` | `--backend pytorch` is fixed; do not search backend choice |
|
||||
| CUDA graph and compile | `--disable-cuda-graph`, `--cuda-graph-bs`, `--cuda-graph-max-bs`, `--disable-piecewise-cuda-graph`, `--enable-torch-compile` | `--enforce-eager`, `--compilation-config`, `--cudagraph-capture-sizes`, `--max-cudagraph-capture-size` | use direct flags or `--extra_llm_api_options`; record resolved PyTorch config from logs |
|
||||
| Prefix/speculative | `--disable-radix-cache`, `--disable-chunked-prefix-cache`, speculative decoding flags | `--enable-prefix-caching`, `--speculative-config` | only use PyTorch-backend options accepted by the target image |
|
||||
| Dtype, quantization, loading | `--dtype`, `--quantization`, `--load-format`, `--model-loader-extra-config`, `--trust-remote-code` | `--dtype`, `--quantization`, `--load-format`, `--model-loader-extra-config`, `--trust-remote-code`, `--hf-token` | `--trust_remote_code`, `--tokenizer`; engine build and non-PyTorch quantization flows are out of scope |
|
||||
|
||||
## Version Rules
|
||||
|
||||
Framework CLIs move quickly. For every real run:
|
||||
|
||||
1. Record the framework package version, git commit, image tag, and help files.
|
||||
2. Validate concrete flags with
|
||||
`scripts/validate_cookbook_configs.py --help-dir <artifact-help-dir>`.
|
||||
3. Move renamed or removed flags out of the run plan before benchmarking.
|
||||
4. Record which frameworks were model-smoked and which only passed preflight.
|
||||
|
||||
Historical validation from April 2026 used SGLang `0.5.10rc0`, vLLM `0.19.1`,
|
||||
and TensorRT-LLM `1.0.0`. Treat those notes as old evidence, not as current
|
||||
compatibility guarantees.
|
||||
@@ -1,161 +0,0 @@
|
||||
# Result Schema
|
||||
|
||||
Write one JSON object per candidate. Keep failed candidates in the same file so
|
||||
the final summary explains what was tried.
|
||||
|
||||
## SLA Key Convention
|
||||
|
||||
One canonical naming across this skill. Config files and normalized result rows
|
||||
must agree.
|
||||
|
||||
| Key | Where | Type |
|
||||
| --- | --- | --- |
|
||||
| `max_p99_ttft_ms` | both | float, milliseconds, p99 |
|
||||
| `max_p99_tpot_ms` | both | float, milliseconds, p99 |
|
||||
| `min_success_rate` | both | float in [0, 1] |
|
||||
| `passed` | result only | bool; recomputed after the run |
|
||||
|
||||
Do not use `max_ttft_ms` or `max_tpot_ms` without the `p99_` prefix; those names
|
||||
hide whether the target is a mean or a tail. Older cookbook configs used mean
|
||||
latency targets by accident and have been migrated to the p99 names above.
|
||||
|
||||
The config-level SLA block lives under `benchmark.sla` (cookbook configs) or at
|
||||
the top level (example plan). Either location is acceptable, but the key names
|
||||
must match this table.
|
||||
|
||||
## JSONL Row
|
||||
|
||||
The values below (`gpu_model`, `gpu_count`, file paths, numeric metrics, etc.)
|
||||
are illustrative. Replace them with the actual target hardware and measured
|
||||
values; this schema is not tied to H100.
|
||||
|
||||
```json
|
||||
{
|
||||
"framework": "sglang",
|
||||
"framework_version": "0.5.0",
|
||||
"framework_commit": "abcdef0",
|
||||
"candidate_id": "sglang-tp8-flashinfer",
|
||||
"model": "meta-llama/Llama-3.1-70B-Instruct",
|
||||
"status": "ok",
|
||||
"failure_reason": "",
|
||||
"hardware": {
|
||||
"gpu_model": "NVIDIA H100 80GB HBM3",
|
||||
"gpu_count": 8,
|
||||
"visible_devices": "0,1,2,3,4,5,6,7"
|
||||
},
|
||||
"workload": {
|
||||
"kind": "custom",
|
||||
"scenario": "chat",
|
||||
"dataset_path": "/bench/workload.autobench.jsonl",
|
||||
"input_len": 2048,
|
||||
"output_len": 512,
|
||||
"input_len_p50": 1800,
|
||||
"input_len_p95": 4096,
|
||||
"output_len_p50": 384,
|
||||
"output_len_p95": 1024,
|
||||
"num_prompts": 1000,
|
||||
"request_rate": 16,
|
||||
"max_concurrency": 256,
|
||||
"endpoint": "/v1/chat/completions"
|
||||
},
|
||||
"sla": {
|
||||
"max_p99_ttft_ms": 2000,
|
||||
"max_p99_tpot_ms": 80,
|
||||
"min_success_rate": 0.99,
|
||||
"passed": true
|
||||
},
|
||||
"metrics": {
|
||||
"request_throughput": 15.8,
|
||||
"output_token_throughput": 12500.0,
|
||||
"total_token_throughput": 42000.0,
|
||||
"mean_ttft_ms": 430.0,
|
||||
"p99_ttft_ms": 1550.0,
|
||||
"mean_tpot_ms": 26.0,
|
||||
"p99_tpot_ms": 72.0,
|
||||
"mean_e2e_ms": 8200.0,
|
||||
"p99_e2e_ms": 19000.0,
|
||||
"success_rate": 0.995
|
||||
},
|
||||
"server_command": "python -m sglang.launch_server ...",
|
||||
"benchmark_command": "python -m sglang.bench_serving ...",
|
||||
"validated_cli_flags": {
|
||||
"server": ["tp_size", "attention_backend"],
|
||||
"benchmark": ["dataset_name", "request_rate", "max_concurrency"]
|
||||
},
|
||||
"artifacts": {
|
||||
"server_log": "/bench/sglang/server.log",
|
||||
"raw_result": "/bench/sglang/results.jsonl",
|
||||
"server_help": "/bench/sglang/help_launch_server.txt",
|
||||
"benchmark_help": "/bench/sglang/help_bench_serving.txt"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`input_len` and `output_len` are the representative scenario lengths used for
|
||||
synthetic workloads or a named bucket. For custom production-like datasets,
|
||||
also include p50/p95 buckets when available. These fields let
|
||||
`sglang-sota-performance` pass the slow benchmark shape directly into
|
||||
`llm-torch-profiler-analysis`:
|
||||
|
||||
- prefill profile: `--prefill-input-len <slow input len>` and
|
||||
`--prefill-output-len 1`
|
||||
- decode profile: `--decode-input-len 1` and
|
||||
`--decode-output-len <slow output len>`
|
||||
|
||||
## Status Values
|
||||
|
||||
- `ok`: benchmark finished and metrics are trustworthy
|
||||
- `failed`: command failed for a known non-OOM reason
|
||||
- `oom`: model or candidate exhausted GPU/host memory
|
||||
- `timeout`: server or benchmark timed out
|
||||
- `skipped`: intentionally not run, with a reason in `failure_reason`
|
||||
|
||||
## Ranking Rule
|
||||
|
||||
The default ranking is:
|
||||
|
||||
1. `status == "ok"`
|
||||
2. `sla.passed == true`
|
||||
3. higher `metrics.request_throughput`
|
||||
4. higher `metrics.output_token_throughput`
|
||||
5. lower `metrics.mean_ttft_ms`
|
||||
6. lower `metrics.mean_tpot_ms`
|
||||
7. lower `hardware.gpu_count`
|
||||
|
||||
If the user cares more about token throughput than request throughput, swap
|
||||
steps 3 and 4 and state that in the final report.
|
||||
|
||||
This ranking rule does not change the SLA gate. Keep `sla.max_p99_ttft_ms` and
|
||||
`sla.max_p99_tpot_ms` as the tail-latency constraints; use mean TTFT and mean
|
||||
TPOT only for default winner selection among rows that have already passed SLA.
|
||||
|
||||
Missing metric semantics:
|
||||
|
||||
- If `metrics.mean_ttft_ms` is absent from a row, the ranking script treats it
|
||||
as the worst possible value, so that row falls below any candidate with a
|
||||
real mean-TTFT measurement. Do not write `0` as a placeholder for "no
|
||||
measurement"; leave the field out or set it to `null`.
|
||||
- If `metrics.mean_tpot_ms` is absent from a row, the ranking script treats it
|
||||
as the worst possible value, so that row falls below any candidate with a
|
||||
real mean-TPOT measurement. Do not write `0` as a placeholder for "no
|
||||
measurement"; leave the field out or set it to `null`.
|
||||
- If `metrics.request_throughput` or `metrics.output_token_throughput` is
|
||||
missing, the row ranks below any candidate with a real measurement in those
|
||||
keys. A failed candidate that still produced partial metrics should keep the
|
||||
metrics it did produce.
|
||||
|
||||
## Final Report Tables
|
||||
|
||||
The markdown summary must include these sections:
|
||||
|
||||
1. `Best Commands By Framework`: one table per framework. Each table has one row
|
||||
per workload scenario and includes the best candidate, SLA result, throughput,
|
||||
latency metrics, GPU count, exact server command, and artifacts.
|
||||
2. `Cross-Framework Best Comparison`: one table that compares the best SGLang,
|
||||
vLLM, and TensorRT-LLM command for each scenario. Sort each scenario by the
|
||||
ranking rule above so the best deployment choice is first.
|
||||
3. `Failed Or SLA-Failing Candidates`: include this table when any candidate
|
||||
failed, was skipped, or completed without passing SLA. This table records
|
||||
tried configs that were not selected. Keep each reason concrete enough to
|
||||
tell whether the candidate needs a retry, lower concurrency, a parameter fix,
|
||||
or no further action.
|
||||
@@ -1,308 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Summarize normalized cross-framework benchmark JSONL results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _get(row: dict[str, Any], path: str, default: Any = None) -> Any:
|
||||
current: Any = row
|
||||
for part in path.split("."):
|
||||
if not isinstance(current, dict) or part not in current:
|
||||
return default
|
||||
current = current[part]
|
||||
return current
|
||||
|
||||
|
||||
def _float(row: dict[str, Any], path: str, default: float = 0.0) -> float:
|
||||
value = _get(row, path, default)
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _bool(row: dict[str, Any], path: str, default: bool = False) -> bool:
|
||||
value = _get(row, path, default)
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
return value.lower() in {"1", "true", "yes", "y"}
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _mean_ttft_ms(row: dict[str, Any]) -> float:
|
||||
return _float(row, "metrics.mean_ttft_ms", 1e30)
|
||||
|
||||
|
||||
def _mean_tpot_ms(row: dict[str, Any]) -> float:
|
||||
return _float(row, "metrics.mean_tpot_ms", 1e30)
|
||||
|
||||
|
||||
def _rank_key(row: dict[str, Any]) -> tuple[Any, ...]:
|
||||
return (
|
||||
_get(row, "status") == "ok",
|
||||
_bool(row, "sla.passed"),
|
||||
_float(row, "metrics.request_throughput"),
|
||||
_float(row, "metrics.output_token_throughput"),
|
||||
-_mean_ttft_ms(row),
|
||||
-_mean_tpot_ms(row),
|
||||
-_float(row, "hardware.gpu_count", 1e30),
|
||||
)
|
||||
|
||||
|
||||
def _is_winner_candidate(row: dict[str, Any]) -> bool:
|
||||
return _get(row, "status") == "ok" and _bool(row, "sla.passed")
|
||||
|
||||
|
||||
def _fmt(value: Any, digits: int = 2) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, float):
|
||||
return f"{value:.{digits}f}"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _cell(value: Any, digits: int = 2) -> str:
|
||||
text = _fmt(value, digits)
|
||||
return text.replace("\n", "<br>").replace("|", "\\|")
|
||||
|
||||
|
||||
def _scenario(row: dict[str, Any]) -> str:
|
||||
for path in (
|
||||
"workload.scenario",
|
||||
"workload.scenario_name",
|
||||
"workload.dataset_scenario",
|
||||
"workload.dataset_name",
|
||||
"workload.kind",
|
||||
"scenario",
|
||||
):
|
||||
value = _get(row, path)
|
||||
if value:
|
||||
return str(value)
|
||||
return "default"
|
||||
|
||||
|
||||
def _server_command(row: dict[str, Any]) -> str:
|
||||
return str(_get(row, "server_command") or _get(row, "launch_command") or "")
|
||||
|
||||
|
||||
def _artifact_summary(row: dict[str, Any]) -> str:
|
||||
artifacts = _get(row, "artifacts", {})
|
||||
if not isinstance(artifacts, dict):
|
||||
return ""
|
||||
parts = []
|
||||
for key in ("raw_result", "server_log", "benchmark_log", "summary"):
|
||||
value = artifacts.get(key)
|
||||
if value:
|
||||
parts.append(f"{key}: {value}")
|
||||
return "<br>".join(parts)
|
||||
|
||||
|
||||
def load_rows(path: Path) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
with path.open(encoding="utf-8") as f:
|
||||
for line_no, line in enumerate(f, 1):
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
try:
|
||||
row = json.loads(stripped)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise SystemExit(f"{path}:{line_no}: invalid JSON: {exc}") from exc
|
||||
if not isinstance(row, dict):
|
||||
raise SystemExit(f"{path}:{line_no}: expected a JSON object")
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def best_by_framework_and_scenario(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
best: dict[tuple[str, str], dict[str, Any]] = {}
|
||||
for row in rows:
|
||||
if not _is_winner_candidate(row):
|
||||
continue
|
||||
key = (str(_get(row, "framework", "unknown")), _scenario(row))
|
||||
if key not in best or _rank_key(row) > _rank_key(best[key]):
|
||||
best[key] = row
|
||||
return sorted(
|
||||
best.values(), key=lambda row: (_scenario(row), _rank_key(row)), reverse=True
|
||||
)
|
||||
|
||||
|
||||
def write_csv(path: Path, rows: list[dict[str, Any]]) -> None:
|
||||
fields = [
|
||||
"framework",
|
||||
"scenario",
|
||||
"candidate_id",
|
||||
"status",
|
||||
"sla_passed",
|
||||
"request_throughput",
|
||||
"output_token_throughput",
|
||||
"mean_ttft_ms",
|
||||
"mean_tpot_ms",
|
||||
"p99_ttft_ms",
|
||||
"p99_tpot_ms",
|
||||
"gpu_count",
|
||||
"server_command",
|
||||
"failure_reason",
|
||||
]
|
||||
with path.open("w", encoding="utf-8", newline="") as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fields)
|
||||
writer.writeheader()
|
||||
for row in rows:
|
||||
writer.writerow(
|
||||
{
|
||||
"framework": _get(row, "framework", ""),
|
||||
"scenario": _scenario(row),
|
||||
"candidate_id": _get(row, "candidate_id", ""),
|
||||
"status": _get(row, "status", ""),
|
||||
"sla_passed": _bool(row, "sla.passed"),
|
||||
"request_throughput": _get(row, "metrics.request_throughput", ""),
|
||||
"output_token_throughput": _get(
|
||||
row, "metrics.output_token_throughput", ""
|
||||
),
|
||||
"mean_ttft_ms": _get(row, "metrics.mean_ttft_ms", ""),
|
||||
"mean_tpot_ms": _get(row, "metrics.mean_tpot_ms", ""),
|
||||
"p99_ttft_ms": _get(row, "metrics.p99_ttft_ms", ""),
|
||||
"p99_tpot_ms": _get(row, "metrics.p99_tpot_ms", ""),
|
||||
"gpu_count": _get(row, "hardware.gpu_count", ""),
|
||||
"server_command": _server_command(row),
|
||||
"failure_reason": _get(row, "failure_reason", ""),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _append_best_commands_by_framework(
|
||||
lines: list[str], scenario_winners: list[dict[str, Any]]
|
||||
) -> None:
|
||||
frameworks = sorted(
|
||||
{str(_get(row, "framework", "unknown")) for row in scenario_winners}
|
||||
)
|
||||
lines.extend(["## Best Commands By Framework", ""])
|
||||
for framework in frameworks:
|
||||
lines.extend(
|
||||
[
|
||||
f"### `{framework}`",
|
||||
"",
|
||||
"| Scenario | Candidate | Status | SLA | Req/s | Output tok/s | Total tok/s | Mean TTFT ms | Mean TPOT ms | Success rate | GPUs | Server command | Artifacts |",
|
||||
"| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | --- |",
|
||||
]
|
||||
)
|
||||
rows = [row for row in scenario_winners if _get(row, "framework") == framework]
|
||||
for row in sorted(rows, key=_scenario):
|
||||
lines.append(
|
||||
"| {scenario} | {candidate} | {status} | {sla} | {rps} | {otps} | {ttps} | {ttft} | {tpot} | {success} | {gpus} | {command} | {artifacts} |".format(
|
||||
scenario=_cell(_scenario(row)),
|
||||
candidate=_cell(_get(row, "candidate_id", "")),
|
||||
status=_cell(_get(row, "status", "")),
|
||||
sla=_cell(_bool(row, "sla.passed")),
|
||||
rps=_cell(_get(row, "metrics.request_throughput")),
|
||||
otps=_cell(_get(row, "metrics.output_token_throughput")),
|
||||
ttps=_cell(_get(row, "metrics.total_token_throughput")),
|
||||
ttft=_cell(_get(row, "metrics.mean_ttft_ms")),
|
||||
tpot=_cell(_get(row, "metrics.mean_tpot_ms")),
|
||||
success=_cell(_get(row, "metrics.success_rate")),
|
||||
gpus=_cell(_get(row, "hardware.gpu_count")),
|
||||
command=_cell(_server_command(row)),
|
||||
artifacts=_cell(_artifact_summary(row)),
|
||||
)
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
|
||||
def _append_cross_framework_table(
|
||||
lines: list[str], scenario_winners: list[dict[str, Any]]
|
||||
) -> None:
|
||||
lines.extend(
|
||||
[
|
||||
"## Cross-Framework Best Comparison",
|
||||
"",
|
||||
"| Scenario | Rank | Framework | Candidate | SLA | Req/s | Output tok/s | Mean TTFT ms | Mean TPOT ms | GPUs | Server command |",
|
||||
"| --- | ---: | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- |",
|
||||
]
|
||||
)
|
||||
scenario_names = sorted({_scenario(row) for row in scenario_winners})
|
||||
for scenario_name in scenario_names:
|
||||
rows = [row for row in scenario_winners if _scenario(row) == scenario_name]
|
||||
for rank, row in enumerate(sorted(rows, key=_rank_key, reverse=True), 1):
|
||||
lines.append(
|
||||
"| {scenario} | {rank} | {framework} | {candidate} | {sla} | {rps} | {otps} | {ttft} | {tpot} | {gpus} | {command} |".format(
|
||||
scenario=_cell(scenario_name),
|
||||
rank=rank,
|
||||
framework=_cell(_get(row, "framework", "")),
|
||||
candidate=_cell(_get(row, "candidate_id", "")),
|
||||
sla=_cell(_bool(row, "sla.passed")),
|
||||
rps=_cell(_get(row, "metrics.request_throughput")),
|
||||
otps=_cell(_get(row, "metrics.output_token_throughput")),
|
||||
ttft=_cell(_get(row, "metrics.mean_ttft_ms")),
|
||||
tpot=_cell(_get(row, "metrics.mean_tpot_ms")),
|
||||
gpus=_cell(_get(row, "hardware.gpu_count")),
|
||||
command=_cell(_server_command(row)),
|
||||
)
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
|
||||
def render_markdown(rows: list[dict[str, Any]]) -> str:
|
||||
scenario_winners = best_by_framework_and_scenario(rows)
|
||||
|
||||
lines = ["# Benchmark Summary", ""]
|
||||
if not rows:
|
||||
lines.append("No rows found.")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
_append_best_commands_by_framework(lines, scenario_winners)
|
||||
_append_cross_framework_table(lines, scenario_winners)
|
||||
|
||||
failed = [
|
||||
row
|
||||
for row in rows
|
||||
if _get(row, "status") != "ok" or not _bool(row, "sla.passed")
|
||||
]
|
||||
if failed:
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Failed Or SLA-Failing Candidates",
|
||||
"",
|
||||
"This table records tried configs that were not selected. They either failed, were skipped by policy, or completed without passing the SLA.",
|
||||
"",
|
||||
"| Framework | Candidate | Status | SLA | Reason |",
|
||||
"| --- | --- | --- | --- | --- |",
|
||||
]
|
||||
)
|
||||
for row in failed:
|
||||
lines.append(
|
||||
"| {framework} | {candidate} | {status} | {sla} | {reason} |".format(
|
||||
framework=_cell(_get(row, "framework", "")),
|
||||
candidate=_cell(_get(row, "candidate_id", "")),
|
||||
status=_cell(_get(row, "status", "")),
|
||||
sla=_cell(_bool(row, "sla.passed")),
|
||||
reason=_cell(_get(row, "failure_reason", "")),
|
||||
)
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input", required=True, type=Path, help="Normalized JSONL")
|
||||
parser.add_argument("--output", required=True, type=Path, help="Markdown summary")
|
||||
parser.add_argument("--csv", type=Path, help="Optional CSV table")
|
||||
args = parser.parse_args()
|
||||
|
||||
rows = load_rows(args.input)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(render_markdown(rows), encoding="utf-8")
|
||||
if args.csv:
|
||||
args.csv.parent.mkdir(parents=True, exist_ok=True)
|
||||
write_csv(args.csv, sorted(rows, key=_rank_key, reverse=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,434 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate cross-framework cookbook benchmark configs.
|
||||
|
||||
The validator is intentionally shallow: it proves that every config can be
|
||||
loaded, translated into bounded candidate commands, and checked against the
|
||||
known server flag surface. It does not launch model servers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import itertools
|
||||
import re
|
||||
import shlex
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
FRAMEWORKS = ("sglang", "vllm", "tensorrt_llm")
|
||||
ALLOWED_SOURCE_KINDS = {"llm_serving_cookbook"}
|
||||
|
||||
SEQUENCE_LIMIT_KEY = {
|
||||
"sglang": "context_length",
|
||||
"vllm": "max_model_len",
|
||||
"tensorrt_llm": "max_seq_len",
|
||||
}
|
||||
|
||||
ALLOWED_SLA_KEYS = {
|
||||
"max_p99_ttft_ms",
|
||||
"max_p99_tpot_ms",
|
||||
"min_success_rate",
|
||||
"max_p99_e2e_ms",
|
||||
}
|
||||
|
||||
DEPRECATED_SLA_KEYS = {
|
||||
"max_ttft_ms": "max_p99_ttft_ms",
|
||||
"max_tpot_ms": "max_p99_tpot_ms",
|
||||
"max_e2e_ms": "max_p99_e2e_ms",
|
||||
}
|
||||
|
||||
STATIC_SERVER_FLAGS = {
|
||||
"sglang": {
|
||||
"attention_backend",
|
||||
"chunked_prefill_size",
|
||||
"context_length",
|
||||
"decode_attention_backend",
|
||||
"dllm_algorithm",
|
||||
"dtype",
|
||||
"enable_multimodal",
|
||||
"enable_symm_mem",
|
||||
"ep_size",
|
||||
"host",
|
||||
"kv_cache_dtype",
|
||||
"max_running_requests",
|
||||
"mem_fraction_static",
|
||||
"model_loader_extra_config",
|
||||
"model_path",
|
||||
"moe_runner_backend",
|
||||
"nnodes",
|
||||
"port",
|
||||
"pp_size",
|
||||
"prefill_attention_backend",
|
||||
"reasoning_parser",
|
||||
"schedule_policy",
|
||||
"tool_call_parser",
|
||||
"tp_size",
|
||||
"trust_remote_code",
|
||||
},
|
||||
"vllm": {
|
||||
"block_size",
|
||||
"dtype",
|
||||
"enable_chunked_prefill",
|
||||
"enable_prefix_caching",
|
||||
"gpu_memory_utilization",
|
||||
"host",
|
||||
"kv_cache_dtype",
|
||||
"long_prefill_token_threshold",
|
||||
"max_long_partial_prefills",
|
||||
"max_model_len",
|
||||
"max_num_batched_tokens",
|
||||
"max_num_partial_prefills",
|
||||
"max_num_seqs",
|
||||
"pipeline_parallel_size",
|
||||
"port",
|
||||
"tensor_parallel_size",
|
||||
"trust_remote_code",
|
||||
},
|
||||
"tensorrt_llm": {
|
||||
"backend",
|
||||
"ep_size",
|
||||
"extra_llm_api_options",
|
||||
"host",
|
||||
"kv_cache_free_gpu_memory_fraction",
|
||||
"max_batch_size",
|
||||
"max_num_tokens",
|
||||
"max_seq_len",
|
||||
"port",
|
||||
"pp_size",
|
||||
"tp_size",
|
||||
"trust_remote_code",
|
||||
},
|
||||
}
|
||||
|
||||
HELP_FILE_HINTS = {
|
||||
"sglang": ("sglang", "launch"),
|
||||
"vllm": ("vllm", "serve"),
|
||||
"tensorrt_llm": ("trtllm", "serve"),
|
||||
}
|
||||
|
||||
|
||||
def flag_name(framework: str, key: str) -> str:
|
||||
if framework in {"sglang", "vllm"}:
|
||||
return "--" + key.replace("_", "-")
|
||||
return "--" + key
|
||||
|
||||
|
||||
def load_yaml(path: Path) -> dict[str, Any]:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = yaml.safe_load(f)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"{path}: expected a YAML mapping")
|
||||
return data
|
||||
|
||||
|
||||
def _as_list(value: Any) -> list[Any]:
|
||||
if isinstance(value, list):
|
||||
return value
|
||||
return [value]
|
||||
|
||||
|
||||
def _enabled(config: dict[str, Any], framework: str) -> bool:
|
||||
return bool(config.get("frameworks", {}).get(framework, {}).get("enabled", False))
|
||||
|
||||
|
||||
def _max_required_sequence(dataset: dict[str, Any]) -> int:
|
||||
input_len = dataset.get("input_len")
|
||||
output_len = dataset.get("output_len")
|
||||
if not isinstance(input_len, list) or not isinstance(output_len, list):
|
||||
raise ValueError("dataset.input_len and dataset.output_len must be lists")
|
||||
if len(input_len) != len(output_len):
|
||||
raise ValueError("dataset.input_len and dataset.output_len must be aligned")
|
||||
if not input_len:
|
||||
raise ValueError("dataset.input_len and dataset.output_len must not be empty")
|
||||
return max(int(i) + int(o) for i, o in zip(input_len, output_len, strict=True))
|
||||
|
||||
|
||||
def _candidate_dicts(
|
||||
base_flags: dict[str, Any],
|
||||
search_space: dict[str, Any],
|
||||
limit: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
candidates = [dict(base_flags)]
|
||||
keys = list(search_space)
|
||||
values = [_as_list(search_space[key]) for key in keys]
|
||||
for combo in itertools.product(*values):
|
||||
candidate = dict(base_flags)
|
||||
candidate.update(dict(zip(keys, combo, strict=True)))
|
||||
if candidate not in candidates:
|
||||
candidates.append(candidate)
|
||||
if len(candidates) >= limit:
|
||||
break
|
||||
return candidates
|
||||
|
||||
|
||||
def _command_tokens(
|
||||
framework: str,
|
||||
config: dict[str, Any],
|
||||
flags: dict[str, Any],
|
||||
) -> list[str]:
|
||||
server = config["frameworks"][framework]
|
||||
command = shlex.split(server["server_command"])
|
||||
model = config["model"]["name"]
|
||||
|
||||
if framework in {"vllm", "tensorrt_llm"}:
|
||||
command.append(model)
|
||||
|
||||
for key, value in flags.items():
|
||||
if value is None or value is False:
|
||||
continue
|
||||
command.append(flag_name(framework, key))
|
||||
if value is not True:
|
||||
command.append(str(value))
|
||||
|
||||
return command
|
||||
|
||||
|
||||
def render_command(
|
||||
framework: str, config: dict[str, Any], flags: dict[str, Any]
|
||||
) -> str:
|
||||
return shlex.join(_command_tokens(framework, config, flags))
|
||||
|
||||
|
||||
def _extract_help_flags(text: str) -> set[str]:
|
||||
return {
|
||||
item.lstrip("-") for item in re.findall(r"--[A-Za-z0-9][A-Za-z0-9_-]*", text)
|
||||
}
|
||||
|
||||
|
||||
def load_help_flags(help_dir: Path) -> dict[str, set[str]]:
|
||||
help_flags: dict[str, set[str]] = {}
|
||||
for framework, hints in HELP_FILE_HINTS.items():
|
||||
matches = []
|
||||
for path in help_dir.rglob("*.txt"):
|
||||
name = path.name.lower()
|
||||
if all(hint in name for hint in hints):
|
||||
matches.append(path)
|
||||
if matches:
|
||||
text = "\n".join(
|
||||
path.read_text(encoding="utf-8", errors="replace") for path in matches
|
||||
)
|
||||
help_flags[framework] = _extract_help_flags(text)
|
||||
return help_flags
|
||||
|
||||
|
||||
def _known_flag(
|
||||
framework: str,
|
||||
key: str,
|
||||
help_flags: dict[str, set[str]] | None,
|
||||
) -> bool:
|
||||
static_keys = STATIC_SERVER_FLAGS[framework]
|
||||
if key not in static_keys:
|
||||
return False
|
||||
if not help_flags or framework not in help_flags:
|
||||
return True
|
||||
|
||||
concrete = flag_name(framework, key).lstrip("-")
|
||||
aliases = {concrete, concrete.replace("-", "_"), concrete.replace("_", "-")}
|
||||
return bool(aliases & help_flags[framework])
|
||||
|
||||
|
||||
def _validate_framework(
|
||||
config: dict[str, Any],
|
||||
framework: str,
|
||||
help_flags: dict[str, set[str]] | None,
|
||||
max_candidates: int,
|
||||
) -> list[str]:
|
||||
errors: list[str] = []
|
||||
server = config["frameworks"].get(framework)
|
||||
if not isinstance(server, dict):
|
||||
return [f"missing frameworks.{framework}"]
|
||||
if not server.get("enabled", False):
|
||||
return []
|
||||
|
||||
base_flags = server.get("base_server_flags")
|
||||
search_space = server.get("search_space")
|
||||
if not isinstance(base_flags, dict):
|
||||
errors.append(f"{framework}: base_server_flags must be a mapping")
|
||||
base_flags = {}
|
||||
if not isinstance(search_space, dict):
|
||||
errors.append(f"{framework}: search_space must be a mapping")
|
||||
search_space = {}
|
||||
server_command_is_valid = isinstance(server.get("server_command"), str)
|
||||
if not server_command_is_valid:
|
||||
errors.append(f"{framework}: server_command must be a string")
|
||||
|
||||
for key in set(base_flags) | set(search_space):
|
||||
if not _known_flag(framework, key, help_flags):
|
||||
errors.append(f"{framework}: unknown or unsupported server flag {key!r}")
|
||||
|
||||
if framework == "tensorrt_llm":
|
||||
if server.get("backend_policy") != "fixed_pytorch":
|
||||
errors.append("tensorrt_llm: backend_policy must be fixed_pytorch")
|
||||
if base_flags.get("backend") != "pytorch":
|
||||
errors.append("tensorrt_llm: base backend must be pytorch")
|
||||
if "backend" in search_space:
|
||||
errors.append("tensorrt_llm: backend must not appear in search_space")
|
||||
|
||||
candidates = _candidate_dicts(base_flags, search_space, max_candidates)
|
||||
if not candidates:
|
||||
errors.append(f"{framework}: no candidates generated")
|
||||
can_render = server_command_is_valid and isinstance(
|
||||
config.get("model", {}).get("name"), str
|
||||
)
|
||||
if can_render:
|
||||
for candidate in candidates:
|
||||
command = render_command(framework, config, candidate)
|
||||
if not command:
|
||||
errors.append(f"{framework}: rendered an empty command")
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
def validate_config(
|
||||
path: Path,
|
||||
help_flags: dict[str, set[str]] | None = None,
|
||||
) -> list[str]:
|
||||
errors: list[str] = []
|
||||
try:
|
||||
config = load_yaml(path)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return [str(exc)]
|
||||
|
||||
if config.get("schema_version") != 1:
|
||||
errors.append("schema_version must be 1")
|
||||
if not isinstance(config.get("model", {}).get("name"), str):
|
||||
errors.append("model.name must be set")
|
||||
if config.get("source", {}).get("kind") not in ALLOWED_SOURCE_KINDS:
|
||||
errors.append(f"source.kind must be one of {sorted(ALLOWED_SOURCE_KINDS)}")
|
||||
|
||||
try:
|
||||
required_sequence = _max_required_sequence(config["dataset"])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
errors.append(str(exc))
|
||||
required_sequence = 0
|
||||
|
||||
search = config.get("search")
|
||||
if not isinstance(search, dict):
|
||||
errors.append("search must be a mapping")
|
||||
max_candidates = 1
|
||||
else:
|
||||
try:
|
||||
max_candidates = int(search.get("max_candidates_per_framework", 0))
|
||||
except (TypeError, ValueError):
|
||||
errors.append("search.max_candidates_per_framework must be an integer")
|
||||
max_candidates = 1
|
||||
if max_candidates < 1:
|
||||
errors.append("search.max_candidates_per_framework must be positive")
|
||||
max_candidates = 1
|
||||
|
||||
frameworks = config.get("frameworks")
|
||||
if not isinstance(frameworks, dict):
|
||||
return errors + ["frameworks must be a mapping"]
|
||||
|
||||
for framework in FRAMEWORKS:
|
||||
errors.extend(
|
||||
_validate_framework(config, framework, help_flags, max_candidates)
|
||||
)
|
||||
|
||||
for framework in FRAMEWORKS:
|
||||
if not _enabled(config, framework):
|
||||
continue
|
||||
key = SEQUENCE_LIMIT_KEY[framework]
|
||||
fw = frameworks[framework]
|
||||
base_flags = fw.get("base_server_flags", {}) or {}
|
||||
search_space = fw.get("search_space", {}) or {}
|
||||
if not isinstance(base_flags, dict) or not isinstance(search_space, dict):
|
||||
continue
|
||||
|
||||
try:
|
||||
if framework == "sglang":
|
||||
base_value = int(base_flags.get(key, required_sequence))
|
||||
else:
|
||||
base_value = int(base_flags.get(key, 0))
|
||||
except (TypeError, ValueError):
|
||||
errors.append(f"{framework}: base {key} is not an integer")
|
||||
continue
|
||||
if base_value < required_sequence:
|
||||
errors.append(
|
||||
f"{framework}: base {key} ({base_value}) is smaller than the largest dataset scenario ({required_sequence})"
|
||||
)
|
||||
|
||||
if key in search_space:
|
||||
for value in _as_list(search_space[key]):
|
||||
try:
|
||||
if int(value) < required_sequence:
|
||||
errors.append(
|
||||
f"{framework}: search_space {key} candidate {value} is smaller than the largest dataset scenario ({required_sequence})"
|
||||
)
|
||||
except (TypeError, ValueError):
|
||||
errors.append(
|
||||
f"{framework}: search_space {key} candidate {value!r} is not an integer"
|
||||
)
|
||||
|
||||
sla_block = (
|
||||
config.get("benchmark", {}).get("sla")
|
||||
if isinstance(config.get("benchmark"), dict)
|
||||
else None
|
||||
)
|
||||
if sla_block is None:
|
||||
sla_block = config.get("sla")
|
||||
if isinstance(sla_block, dict):
|
||||
for key in sla_block:
|
||||
if key in DEPRECATED_SLA_KEYS:
|
||||
errors.append(
|
||||
f"sla: {key!r} is deprecated; use {DEPRECATED_SLA_KEYS[key]!r} (see references/result-schema.md)"
|
||||
)
|
||||
elif key not in ALLOWED_SLA_KEYS:
|
||||
errors.append(
|
||||
f"sla: unknown key {key!r}; allowed keys are {sorted(ALLOWED_SLA_KEYS)}"
|
||||
)
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
def iter_config_files(paths: list[Path]) -> list[Path]:
|
||||
files: list[Path] = []
|
||||
for path in paths:
|
||||
if path.is_dir():
|
||||
files.extend(sorted(path.rglob("*.yaml")))
|
||||
files.extend(sorted(path.rglob("*.yml")))
|
||||
else:
|
||||
files.append(path)
|
||||
return sorted(dict.fromkeys(files))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("paths", nargs="+", type=Path)
|
||||
parser.add_argument("--help-dir", type=Path)
|
||||
parser.add_argument("--print-commands", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
help_flags = load_help_flags(args.help_dir) if args.help_dir else None
|
||||
failed = False
|
||||
for path in iter_config_files(args.paths):
|
||||
errors = validate_config(path, help_flags)
|
||||
if errors:
|
||||
failed = True
|
||||
for error in errors:
|
||||
print(f"{path}: {error}")
|
||||
continue
|
||||
|
||||
if args.print_commands:
|
||||
config = load_yaml(path)
|
||||
limit = int(config["search"].get("max_candidates_per_framework", 1))
|
||||
for framework in FRAMEWORKS:
|
||||
if not _enabled(config, framework):
|
||||
continue
|
||||
server = config["frameworks"][framework]
|
||||
candidates = _candidate_dicts(
|
||||
server["base_server_flags"],
|
||||
server["search_space"],
|
||||
limit,
|
||||
)
|
||||
print(f"# {path.name} {framework}")
|
||||
print(render_command(framework, config, candidates[0]))
|
||||
|
||||
if failed:
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: llm-torch-profiler-analysis
|
||||
description: "Unified LLM torch-profiler triage skill for `sglang`, `vllm`, and `TensorRT-LLM`. Use it to inspect an existing `trace.json(.gz)` or profile directory, or to drive live profiling against a running server and return one three-table report with kernel, overlap-opportunity, and fuse-pattern tables."
|
||||
description: "Unified LLM torch-profiler triage skill for `sglang`, `vllm`, `TensorRT-LLM`, and `TokenSpeed`. Use it to inspect an existing `trace.json(.gz)` or profile directory, or to drive live profiling against a running server when supported and return one three-table report with kernel, overlap-opportunity, and fuse-pattern tables."
|
||||
---
|
||||
|
||||
# Unified LLM Torch Profiler Analysis
|
||||
@@ -12,6 +12,7 @@ Use this skill for `torch.profiler` analysis across:
|
||||
- `sglang`
|
||||
- `vllm`
|
||||
- `TensorRT-LLM`
|
||||
- `TokenSpeed`
|
||||
|
||||
There is only one public workflow:
|
||||
|
||||
@@ -50,19 +51,27 @@ add one short note after the tables with exactly one of:
|
||||
|
||||
## Capability Matrix
|
||||
|
||||
| Capability | SGLang | vLLM | TensorRT-LLM |
|
||||
| --- | --- | --- | --- |
|
||||
| Existing trace triage | yes | yes | yes |
|
||||
| Single-trace live capture | yes | yes, if torch profiler is enabled on server | requires profiler control endpoints |
|
||||
| Two-trace mapping+formal triage | yes | yes | yes |
|
||||
| Stage-separated live workload | yes | yes | yes, with a writable shared trace dir or per-stage host runner |
|
||||
| `--profile-by-stage` capture | yes | no | no |
|
||||
| `--profile-prefix` control | yes | usually ignored on HTTP profiler route | usually ignored on HTTP profiler route |
|
||||
| Capability | SGLang | vLLM | TensorRT-LLM | TokenSpeed |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Existing trace triage | yes | yes | yes | yes |
|
||||
| Single-trace live capture | yes | yes, if torch profiler is enabled on server | requires profiler control endpoints | yes, if `/start_profile` and `/stop_profile` are exposed |
|
||||
| Two-trace mapping+formal triage | yes | yes | yes | yes |
|
||||
| Stage-separated live workload | yes | yes | yes, with a writable shared trace dir or per-stage host runner | yes, via workload-separated HTTP capture |
|
||||
| `--profile-by-stage` capture | yes | no | no | no |
|
||||
| `--profile-prefix` control | yes | usually ignored on HTTP profiler route | usually ignored on HTTP profiler route | yes, mapped to `profile_id` |
|
||||
|
||||
For TensorRT-LLM, live capture only works when the server exposes `/start_profile` and
|
||||
`/stop_profile`, and when the deployment already provides a shared trace path plus the
|
||||
required env vars.
|
||||
|
||||
For TokenSpeed, this skill supports both existing trace triage and live capture
|
||||
against current servers that expose `/start_profile` and `/stop_profile`.
|
||||
The live helper sends `output_dir`, `activities`, `with_stack`,
|
||||
`record_shapes`, and `profile_id` in the start payload. TokenSpeed also has its
|
||||
own native `profile_by_stage` field for manual capture, but the unified helper
|
||||
uses workload-separated `prefill/` and `decode/` directories by default so the
|
||||
tables stay comparable across frameworks.
|
||||
|
||||
## Real H100 Validation
|
||||
|
||||
The current reference run is the `4x H100` matrix captured on `2026-04-23` on
|
||||
@@ -98,6 +107,9 @@ Validated matrix:
|
||||
Use this run as the main H100 reference.
|
||||
The older `2026-04-22` single-card Qwen3 matrix is still useful for bring-up, but it is
|
||||
not the default reference anymore.
|
||||
TokenSpeed support was added later and is covered by existing-trace triage and
|
||||
HTTP profiler-control support, but it is not part of this older H100 validation
|
||||
matrix yet.
|
||||
|
||||
Stage-separated workload validation captured on `2026-05-01` on `h100_sglang`:
|
||||
|
||||
@@ -145,12 +157,14 @@ H100 notes:
|
||||
- SGLang kernel-site reconstruction keeps sampling disabled in the mapping path so the optimized parser does not perturb SGLang table output; equality rechecks matched for `Mixtral-8x7B-Instruct-v0.1`, `Qwen3-32B`, and `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`
|
||||
- vLLM live capture requires `--output-dir` to match the server `torch_profiler_dir`; the validated H100 flow uses `--profiler-config {"profiler":"torch","torch_profiler_dir":"..."}` and then drives `/start_profile` and `/stop_profile`
|
||||
- TensorRT-LLM validation stays on `--backend pytorch`; the H100 flow writes the trace with `TLLM_TORCH_PROFILE_TRACE` and then analyzes the saved trace
|
||||
- the 2026-04-22 TensorRT-LLM 1.0.0 `py_executor.py` profiler setup still needed a `with_stack=True` override for table-quality Python locations, and the matrix runner generated that override under `/data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm`; re-check this on TensorRT-LLM 1.2.1 or any 1.3.x release-candidate image before assuming the override is still required
|
||||
- TensorRT-LLM current mainline was rechecked at `0722c5f47d2cae69ac1a237da51e550dd214532c` on 2026-06-26; the latest delta affects KV eviction / block-offset staging rather than profiler trace controls, so the `b9e1945` profiler evidence still applies: PyTorch profiling uses `record_shapes=True` and `with_modules=True`, but not `with_stack=True`; keep the override path for table-quality Python locations unless the target image proves otherwise
|
||||
- TokenSpeed trace analysis has first-class registry rows for native TokenSpeed CuTe DSL MLA, MLA KV pack + FP8 quantize, fused top-k/top-p sampling, persistent lm_head GEMM, and NVFP4 GEMM + SwiGLU + quant; live capture still requires an existing torch-profiler trace until the target TokenSpeed image exposes a supported profiler API
|
||||
- on this host, keep all trace roots under `/data/...`, not `/home/...`
|
||||
|
||||
## When To Use It
|
||||
|
||||
- inspect a `torch.profiler` trace or profile directory from `sglang`, `vllm`, or `TensorRT-LLM`
|
||||
- inspect a `torch.profiler` trace or profile directory from `sglang`, `vllm`,
|
||||
`TensorRT-LLM`, or `TokenSpeed`
|
||||
- profile a live serving endpoint and analyze the result
|
||||
- summarize which kernel families dominate prefill or decode
|
||||
- map kernels back to Python code paths
|
||||
@@ -202,7 +216,7 @@ Allowed `--profile-workload` values:
|
||||
- `decode`: capture only the one-input / long-output workload
|
||||
- `legacy`: keep the old `--probe-prompt` / `--probe-max-new-tokens` behavior
|
||||
|
||||
For `sglang-sota-performance`, do not use the defaults if the slow SGLang
|
||||
For `sglang-sota-humanize-loop`, do not use the defaults if the slow SGLang
|
||||
benchmark scenario has a known input/output distribution.
|
||||
Set the profiler lengths from that slow scenario instead: prefill uses the slow
|
||||
input length with output `1`, and decode uses input `1` with the slow output
|
||||
@@ -283,7 +297,7 @@ and the trace path is shared with the current machine.
|
||||
|
||||
Typical env expectations are:
|
||||
|
||||
- `TLLM_PROFILE_START_STOP=1`
|
||||
- `TLLM_PROFILE_START_STOP=<start>-<stop>` such as `10-20`
|
||||
- `TLLM_TORCH_PROFILE_TRACE=/shared/path/trace.json` or `.json.gz`
|
||||
|
||||
Then run:
|
||||
@@ -318,11 +332,101 @@ The matrix runner does this automatically on H100 before TensorRT-LLM capture st
|
||||
|
||||
This is the validated TensorRT-LLM flow on `h100_sglang`:
|
||||
|
||||
1. launch `trtllm-serve` with `TLLM_TORCH_PROFILE_TRACE=/data/.../trace.json`
|
||||
1. launch `trtllm-serve` with `TLLM_PROFILE_START_STOP=<start>-<stop>` and `TLLM_TORCH_PROFILE_TRACE=/data/.../trace.json`
|
||||
2. run a few benchmark requests
|
||||
3. analyze the emitted trace with `--input /data/.../trace.json`
|
||||
|
||||
### 5. Two-trace triage from existing profile dirs or traces
|
||||
### 5. Single-trace live capture or triage from TokenSpeed
|
||||
|
||||
For a running TokenSpeed server that exposes the profiler routes, the unified
|
||||
helper can drive live capture:
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
--framework tokenspeed \
|
||||
--url http://127.0.0.1:8000 \
|
||||
--output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/tokenspeed_profile \
|
||||
--num-steps 5 \
|
||||
--warmup-steps 10 \
|
||||
--no-profile-by-stage \
|
||||
--profile-workload both \
|
||||
--profile-prefix ts-triage
|
||||
```
|
||||
|
||||
The helper sends `POST /start_profile` with:
|
||||
|
||||
- `output_dir`: the `--output-dir` path
|
||||
- `activities`: `["CPU", "GPU"]`
|
||||
- `with_stack`: `true`
|
||||
- `record_shapes`: `false`
|
||||
- `profile_id`: `--profile-prefix`, with `-prefill` or `-decode` appended during workload-separated capture
|
||||
|
||||
It then sends OpenAI-compatible probe requests and calls `POST /stop_profile`.
|
||||
TokenSpeed writes files such as `ts-triage-prefill-TP-0.trace.json.gz` under the
|
||||
output directory. If the server was launched with multiple TP ranks, expect one
|
||||
trace per rank.
|
||||
|
||||
Existing TokenSpeed torch-profiler traces can still be analyzed directly:
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
--framework tokenspeed \
|
||||
--input /path/to/tokenspeed_profile_dir_or_trace.json.gz
|
||||
```
|
||||
|
||||
TokenSpeed's own manual profiler control surface can also be used:
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:8000/start_profile \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"output_dir":"/data/bbuf/profiles/tokenspeed","activities":["CPU","GPU"],"with_stack":true,"record_shapes":false,"profile_id":"ts-manual"}'
|
||||
|
||||
# send representative workload here
|
||||
|
||||
curl -X POST http://127.0.0.1:8000/stop_profile
|
||||
```
|
||||
|
||||
For server-side automatic stop, pass `num_steps`. For TokenSpeed-native
|
||||
EXTEND/DECODE split, pass `profile_by_stage: true`; this produces files with
|
||||
stage suffixes such as `-EXTEND` and `-DECODE`.
|
||||
|
||||
TokenSpeed's benchmark driver can capture traces too:
|
||||
|
||||
```bash
|
||||
tokenspeed bench serve \
|
||||
--base-url http://127.0.0.1:8000 \
|
||||
--model <model> \
|
||||
--dataset-name random \
|
||||
--random-input-len 4090 \
|
||||
--random-output-len 1 \
|
||||
--num-prompts 64 \
|
||||
--profile \
|
||||
--profile-num-steps 5 \
|
||||
--extra-body '{"output_dir":"/data/bbuf/profiles/tokenspeed","activities":["CPU","GPU"],"with_stack":true,"profile_id":"ts-bench"}'
|
||||
```
|
||||
|
||||
If `output_dir` is omitted, TokenSpeed falls back to `TOKENSPEED_PROFILER_DIR`
|
||||
and then `/tmp`.
|
||||
|
||||
Use [scripts/probe_llm_server.py](scripts/probe_llm_server.py) with
|
||||
`--framework tokenspeed` for a small OpenAI-compatible endpoint probe before or
|
||||
after trace collection:
|
||||
|
||||
```bash
|
||||
python3 scripts/probe_llm_server.py \
|
||||
--framework tokenspeed \
|
||||
--url http://127.0.0.1:8000 \
|
||||
--requests 6 \
|
||||
--max-tokens 48
|
||||
```
|
||||
|
||||
For `sglang-sota-humanize-loop`, keep TokenSpeed profiler evidence aligned to
|
||||
the same slow scenario bucket as the benchmark result. Prefer the unified
|
||||
workload-separated live capture when possible; if only a mixed agentic trace is
|
||||
available, label that limitation in `analysis/root-cause.md` before comparing
|
||||
it to SGLang prefill/decode traces.
|
||||
|
||||
### 6. Two-trace triage from existing profile dirs or traces
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
@@ -332,7 +436,7 @@ python3 scripts/analyze_llm_torch_profile.py \
|
||||
|
||||
Use this when you need stronger overlap attribution and kernel-to-source mapping.
|
||||
|
||||
### 6. Two-trace triage from running servers
|
||||
### 7. Two-trace triage from running servers
|
||||
|
||||
```bash
|
||||
python3 scripts/analyze_llm_torch_profile.py \
|
||||
@@ -350,6 +454,10 @@ For `vllm` or `TensorRT-LLM`, use the same shape but pass:
|
||||
- `--formal-output-dir ...`
|
||||
- `--no-profile-by-stage`
|
||||
|
||||
For TokenSpeed, either use `--mapping-url` and `--formal-url` against servers
|
||||
that expose `/start_profile` and `/stop_profile`, or pass two existing trace
|
||||
directories with `--mapping-input` and `--formal-input`.
|
||||
|
||||
## `profile_by_stage`
|
||||
|
||||
`--profile-by-stage` is only meaningful on the SGLang live-capture path.
|
||||
@@ -363,7 +471,8 @@ For `vllm` or `TensorRT-LLM`, use the same shape but pass:
|
||||
bottlenecks.
|
||||
- On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
|
||||
- PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary `profile_by_stage`.
|
||||
- For `vllm` and `TensorRT-LLM`, disable it with `--no-profile-by-stage`.
|
||||
- For `vllm`, `TensorRT-LLM`, and `TokenSpeed`, disable it with
|
||||
`--no-profile-by-stage`.
|
||||
|
||||
## How To Choose The Triage Shape
|
||||
|
||||
|
||||
@@ -28,14 +28,18 @@ overlap opportunity as novel.
|
||||
|
||||
The catalog is grouped by reusable optimization family, not by one specific model.
|
||||
|
||||
Refresh note `2026-05-01`: rescanned current `sglang` and vLLM mainline, then
|
||||
rechecked recent merged and open optimization PRs through the GitHub CLI/API.
|
||||
The vLLM torch.compile pass inventory is now split out in
|
||||
Refresh note `2026-06-26`: rechecked official main heads for SGLang
|
||||
`8524678889485801e7a4a12d62015be0c68f7a90`, vLLM
|
||||
`abc71548ef029132c3316b902207f254a246d593`, TensorRT-LLM
|
||||
`0722c5f47d2cae69ac1a237da51e550dd214532c`, and TokenSpeed
|
||||
`5aedf69d6b476baa65571011de6ea60fd5a238a8`. The vLLM torch.compile pass
|
||||
inventory is split out in
|
||||
[`vllm-torch-compile-fusions.md`](vllm-torch-compile-fusions.md). Stable
|
||||
current-code families remain folded into the mainline rows below. New
|
||||
status-sensitive rows were added for DeepSeek-V4, GLM5 DSA / PDL, NVFP4 MoE,
|
||||
torch.compile decode, vLLM DSV4, vLLM ROCm WMMA, and vLLM GPU/CPU sync-removal
|
||||
work. Recheck PR state before treating an in-flight row as shipped.
|
||||
current-code families remain folded into the mainline rows below. This refresh
|
||||
adds first-class TokenSpeed-origin rows for CuTe DSL MLA, MLA KV pack+FP8
|
||||
quantize, sampling, lm_head GEMM, and NVFP4 GEMM+SwiGLU+quant, plus the latest
|
||||
SGLang LTX2 Ada-value diffusion fusion. Recheck PR state before treating an
|
||||
in-flight row as shipped.
|
||||
|
||||
## 1. LLM / SRT fused-kernel families
|
||||
|
||||
@@ -53,13 +57,13 @@ work. Recheck PR state before treating an in-flight row as shipped.
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/jit_kernel/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
||||
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
||||
| DSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_dsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/jit_kernel/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around DSA replay as a missed existing replay fusion. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/jit_kernel/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
||||
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
||||
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
||||
| NPU fused split-QKV + RMSNorm + RoPE | `split_qkv_rmsnorm_rope` | `python/sglang/srt/models/llama.py`<br>`python/sglang/srt/models/qwen3.py`<br>`python/sglang/srt/models/qwen3_moe.py`<br>`python/sglang/srt/models/glm4_moe.py` | Ascend path fuses QKV split, Q / K RMSNorm, and RoPE in one op | On NPU traces, separate split / norm / rope kernels usually mean the fused path is unavailable or bypassed. |
|
||||
| Fused FP8 quantize + paged KV cache write | `trtllm_fp8_kv_kernel`<br>`fp8 kv cache write`<br>`paged KV cache write` | `python/sglang/srt/layers/attention/triton_ops/trtllm_fp8_kv_kernel.py` | TRTLLM MHA path fuses FP8 quantization, scale computation, and paged K / V cache write | If FP8 KV cache traces show standalone quant plus write kernels, first compare against this existing Triton fuse. |
|
||||
| Fused MLA KV cache write + FP8 quant | `set_mla_kv_buffer_fp8_quant*`<br>`set_mla_kv_buffer_triton_fp8_quant` | `python/sglang/srt/mem_cache/utils.py`<br>`python/sglang/srt/mem_cache/memory_pool.py` | MLA / DSA KV pool path can quantize K and write directly into KV storage without a separate concat-and-quant chain | Treat standalone quant + KV-buffer write on MLA paths as missing existing fusion first. |
|
||||
| Fused MLA KV cache write + FP8 quant | `set_mla_kv_buffer_fp8_quant*`<br>`set_mla_kv_buffer_triton_fp8_quant` | `python/sglang/srt/mem_cache/utils.py`<br>`python/sglang/srt/mem_cache/memory_pool.py` | MLA / NSA KV pool path can quantize K and write directly into KV storage without a separate concat-and-quant chain | Treat standalone quant + KV-buffer write on MLA paths as missing existing fusion first. |
|
||||
| Fused MoE router / top-k / softcapping | `FusedMoeRouter`<br>`fused_moe_router*`<br>router GEMM + `topk` + `tanh` | `python/sglang/srt/layers/moe/router.py` | Single fused router kernel covers router matmul, softcapping, and top-k selection | Treat exposed router matmul + softcap + top-k chains as an existing MoE fusion family. |
|
||||
| Fused MoE grouped-topk / gate kernels | `fused_topk_deepseek`<br>`moe_fused_gate`<br>`aiter_fused_topk`<br>`kimi_k2_moe_fused_gate` | `python/sglang/srt/layers/moe/topk.py` | CUDA / ROCm / FlashInfer kernels fuse bias, grouped-topk, renorm, and routed scaling into one gate op | Check backend / model eligibility before proposing a novel router-gate fusion. |
|
||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||
@@ -67,8 +71,8 @@ work. Recheck PR state before treating an in-flight row as shipped.
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/jit_kernel/nvfp4.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/fused_moe_method_npu.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| DSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | DSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If DSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||
| DSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/jit_kernel/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/dsa/dsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into DSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/jit_kernel/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/jit_kernel/triton/gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
||||
@@ -90,7 +94,7 @@ work. Recheck PR state before treating an in-flight row as shipped.
|
||||
| Llama4 shared branch vs routed branch overlap | shared expert branch plus routed MoE branch as adjacent windows | `python/sglang/srt/models/llama4.py` | shared expert on current stream, router + topk + routed experts on `alt_stream` | Use Llama4 as the first precedent for branch-level overlap in similar sparse models. |
|
||||
| ExaoneMoE shared experts vs router experts overlap | shared expert output and router-expert output form a two-branch window | `python/sglang/srt/models/exaone_moe.py::forward_normal_dual_stream` | shared experts on current stream, router + routed experts on `alt_stream`, explicit join before combine | This is an existing dual-stream MoE overlap family. |
|
||||
| Grok residual-MoE branch overlap | dense MLP and block-sparse MoE branches in parallel | `python/sglang/srt/models/grok.py::moe_with_rmoe` | dense MLP on current stream, MoE on `alt_stream`, fused dual residual RMSNorm around boundaries | Treat exposed Grok branch overlap as an existing pattern. |
|
||||
| DSA dual-stream overlap | Q-proj, K-proj, RoPE, cache-store, quantization in tight two-stream windows | `python/sglang/srt/layers/attention/dsa/dsa_indexer.py` | Q / K projection split, RoPE split, cache-store vs quantization overlap | DSA already contains several dual-stream overlap precedents. |
|
||||
| NSA dual-stream overlap | Q-proj, K-proj, RoPE, cache-store, quantization in tight two-stream windows | `python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Q / K projection split, RoPE split, cache-store vs quantization overlap | NSA already contains several dual-stream overlap precedents. |
|
||||
| MoriEP async dispatch / combine comm stream | `MoriEP`<br>`_comm_stream`<br>`dispatch`<br>`combine`<br>`done_event` | `python/sglang/srt/layers/moe/token_dispatcher/moriep.py` | MoriEP can submit dispatch and combine onto a dedicated communication stream and synchronize only through events | Treat MoriEP comm / compute interleave as an existing MoE overlap family. |
|
||||
| Heterogeneous-TP staging scatter overlap | `scatter_stream`<br>`_scatter_stream`<br>`staging` | `python/sglang/srt/disaggregation/common/staging_handler.py`<br>`python/sglang/srt/disaggregation/common/staging_buffer.py` | decode-side staging scatter kernels can run on a dedicated stream while forward continues on the main stream | If decode traces show staging scatter kernels adjacent to forward kernels, classify them against this existing overlap family first. |
|
||||
| Generic `alt_stream` overlap families | `alt_stream` plus explicit `wait_stream` / `with torch.cuda.stream(...)` | `qwen2_moe.py`<br>`qwen3_moe.py`<br>`glm4_moe.py`<br>`bailing_moe.py`<br>`llada2.py`<br>`grok.py`<br>`olmo2.py`<br>`step3p5.py`<br>`longcat_flash.py`<br>`falcon_h1.py` | model-specific overlap on attention prep, MoE branches, or cache-store | Search these families before designing a new overlap scheme from scratch. |
|
||||
@@ -114,6 +118,8 @@ work. Recheck PR state before treating an in-flight row as shipped.
|
||||
| Fused diffusion QK norm + RoPE | split QK norm and RoPE in diffusion attention blocks | `python/sglang/jit_kernel/diffusion/qknorm_rope.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py::apply_qk_norm_rope` | `fused_inplace_qknorm_rope(...)`, with fallback to QK norm plus `apply_flashinfer_rope_qk_inplace(...)` | Distinguish between missing fused qknorm + rope and the existing FlashInfer RoPE fallback. |
|
||||
| Z-Image fused `norm(x) * tanh(scale) + shift` | `fused_norm_tanh_mul_add`<br>`tanh(gate) * rmsnorm(x)` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py` | CuTeDSL kernel plus runtime helper for Z-Image residual-form modulation | Treat split Z-Image residual-form modulation as a missing existing diffusion fusion, not a novel idea. |
|
||||
| Z-Image fused residual modulation + next norm-scale | `fused_norm_tanh_mul_add_norm_scale`<br>`residual + tanh(gate) * rmsnorm(x)`<br>`ffn_norm1(x) * scale_mlp` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/models/dits/zimage.py` | One CuTeDSL kernel fuses the first residual-form modulation and the next normalization / scale stage | If you see this chain split in Z-Image traces, report it as a missing existing mainline fusion family. |
|
||||
| LTX2 fused Ada values | `ltx2_ada_values9`<br>`get_ada_values`<br>`scale_shift_table + timestep.reshape` | `python/sglang/jit_kernel/diffusion/triton/ltx2_ada_values.py`<br>`python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py` | PR `#29390` fuses LTX-2.3 Ada value materialization for video/audio streams and reuses the 9 Ada tensors across self-attention, MLP, and prompt-cross-attention blocks | Treat repeated Ada add/reshape/slice ladders in LTX2 traces as a missing shipped SGLang fusion first. |
|
||||
| LTX2 residual-gate add | `diffusion_residual_gate_add`<br>`residual_gate_add`<br>`residual + update * gate` | `python/sglang/jit_kernel/diffusion/residual_gate_add.py`<br>`python/sglang/jit_kernel/csrc/diffusion/residual_gate_add.cuh`<br>`python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py` | PR `#29361` fuses LTX2 `residual + update * gate` sites for attention, cross-attention, and feed-forward updates into one CUDA custom op when dtype, shape, device, and contiguity guards pass | Treat split add/mul gate ladders in LTX2 traces as a missing shipped SGLang fusion first. |
|
||||
| Nunchaku fused GELU MLP | `_fused_gelu_mlp`<br>`fused_gelu_mlp` | `python/sglang/multimodal_gen/runtime/models/dits/flux.py` | Nunchaku path fuses `fc1 GEMM + GELU + shift + re-quant + fc2.lora_down` before the second GEMM | Treat split GELU-MLP on Nunchaku checkpoints as an existing fused family, not a new discovery. |
|
||||
|
||||
## 5. Diffusion kernel-overlap and async-communication families
|
||||
@@ -134,16 +140,16 @@ Stable entries should be folded into the mainline family rows above.
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#21877` fused grouped down-GEMM + combine | `grouped_gemm_nt_masked`<br>`combine`<br>`fused grouped gemm combine` | `PR #21877`<br>`python/sglang/srt/layers/moe/ep_moe/flashinfer_cutedsl_moe.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | FlashInfer CuTeDSL kernel fuses the second expert GEMM with DeepEP low-latency combine | Treat this as a concrete upstream MoE fuse / overlap family, not a new thought experiment. |
|
||||
| PR `#21889` fused BF16 to FP4 quant + paged KV write | `set_mla_kv_buffer_fp4_quant_kernel`<br>`fp4 kv cache` | `PR #21889`<br>`python/sglang/srt/mem_cache/utils.py` | Triton kernel writes FP4 DSA KV pages directly while quantizing BF16 input | If DSA FP4 KV paths are split into quant plus store, classify them as an in-flight upstream fuse family. |
|
||||
| PR `#21889` fused FP4 paged dequant to FP8 + page-table remap | `_dequant_fp4_to_fp8_paged_kernel`<br>`WRITE_PT`<br>`dequant_fp4_paged_decode` | `PR #21889`<br>`python/sglang/srt/layers/attention/dsa/dequant_fp4_to_fp8.py` | Triton kernel reads FP4 pages, writes FP8 directly, and can fuse decode-side page-table remap | Treat this as an upstream in-flight decode-prep fusion family. |
|
||||
| PR `#21889` fused BF16 to FP4 quant + paged KV write | `set_mla_kv_buffer_fp4_quant_kernel`<br>`fp4 kv cache` | `PR #21889`<br>`python/sglang/srt/mem_cache/utils.py` | Triton kernel writes FP4 NSA KV pages directly while quantizing BF16 input | If NSA FP4 KV paths are split into quant plus store, classify them as an in-flight upstream fuse family. |
|
||||
| PR `#21889` fused FP4 paged dequant to FP8 + page-table remap | `_dequant_fp4_to_fp8_paged_kernel`<br>`WRITE_PT`<br>`dequant_fp4_paged_decode` | `PR #21889`<br>`python/sglang/srt/layers/attention/nsa/dequant_fp4_to_fp8.py` | Triton kernel reads FP4 pages, writes FP8 directly, and can fuse decode-side page-table remap | Treat this as an upstream in-flight decode-prep fusion family. |
|
||||
| PR `#21491` FlashInfer TRTLLM FP8 MoE with fused shared experts | `num_fused_shared_experts`<br>`trtllm_fp8_block_scale_moe` | `PR #21491`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/models/deepseek_v2.py` | FlashInfer TRTLLM FP8 MoE path can fuse shared experts inside the routed MoE kernel | On FP8 TRTLLM MoE discussions, treat fused shared experts as an upstream pattern that already has a concrete PR. |
|
||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/jit_kernel/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/jit_kernel/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`sgl-kernel/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/jit_kernel/nvfp4.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||
| PR `#22851` DSA top-k backend and FlashInfer / PyTorch top-k split | `dsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes DSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When DSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 DSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`DSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 DSA TileLang decode path | Treat cat/copy bursts in GLM5 DSA decode as a concrete in-flight cleanup opportunity. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
|
||||
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
|
||||
|
||||
@@ -237,24 +243,43 @@ contain the same implementation.
|
||||
| vLLM-origin RMSNorm (+ residual add) + quant | `RMSNormQuantFusionPass`<br>`fused_add_rms_norm_static_fp8_quant`<br>`per_token_quant`<br>`per_group_quant` | `vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/compilation/passes/fusion/rocm_aiter_fusion.py` | Compile-time and ROCm AITER paths fuse RMSNorm or fused-add-RMSNorm with FP8 / FP4 quant output | Treat split norm/add + quant as an upstream fused family, not an unexplored direction. |
|
||||
| vLLM-origin SiLU+Mul + quant | `ActivationQuantFusionPass`<br>`SiluMulFp8*`<br>`Nvfp4`<br>`rocm_aiter` | `vllm/compilation/passes/fusion/act_quant_fusion.py`<br>`vllm/compilation/passes/fusion/rocm_aiter_fusion.py` | Activation epilogues fuse `SiLU+Mul` with FP8 / NVFP4 / AITER group quant instead of materializing the BF16 activation first | Treat standalone activation then quant kernels as matching a vLLM-origin precedent. |
|
||||
| vLLM-origin add + RMSNorm + pad | `fuse_act_padding`<br>`RocmAiterTritonAddRMSNormPadFusionPass`<br>`add_rmsnorm_pad` | `vllm/compilation/passes/fusion/rocm_aiter_fusion.py`<br>`docs/design/fusions.md` | ROCm / AITER path fuses residual add + RMSNorm directly into the padded layout expected by the next kernel | Treat norm-plus-padding ladders as an existing backend-specific fuse family first. |
|
||||
| vLLM-origin attention + output quant | `fuse_attn_quant`<br>`AttnQuantFusionPass`<br>`merge_attn_states`<br>`output_scale`<br>`output_group_scale`<br>`output_block_scale` | `vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/ops/merge_attn_states.py`<br>`vllm/csrc/attention/merge_attn_states.cu`<br>`docs/design/fusions.md` | Compile-time fusion pushes FP8 / NVFP4 quantization into the attention epilogue on supported Triton / FlashInfer / ROCm / AITER backends, and mainline `merge_attn_states` kernels already support FP8 output when `output_scale` is provided | Treat attention-output quant and merged-attention quant epilogues as a known upstream family before calling them novel. |
|
||||
| vLLM-origin attention + output quant | `fuse_attn_quant`<br>`AttnQuantFusionPass`<br>`merge_attn_states`<br>`output_scale`<br>`output_group_scale`<br>`output_block_scale` | `vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/ops/merge_attn_states.py`<br>`csrc/attention/merge_attn_states.cu`<br>`docs/design/fusions.md` | Compile-time fusion pushes FP8 / NVFP4 quantization into the attention epilogue on supported Triton / FlashInfer / ROCm / AITER backends, and mainline `merge_attn_states` kernels already support FP8 output when `output_scale` is provided | Treat attention-output quant and merged-attention quant epilogues as a known upstream family before calling them novel. |
|
||||
| vLLM-origin fused QK RMSNorm + RoPE | `fused_qk_norm_rope`<br>`QKNormRoPEFusionPass`<br>`qk norm + rope` | `vllm/compilation/passes/fusion/qk_norm_rope_fusion.py`<br>`vllm/_custom_ops.py`<br>`csrc/fused_qknorm_rope_kernel.cu` | Compile-time and direct custom-op paths fuse per-head Q / K RMSNorm with RoPE | Treat split QK norm + RoPE as a clear vLLM-origin precedent. |
|
||||
| vLLM-origin fused reshape + KV cache write | `reshape_and_cache`<br>`triton_reshape_and_cache_flash`<br>`kv cache write` | `vllm/v1/attention/ops/triton_reshape_and_cache_flash.py`<br>`vllm/v1/attention/backends/triton_attn.py` | Triton cache-update kernels reshape K / V into paged-cache layout and can include FP8 KV-cache scale/write logic | Treat reshape / transpose / cache-write ladders as an existing cache-store fusion family. |
|
||||
| vLLM-origin fused RoPE + KV cache update | `fuse_rope_kvcache`<br>`RopeKVCacheFusionPass`<br>`triton_rope_and_cache` | `vllm/compilation/passes/fusion/rope_kvcache_fusion.py`<br>`vllm/_aiter_ops.py`<br>`docs/design/fusions.md` | ROCm / AITER compile-time fusion combines RoPE with paged KV cache update instead of launching them separately | Treat split RoPE + cache-store as a known upstream family, especially on ROCm-like paths. |
|
||||
| vLLM-origin fused MLA RoPE + concat/cache write | `concat_and_cache_mla_rope_fused`<br>`mla rope cache` | `vllm/_custom_ops.py`<br>`csrc/cache_kernels_fused.cu` | CUDA kernel fuses MLA-oriented RoPE preparation, concat, and cache write into a direct paged-store path | Treat MLA concat + cache-write ladders as a vLLM-origin precedent before calling them novel. |
|
||||
| vLLM-origin fused MLA RoPE + unified KV-cache update | `fused_rope_unified_mla_kv_cache_update`<br>`concat_and_cache_mla_rope_fused`<br>`unified_mla_kv_cache_update` | `vllm/compilation/passes/fusion/mla_rope_kvcache_cat_fusion.py`<br>`vllm/_custom_ops.py`<br>`csrc/cache_kernels_fused.cu` | Current vLLM compile pass fuses MLA-oriented RoPE on `q_pe` / `k_pe`, concat, and unified MLA KV-cache update into a direct paged-store path | Treat MLA RoPE + concat + cache-write ladders as a vLLM-origin mainline precedent before calling them novel. |
|
||||
| vLLM-origin fused grouped top-k / biased grouped top-k router | `grouped_topk`<br>`biased_grouped_topk`<br>`grouped_topk_fused_kernel` | `vllm/_custom_ops.py`<br>`vllm/_aiter_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/grouped_topk_router.py`<br>`csrc/moe/grouped_topk_kernels.cu` | CUDA / ROCm router kernels fuse grouped score processing, top-k selection, and routed renorm / bias handling | Treat MoE router ladders as matching an upstream grouped-topk family first. |
|
||||
| vLLM-origin fused top-k softmax / sigmoid router | `topk_softmax`<br>`topk_sigmoid`<br>`topkGating`<br>`fused_topk` | `vllm/_custom_ops.py`<br>`vllm/_aiter_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/fused_topk_router.py`<br>`vllm/model_executor/layers/fused_moe/router/fused_topk_bias_router.py`<br>`csrc/moe/topk_softmax_kernels.cu` | CUDA and ROCm / AITER router kernels fuse score activation (`softmax` / `sigmoid`), top-k selection, optional bias correction, and routed renorm into one op instead of routing through grouped-topk or eager softmax-plus-topk ladders | Treat standalone score activation -> top-k -> bias / renorm chains as a known upstream fused router family first. |
|
||||
| vLLM-origin DSV3 router GEMM | `dsv3_router_gemm`<br>`allow_dsv3_router_gemm`<br>`router logits` | `vllm/_custom_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/dsv3_router_gemm_entry.cu`<br>`csrc/moe/dsv3_router_gemm_float_out.cu` | Hopper-class CUDA kernel specializes the DeepSeek router linear for small decode batches and can emit FP32 logits directly without a generic GEMM chain | Treat DeepSeek-style router linear paths as an existing upstream specialized fuse, distinct from grouped-topk itself. |
|
||||
| vLLM-origin DeepSeek-V4 fused norm + router GEMM | `dsv4_norm_router_gemm`<br>`norm_gate_linear`<br>`router_gemm`<br>`DeepseekV4ForCausalLM` | `vllm/model_executor/layers/fused_moe/router/norm_gate_linear.py`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`csrc/moe/dsv4_norm_router_gemm*` | Current vLLM mainline has a low-latency DeepSeek-V4 path that fuses norm/router-adjacent work into specialized DSV4 router GEMM kernels | Treat DSV4 norm + router ladders as an upstream mainline fused-router family. |
|
||||
| vLLM-origin DeepSeek-V4 MHC fused kernels | `mhc_post_pre`<br>`head_compute_mix_kernel`<br>`aiter mhc`<br>`MHC` | `vllm/model_executor/layers/mhc.py`<br>`vllm/model_executor/kernels/mhc/aiter.py`<br>`vllm/model_executor/models/deepseek_v4.py` | Current vLLM mainline includes Tile head-compute kernels, fused `mhc_post_pre`, and ROCm AITER MHC support for DSV4 | Treat DSV4 MHC/head-compute ladders as known upstream kernel families before proposing a new head-compute kernel. |
|
||||
| vLLM-origin GPT-OSS router GEMM | `gpt_oss_router_gemm`<br>`router gemm` | `vllm/_custom_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/gpt_oss_router_gemm.cu` | Model-specific CUDA kernel replaces the router linear plus bias path with one specialized GEMM op | Treat GPT-OSS-style router linear chains as an existing upstream specialized fuse. |
|
||||
| vLLM-origin DeepSeek min-latency fused QKV-A projection | `dsv3_fused_a_gemm`<br>`fused_qkv_a_proj`<br>`q_a_proj` | `vllm/model_executor/models/deepseek_v2.py`<br>`vllm/_custom_ops.py`<br>`csrc/dsv3_fused_a_gemm.cu` | Hopper-class CUDA kernel replaces the tiny-batch DeepSeek QKV-A projection path with one specialized min-latency GEMM instead of a generic linear launch | Treat small-batch DeepSeek QKV-A projection ladders as a known upstream fused kernel family first. |
|
||||
| vLLM-origin DSV3.2 fused indexer projections | `wk_weights_proj`<br>`MergedColumnParallelLinear`<br>`weights_proj` | `vllm/model_executor/models/deepseek_v2.py`<br>`vllm/model_executor/models/deepseek_mtp.py` | DSV3.2 indexer paths can fuse the `wk` and `weights_proj` projections into one GEMM and carry the matching MTP weight-loading path | Treat paired indexer projection chains as a known upstream fused linear family before calling the opportunity novel. |
|
||||
| vLLM-origin ROCm AITER sparse-MLA paged MQA logits | `rocm_aiter_mla_sparse`<br>`paged_mqa_logits`<br>`gluon`<br>`gfx950` | `vllm/v1/attention/ops/rocm_aiter_mla_sparse.py` | Current vLLM enables the AITER/Gluon paged-MQA logits path on gfx950 / MI355X sparse MLA shapes | On AMD sparse-MLA traces, compare against the AITER paged-MQA logits path before proposing a new logits kernel. |
|
||||
| vLLM-origin ROCm DSV4 sparse MLA Triton kernels | `rocm_aiter_mla_sparse_dsv4`<br>`flashmla_sparse`<br>`sparse_swa` | `vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse_dsv4.py`<br>`vllm/v1/attention/backends/mla/flashmla_sparse.py`<br>`vllm/v1/attention/backends/mla/sparse_swa.py` | Current vLLM mainline has ROCm DSV4 sparse-MLA Triton backend coverage | On DSV4 ROCm sparse-MLA traces, compare backend selection and sparse-SWA paths before writing a new MLA backend. |
|
||||
| vLLM-origin DSV4 dequant gather K cache | `dequant_gather_k_cutedsl`<br>`fused_indexer_q_cutedsl`<br>`cache_utils` | `vllm/v1/attention/ops/deepseek_v4_ops/dequant_gather_k_cutedsl.py`<br>`vllm/v1/attention/ops/deepseek_v4_ops/cache_utils.py` | Current vLLM mainline has a newer CuTe DSL dequant-gather K cache path for DSV4 | Treat K-cache dequant/gather ladders as a known upstream DSV4 cache-kernel family. |
|
||||
| vLLM-origin TokenSpeed MLA backend | `TOKENSPEED_MLA`<br>`tokenspeed_mla_decode`<br>`tokenspeed_mla_prefill` | `vllm/v1/attention/backends/mla/tokenspeed_mla.py`<br>`vllm/v1/attention/backends/mla/prefill/tokenspeed_mla.py`<br>`vllm/model_executor/layers/attention/mla_attention.py` | Current vLLM can select TokenSpeed MLA prefill/decode kernels for Blackwell FP8-KV DeepSeek/Kimi MLA shapes when the package and platform gates pass | On Blackwell MLA traces, compare against TokenSpeed MLA backend selection before proposing a new attention kernel. |
|
||||
| vLLM-origin MiniMax allreduce_rms kernels | `minimax_allreduce_rms`<br>`minimax_allreduce_rmsnorm`<br>`MiniMax-M2.5`<br>`allreduce_rms` | `vllm/model_executor/models/minimax_m2.py` | TensorRT-LLM-derived MiniMax allreduce-plus-RMSNorm kernels are a concrete upstream TP decode family | Treat MiniMax TP norm + collective ladders as an upstream specialized fusion family. |
|
||||
| vLLM-origin CUTLASS scaled MM with scale / bias epilogue | `cutlass_scaled_mm`<br>`cutlass_scaled_mm_azp`<br>`scaled mm` | `vllm/_custom_ops.py`<br>`vllm/model_executor/kernels/linear/scaled_mm/cutlass.py`<br>`csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu` | CUTLASS kernels fuse activation scales, weight scales, matmul, and optional bias / AZP epilogues | Treat separate scale-mul + GEMM + bias ladders as a vLLM-origin fused linear family first. |
|
||||
| vLLM-origin fused MoE expert execution | `cpu_fused_moe`<br>`rocm_aiter_fused_moe`<br>`FusedMoE` | `vllm/model_executor/layers/fused_moe/layer.py`<br>`vllm/model_executor/layers/fused_moe/cpu_fused_moe.py`<br>`vllm/model_executor/layers/fused_moe/rocm_aiter_fused_moe.py`<br>`vllm/_aiter_ops.py` | MoE backends on CUDA / ROCm / CPU already collapse packed expert execution into fused expert kernels rather than per-expert eager GEMMs | Treat exposed expert-side tiny GEMM ladders as matching an upstream fused-MoE family. |
|
||||
| vLLM-origin fused MoE LoRA | `fused_moe_lora`<br>`fused_moe_lora_fp8`<br>`w13_shrink`<br>`w2_expand` | `vllm/lora/ops/triton_ops/fused_moe_lora_op.py`<br>`vllm/lora/ops/triton_ops/fused_moe_lora_fp8_op.py`<br>`vllm/lora/layers/fused_moe.py` | Triton kernels fuse LoRA shrink / expand work into MoE expert execution, including FP8 variants | Treat MoE-LoRA adapter work as an upstream fused family before proposing a brand new kernel. |
|
||||
| vLLM-origin ViT fused bilinear position-embedding interpolation | `triton_pos_embed_interpolate`<br>`bilinear_pos_embed`<br>`pos_embed_interpolate_native` | `vllm/model_executor/models/qwen3_vl.py` | Triton kernel fuses bilinear interpolation and spatial-merge reorder for Qwen3-VL ViT position embeddings, replacing many tiny eager kernels | Treat VLM position-embedding ladders as an existing vLLM-origin Triton fusion family. |
|
||||
|
||||
## 15. vLLM-origin kernel-overlap families
|
||||
## 15. TokenSpeed-origin fused-kernel families
|
||||
|
||||
These rows are direct TokenSpeed families from `lightseekorg/tokenspeed`, not
|
||||
only vLLM references to the TokenSpeed package.
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| TokenSpeed CuTe DSL MLA prefill / decode | `tokenspeed_mla_decode`<br>`tokenspeed_mla_prefill`<br>`BlackwellMultiHeadLatentAttentionForward` | `python/tokenspeed/runtime/layers/attention/backends/tokenspeed_mla.py`<br>`tokenspeed-mla/python/tokenspeed_mla/mla_decode.py`<br>`tokenspeed-mla/python/tokenspeed_mla/mla_prefill.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/ops/attention/tokenspeed_mla/__init__.py` | Blackwell SM100 CuTe DSL MLA kernels cover FP8-KV prefill/decode/verify paths through the `tokenspeed_mla` backend | On TokenSpeed or vLLM+TokenSpeed MLA traces, compare backend selection before proposing a new MLA attention kernel. |
|
||||
| TokenSpeed MLA KV pack + FP8 quantize | `_mla_kv_pack_quantize_fp8_kernel`<br>`mla_kv_pack_quantize_fp8`<br>`k_nope` / `k_pe` | `tokenspeed-mla/python/tokenspeed_mla/mla_kv_pack_quantize_fp8.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/ops/attention/tokenspeed_mla/__init__.py` | One Triton kernel packs `k_nope`, broadcast `k_pe`, and `v`, then writes FP8 K/V for MLA chunked prefill | Treat split K/V concat + FP8 cast ladders as a known TokenSpeed fusion family. |
|
||||
| TokenSpeed fused top-k + top-p sampling | `fused_topk_topp`<br>`fused_topk_topp_renorm` | `tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/fused_topk_topp.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/csrc/fused_topk_topp/fused_topk_topp.cu` | CUDA extension fuses top-k, top-p, and renormalization for decode sampling | Treat top-k/top-p/renorm chains in TokenSpeed traces as an existing sampling fusion first. <!-- codespell:ignore thirdparty --> |
|
||||
| TokenSpeed persistent lm_head GEMM | `lm_head_gemm`<br>`should_use_fused`<br>`persistent` | `tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/lm_head_gemm.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/csrc/lm_head_gemm.cu` | Shape-gated persistent GEMM replaces `torch.matmul` for selected lm_head / router-like projection shapes | Treat visible lm_head matmul ladders as a candidate for this existing TokenSpeed path before inventing a new logits GEMM. <!-- codespell:ignore thirdparty --> |
|
||||
| TokenSpeed NVFP4 GEMM + SwiGLU + quant | `nvfp4_gemm_swiglu_nvfp4_quant`<br>`SwiGLU`<br>`SFC` | `tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cute_dsl/nvfp4_gemm_swiglu_nvfp4_quant.py` | CuTe DSL kernel fuses block-scaled NVFP4 GEMM, SwiGLU, and optional output quantization | Treat split expert GEMM + activation + FP4 quant chains as matching an upstream TokenSpeed kernel family. <!-- codespell:ignore thirdparty --> |
|
||||
|
||||
## 16. vLLM-origin kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -263,7 +288,7 @@ contain the same implementation.
|
||||
| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py`<br>`vllm/model_executor/layers/fused_moe/runner/moe_runner_base.py` | MoE shared experts can record the cloned input on `shared_experts_stream`, wait on the caller stream, run in parallel with router-side work, and rejoin before merge | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
|
||||
| vLLM-origin DCP async all-to-all overlap | `dcp_alltoall`<br>`all_to_all_single`<br>`async_op=True` | `vllm/v1/attention/ops/dcp_alltoall.py` | Output / LSE exchange uses async all-to-all handles instead of serializing collective completion on the main path | Treat DCP all-to-all windows as an upstream async-collective family. |
|
||||
|
||||
## 16. vLLM-origin PR-backed / in-flight fused-kernel and kernel-overlap families
|
||||
## 17. vLLM-origin PR-backed / in-flight fused-kernel and kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -273,18 +298,18 @@ contain the same implementation.
|
||||
| PR `#38621` fused QK norm + RoPE + cache + quant | `fused_qk_norm_rope_cache_quant`<br>`QK Norm + RoPE + Cache + Quant` | `PR #38621`<br>`csrc/fused_qk_norm_rope_cache_quant.cu`<br>`vllm/compilation/passes/fusion/qk_norm_rope_cache_quant_fusion.py` | Draft CUDA kernel and compile-time pass try to fuse QK RMSNorm, RoPE, KV cache write, and optional FP8 quant for small-batch decode | Treat this as an in-flight upstream fusion family before calling a similar idea novel. |
|
||||
| PR `#37646` ROCm AITER fused allreduce + RMSNorm | `rocm_aiter_fused_allreduce_rmsnorm`<br>`custom_fused_ar_rms`<br>`RocmAiterAllReduceFusionPass` | `PR #37646`<br>`vllm/_aiter_ops.py`<br>`vllm/compilation/passes/pass_manager.py` | ROCm-specific compile-time path swaps the generic all-reduce fusion pass for an AITER fused allreduce-plus-RMSNorm kernel family | Treat ROCm TP all-reduce + RMSNorm ladders as an in-flight upstream fused-collective family first. |
|
||||
| PR `#36413` FlashInfer RMSNorm + FP4 quant fusion | `fuse_norm_quant`<br>`flashinfer`<br>`NVFP4`<br>`rmsnorm + fp4 quant` | `PR #36413`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/docs/design/fusions.md` | FlashInfer-backed norm-plus-FP4 quant fusion extends the existing RMSNorm+quant family to NVFP4 flows | Treat split RMSNorm + FP4 quant ladders as an upstream in-flight family, not a fresh idea. |
|
||||
| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | Extends the specialized router GEMM family to GLM5 hidden size and uses PDL to overlap the router launch with the preceding fused allreduce-plus-RMS block | Treat this as an in-flight upstream router-kernel plus launch-overlap family before calling it novel. |
|
||||
| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/dsv3_router_gemm_utils.h` | Extends the specialized router GEMM family to GLM5 hidden size and uses PDL to overlap the router launch with the preceding fused allreduce-plus-RMS block | Treat this as an in-flight upstream router-kernel plus launch-overlap family before calling it novel. |
|
||||
| PR `#41455` ROCm WMMA paged prefill and split-K decode | `wmma`<br>`paged prefill`<br>`split-K decode`<br>`ROCm attention` | `PR #41455`<br>`vllm/v1/attention`<br>`vllm/_aiter_ops.py` | Adds ROCm WMMA attention kernels for paged prefill and split-K decode shapes | Treat split attention support kernels on AMD as an in-flight vLLM attention-kernel family before calling them novel. |
|
||||
| PR `#41263` DeepSeek-V4 fused norm / router low-latency path | `DSV4`<br>`fuse norm router`<br>`low latency`<br>`router` | `PR #41263`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/model_executor/layers/fused_moe/router` | Targets DeepSeek-V4 decode latency by fusing norm / router-adjacent work and low-latency model paths | Treat DSV4 norm-router ladders as a concrete in-flight upstream family. |
|
||||
| PR `#41428` DSV4 fused indexer Q quant kernel | `DSV4`<br>`fused Indexer Q quant`<br>`indexer q`<br>`fp4` | `PR #41428`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/csrc` | Improves the fused DeepSeek-V4 indexer Q quant kernel instead of materializing Q then quantizing separately | Treat DSV4 indexer-Q quant ladders as an in-flight upstream fused quant family. |
|
||||
| PR `#41255` DeepSeek-V4 Tile kernels / `head_compute_mix_kernel` | `head_compute_mix_kernel`<br>`Tile kernel`<br>`DSV4`<br>`MLA` | `PR #41255`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/csrc` | Adds DeepSeek-V4 Tile kernels that mix head compute work in one specialized kernel | Treat DSV4 MLA head-compute ladders as a known in-flight specialized-kernel family. |
|
||||
| PR `#41263` DeepSeek-V4 fused norm / router low-latency path | `DSV4`<br>`fuse norm router`<br>`low latency`<br>`router` | `PR #41263`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`vllm/model_executor/layers/fused_moe/router/norm_gate_linear.py` | Merged into current mainline as the DSV4 fused norm + router GEMM family above | Treat this row as provenance for the shipped mainline family, not as merely in-flight. |
|
||||
| PR `#41428` DSV4 fused indexer Q quant kernel | `DSV4`<br>`fused Indexer Q quant`<br>`indexer q`<br>`fp4` | `PR #41428`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`vllm/v1/attention/ops/deepseek_v4_ops/fused_indexer_q.py`<br>`vllm/v1/attention/ops/deepseek_v4_ops/fused_indexer_q_cutedsl.py` | Merged current-main improvement to the fused DeepSeek-V4 indexer Q quant kernel instead of materializing Q then quantizing separately | Treat DSV4 indexer-Q quant ladders as an upstream mainline fused quant family. |
|
||||
| PR `#41255` DeepSeek-V4 Tile kernels / `head_compute_mix_kernel` | `head_compute_mix_kernel`<br>`Tile kernel`<br>`DSV4`<br>`MLA` | `PR #41255`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`vllm/model_executor/layers/mhc.py`<br>`vllm/model_executor/kernels/mhc/tilelang.py` | Merged current-main DeepSeek-V4 Tile kernels that mix head compute work in one specialized kernel | Treat DSV4 MLA head-compute ladders as a known upstream mainline specialized-kernel family. |
|
||||
| PR `#41441` DSV4 all-reduce plus `mhc_post` fusion | `DSV4`<br>`AR+mhc_post`<br>`allreduce`<br>`mhc_post` | `PR #41441`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/compilation/passes/fusion` | Fuses or overlaps DSV4 all-reduce with post-MLA head-compute work | Treat all-reduce followed by `mhc_post` in DSV4 traces as an in-flight vLLM overlap/fusion family. |
|
||||
| PR `#41446` AMD GatedDeltaNet FLA prefill kernels | `GatedDeltaNet`<br>`FLA prefill`<br>`AMD`<br>`Qwen3-Next` | `PR #41446`<br>`vllm/model_executor/models/qwen3_next.py`<br>`vllm/v1/attention` | Optimizes GatedDeltaNet / FLA prefill kernels on AMD linear-attention models | Treat split GDN prefill kernels on ROCm as an in-flight upstream family. |
|
||||
| PR `#39748` dual-stream GDN input projection | `dual-stream`<br>`input projection`<br>`GatedDeltaNet`<br>`Qwen3.5` | `PR #39748`<br>`vllm/model_executor/models/qwen3_next.py` | Overlaps sibling input-projection branches for Qwen3 / Qwen3.5 GDN-style blocks | Treat serial GDN input projections as a known in-flight overlap opportunity. |
|
||||
| PRs `#41433` / `#41434` / `#41429` / `#40561` GPU/CPU sync removal | `GPU->CPU sync`<br>`cpu sync`<br>`item()`<br>`non_blocking` | `PR #41433`<br>`PR #41434`<br>`PR #41429`<br>`PR #40561` | Removes or gates accidental GPU-to-CPU synchronization points and adds sync-detection coverage | Treat CPU gaps next to small GPU kernels as an upstream vLLM sync-removal family before proposing a kernel-only fix. |
|
||||
| PR `#36823` vLLM IR `fused_add_rms_norm` overload | `vllm_ir`<br>`fused_add_rms_norm`<br>`maybe_inplace` | `PR #36823`<br>`vllm/compilation/passes/ir`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py` | Extends vLLM IR lowering so fused-add-RMSNorm variants remain visible to later compile-time fusions | Treat missing norm/quant compile fusion as potentially an IR-lowering visibility issue. |
|
||||
|
||||
## 17. Important toggles and caveats
|
||||
## 18. Important toggles and caveats
|
||||
|
||||
| Toggle / env | Location | Effect on trace interpretation |
|
||||
| --- | --- | --- |
|
||||
@@ -294,7 +319,7 @@ contain the same implementation.
|
||||
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
|
||||
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
|
||||
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
|
||||
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates DSA fused top-k transform / page-table build. |
|
||||
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
|
||||
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
|
||||
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
|
||||
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
|
||||
@@ -313,20 +338,27 @@ contain the same implementation.
|
||||
| `mlir_elementwise_fusion` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Can absorb merge adds into larger fused kernels, so missing explicit merge nodes in multi-stream traces may be intentional. |
|
||||
| `enable_torch_compile` | `python/sglang/srt/server_args.py`<br>`python/sglang/multimodal_gen/runtime/server_args.py` | Compiler-generated fusion / reordering can hide handwritten kernel names; absence of a custom kernel does not always mean absence of fusion. |
|
||||
| `enable_fused_grouped_gemm_combine` | `PR #21877` | In-flight path that intentionally disables SBO because combine is folded into down-GEMM. |
|
||||
| `PassConfig.fuse_allreduce_rms` | `vllm/config/compilation.py` | Enables vLLM's AllReduce -> RMSNorm (+ residual / quant) compile-time fusion family. |
|
||||
| `PassConfig.fuse_allreduce_rms` | `vllm/config/compilation.py` | Enables vLLM's AllReduce -> RMSNorm (+ residual / quant) compile-time fusion family; on ROCm, AITER variants and add-RMSNorm-pad ordering can change the visible kernel split. |
|
||||
| `PassConfig.fuse_norm_quant` | `vllm/config/compilation.py` | Enables vLLM's RMSNorm(+residual add) -> FP8 / FP4 quant compile-time fusion family. |
|
||||
| `PassConfig.fuse_act_quant` | `vllm/config/compilation.py` | Enables vLLM's `SiLU+Mul -> quant` fusion family, plus ROCm AITER variants where applicable. |
|
||||
| `PassConfig.fuse_attn_quant` | `vllm/config/compilation.py` | Enables attention-epilogue quant fusion; requires the right backend / graph visibility, so split kernels may still be expected. |
|
||||
| `PassConfig.fuse_mla_dual_rms_norm` | `vllm/config/compilation.py` | Enables the AITER-backed MLA paired-Q/KV RMSNorm fusion family on ROCm. |
|
||||
| `PassConfig.enable_qk_norm_rope_fusion` | `vllm/config/compilation.py` | Enables the compile-time QK RMSNorm + RoPE family on CUDA-like backends. |
|
||||
| `PassConfig.fuse_rope_kvcache` | `vllm/config/compilation.py` | Enables ROCm / AITER RoPE + KV-cache update fusion and is range-limited by token count. |
|
||||
| `PassConfig.fuse_rope_kvcache_cat_mla` | `vllm/config/compilation.py` | Enables the MLA RoPE + unified MLA KV-cache update compile-time fusion family. |
|
||||
| `AttentionBackendEnum.TOKENSPEED_MLA` | `vllm/model_executor/layers/attention/mla_attention.py`<br>`vllm/platforms/cuda.py` | Selects the TokenSpeed MLA backend on supported Blackwell FP8-KV MLA shapes when `tokenspeed-mla` is installed. |
|
||||
| `rocm_aiter_mla_sparse_dsv4` | `vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse_dsv4.py` | Selects the ROCm DSV4 sparse-MLA backend path on supported AMD shapes. |
|
||||
| `PassConfig.fuse_minimax_qk_norm` | `vllm/config/compilation.py` | Enables the MiniMax decode Q/K allreduce-plus-RMSNorm compile-time fusion family. |
|
||||
| `PassConfig.fuse_act_padding` | `vllm/config/compilation.py` | Enables the ROCm AITER add-RMSNorm-plus-pad fusion family when AITER is available. |
|
||||
| `PassConfig.enable_sp` | `vllm/config/compilation.py` | Rewrites all-reduce into sequence-parallel staging; this is often a prerequisite for the overlap family, not just a pure fuse toggle. |
|
||||
| `PassConfig.fuse_gemm_comms` | `vllm/config/compilation.py` | Enables AsyncTP GEMM + collective overlap and auto-enables `enable_sp` when valid. |
|
||||
| `TRTLLM_ENABLE_PDL` | `vllm/csrc/dsv3_fused_a_gemm.cu`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | Enables programmatic dependent launch for the DSV3 specialized CUDA kernels, which can change launch grouping and trace shape for router / QKV-A paths. |
|
||||
| vLLM PR `#46735` Triton MoE CUDA graph capture fix | `vllm/model_executor/layers/fused_moe/experts/triton_moe.py`<br>`vllm/model_executor/layers/fused_moe/experts/nvfp4_emulation_moe.py` | Latest vLLM mainline fixes CUDA graph capture around Triton / NVFP4-emulation MoE; stale target images may show graph-capture failures or eager fallbacks that are not SGLang kernel wins. |
|
||||
| `TRTLLM_ENABLE_PDL` | `csrc/libtorch_stable/dsv3_fused_a_gemm.cu`<br>`csrc/moe/dsv3_router_gemm_utils.h` | Enables programmatic dependent launch for the DSV3 specialized CUDA kernels, which can change launch grouping and trace shape for router / QKV-A paths. |
|
||||
| TokenSpeed `--attention-backend tokenspeed_mla` | `python/tokenspeed/runtime/layers/attention/backends/tokenspeed_mla.py` | Selects TokenSpeed's native CuTe DSL MLA backend; requires compatible Blackwell FP8-KV MLA shapes, so split MLA support kernels may indicate backend gating rather than a missing kernel. |
|
||||
| TokenSpeed `TOKENSPEED_MLA_PREFILL_BACKEND` | `tokenspeed-mla/python/tokenspeed_mla/mla_prefill.py` | Chooses CuTe DSL JIT vs binary prefill backend; trace kernel names can differ even when the same MLA fused family applies. |
|
||||
| TokenSpeed `--comm-fusion-max-num-tokens` / `--enable-allreduce-fusion` | `docs/configuration/server.md`<br>`python/tokenspeed/runtime/distributed/comm_backend` | Gates TokenSpeed communication-fusion behavior; inspect these before treating all-reduce + compute separation as a novel overlap gap. |
|
||||
|
||||
## 18. Suggested refresh commands
|
||||
## 19. Suggested refresh commands
|
||||
|
||||
These commands are only for maintainers refreshing this catalog by rescanning
|
||||
the local source trees. They are not used by the triage scripts at runtime.
|
||||
|
||||
@@ -26,9 +26,12 @@ necessarily present in the checked-out `sglang` tree, but they should still be
|
||||
treated as upstream or analogous kernel-overlap families before labeling an
|
||||
overlap opportunity as novel.
|
||||
|
||||
Refresh note `2026-04-22`: rescanned current `sglang`, `flashinfer`,
|
||||
`TensorRT-LLM`, and `vllm` mainline overlap paths plus rechecked referenced PR
|
||||
state via the GitHub API on `2026-04-22`. Closed-unmerged SGLang
|
||||
Refresh note `2026-06-26`: rechecked official main heads for SGLang
|
||||
`8524678889485801e7a4a12d62015be0c68f7a90`, vLLM
|
||||
`abc71548ef029132c3316b902207f254a246d593`, TensorRT-LLM
|
||||
`0722c5f47d2cae69ac1a237da51e550dd214532c`, and TokenSpeed
|
||||
`5aedf69d6b476baa65571011de6ea60fd5a238a8`, then added the first
|
||||
TokenSpeed-origin communication-fusion row. Closed-unmerged SGLang
|
||||
[#22410](https://github.com/sgl-project/sglang/pull/22410) and FlashInfer
|
||||
[#2840](https://github.com/flashinfer-ai/flashinfer/pull/2840) were removed
|
||||
from the PR-backed sections. SGLang
|
||||
@@ -48,7 +51,7 @@ upstream overlap references as of this refresh.
|
||||
| Llama4 shared branch vs routed branch overlap | shared expert branch plus routed MoE branch as adjacent windows | `python/sglang/srt/models/llama4.py` | shared expert on current stream, router + topk + routed experts on `alt_stream` | Use Llama4 as the first precedent for branch-level overlap in similar sparse models. |
|
||||
| ExaoneMoE shared experts vs router experts overlap | shared expert output and router-expert output form a two-branch window | `python/sglang/srt/models/exaone_moe.py::forward_normal_dual_stream` | shared experts on current stream, router + routed experts on `alt_stream`, explicit join before combine | This is an existing dual-stream MoE overlap family. |
|
||||
| Grok residual-MoE branch overlap | dense MLP and block-sparse MoE branches in parallel | `python/sglang/srt/models/grok.py::moe_with_rmoe` | dense MLP on current stream, MoE on `alt_stream`, fused dual residual RMSNorm around boundaries | Treat exposed Grok branch overlap as an existing pattern. |
|
||||
| DSA dual-stream overlap | Q-proj, K-proj, RoPE, cache-store, quantization in tight two-stream windows | `python/sglang/srt/layers/attention/dsa/dsa_indexer.py` | Q / K projection split, RoPE split, cache-store vs quantization overlap | DSA already contains several dual-stream overlap precedents. |
|
||||
| NSA dual-stream overlap | Q-proj, K-proj, RoPE, cache-store, quantization in tight two-stream windows | `python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Q / K projection split, RoPE split, cache-store vs quantization overlap | NSA already contains several dual-stream overlap precedents. |
|
||||
| MoriEP async dispatch / combine comm stream | `MoriEP`<br>`_comm_stream`<br>`dispatch`<br>`combine`<br>`done_event` | `python/sglang/srt/layers/moe/token_dispatcher/moriep.py` | MoriEP can submit dispatch and combine onto a dedicated communication stream and synchronize only through events | Treat MoriEP comm / compute interleave as an existing MoE overlap family. |
|
||||
| Generic `alt_stream` overlap families | `alt_stream` plus explicit `wait_stream` / `with torch.cuda.stream(...)` | `qwen2_moe.py`<br>`qwen3_moe.py`<br>`glm4_moe.py`<br>`bailing_moe.py`<br>`llada2.py`<br>`grok.py`<br>`olmo2.py`<br>`step3p5.py`<br>`longcat_flash.py`<br>`falcon_h1.py` | model-specific overlap on attention prep, MoE branches, or cache-store | Search these families before designing a new overlap scheme from scratch. |
|
||||
|
||||
@@ -107,7 +110,16 @@ AutoDeploy rather than same-stream PDL windows.
|
||||
| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node; the same family includes synchronization rules for MLIR-fused kernels and piecewise cudagraph replay | Treat shared-expert vs routed-expert windows, including altered behavior under MLIR / piecewise graph modes, as an existing TensorRT-LLM branch-overlap family. |
|
||||
| TensorRT-LLM multi-stream FP8 GEMM fork parallelism | `multi_stream_gemm`<br>`trtllm_finegrained_fp8_linear`<br>`record_event_passthrough`<br>`_aux` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_gemm.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Compiler pass identifies fork points with multiple FP8 linears and moves the largest GEMM to the auxiliary stream so sibling GEMMs overlap | Treat sibling FP8 linear branches as an existing TensorRT-LLM overlap family before designing a new stream split. |
|
||||
|
||||
## 8. vLLM-origin kernel-overlap families
|
||||
## 8. TokenSpeed-origin kernel-overlap families
|
||||
|
||||
These rows are comparative references from `lightseekorg/tokenspeed`. Use them
|
||||
when the trace is from TokenSpeed or from a vLLM/TokenSpeed hybrid deployment.
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| TokenSpeed allreduce / communication fusion | `enable_allreduce_fusion`<br>`comm_fusion`<br>`comm_fusion_max_num_tokens`<br>`allreduce` | `docs/configuration/server.md`<br>`python/tokenspeed/runtime/distributed/comm_backend` | TokenSpeed exposes runtime knobs for communication fusion and token-count gating, so all-reduce windows may be a disabled or shape-gated TokenSpeed path | Treat split all-reduce + compute windows in TokenSpeed traces as a comm-fusion eligibility question before calling it a novel overlap opportunity. |
|
||||
|
||||
## 9. vLLM-origin kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -116,14 +128,14 @@ AutoDeploy rather than same-stream PDL windows.
|
||||
| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py`<br>`vllm/model_executor/layers/fused_moe/runner/moe_runner_base.py` | MoE shared experts can record the cloned input on `shared_experts_stream`, wait on the caller stream, run in parallel with router-side work, and rejoin before merge | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
|
||||
| vLLM-origin DCP async all-to-all overlap | `dcp_alltoall`<br>`all_to_all_single`<br>`async_op=True` | `vllm/v1/attention/ops/dcp_alltoall.py` | Output / LSE exchange uses async all-to-all handles instead of serializing collective completion on the main path | Treat DCP all-to-all windows as an upstream async-collective family. |
|
||||
|
||||
## 9. vLLM-origin PR-backed / in-flight kernel-overlap families
|
||||
## 10. vLLM-origin PR-backed / in-flight kernel-overlap families
|
||||
|
||||
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Closed PR explored overlapping the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
|
||||
| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | The GLM5 router GEMM path explicitly uses PDL so the router kernel can overlap with the preceding fused allreduce-plus-RMS block on supported GPUs | Treat router-GEMM launch overlap on GLM5-like traces as an in-flight upstream family first. |
|
||||
|
||||
## 10. Important toggles and caveats
|
||||
## 11. Important toggles and caveats
|
||||
|
||||
| Toggle / env | Location | Effect on trace interpretation |
|
||||
| --- | --- | --- |
|
||||
@@ -144,8 +156,9 @@ AutoDeploy rather than same-stream PDL windows.
|
||||
| `enable_fused_grouped_gemm_combine` | `PR #21877` | In-flight path that intentionally disables SBO because combine is folded into down-GEMM. |
|
||||
| `PassConfig.enable_sp` | `vllm/config/compilation.py` | Enables vLLM's sequence-parallel staging family that creates RS / AG overlap opportunities. |
|
||||
| `PassConfig.fuse_gemm_comms` | `vllm/config/compilation.py` | Enables AsyncTP GEMM + collective overlap and auto-enables `enable_sp` when valid. |
|
||||
| TokenSpeed `--comm-fusion-max-num-tokens` / `--enable-allreduce-fusion` | `docs/configuration/server.md` | Gates TokenSpeed communication fusion; inspect it before treating all-reduce + compute separation as a new overlap gap. |
|
||||
|
||||
## 11. Suggested refresh commands
|
||||
## 12. Suggested refresh commands
|
||||
|
||||
These commands are only for maintainers refreshing this catalog by rescanning
|
||||
the local source trees. They are not used by the triage scripts at runtime.
|
||||
@@ -155,6 +168,7 @@ the local source trees. They are not used by the triage scripts at runtime.
|
||||
FLASHINFER_REPO=${FLASHINFER_REPO:-../flashinfer}
|
||||
TRTLLM_REPO=${TRTLLM_REPO:-../TensorRT-LLM}
|
||||
VLLM_REPO=${VLLM_REPO:-../vllm}
|
||||
TOKENSPEED_REPO=${TOKENSPEED_REPO:-../tokenspeed}
|
||||
|
||||
rg -n "single_batch_overlap|alt_stream|shared_expert|scatter_stream|_fused_gather_to_staging_kernel|_fused_scatter_from_staging_kernel|async_op=True" python/sglang
|
||||
rg -n "apply_qk_norm|vision.py|ring_attn|all_to_all_single|reorder_for_compute_comm_overlap|use_dual_stream" python/sglang/multimodal_gen python/sglang/srt
|
||||
@@ -166,6 +180,8 @@ rg -n "mlir_elementwise_fusion|piecewise|cudagraph|caller_stream.synchronize" "$
|
||||
git -C "$TRTLLM_REPO" log --all --format='%h %s' | rg -i 'overlap|multi-stream|aux stream|cudagraph|mlir|stream|flashinfer|moe|mla'
|
||||
rg -n "fuse_gemm_comms|enable_sp|fused_matmul_reduce_scatter|fused_all_gather_matmul|shared_experts_stream|maybe_sync_shared_experts_stream|dcp_alltoall|async_op=True|aux_stream|maybe_execute_in_parallel" "$VLLM_REPO/vllm" "$VLLM_REPO/docs/design/fusions.md"
|
||||
git -C "$VLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|allreduce|reduce-scatter|all-gather|all_to_all|stream|multi-stream|triton|cuda|router'
|
||||
rg -n "enable_allreduce_fusion|comm_fusion|comm_fusion_max_num_tokens|allreduce|reduce_scatter" "$TOKENSPEED_REPO/python" "$TOKENSPEED_REPO/docs"
|
||||
git -C "$TOKENSPEED_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|allreduce|stream|comm|mla|tokenspeed_mla'
|
||||
# GitHub PR scan terms for the connector or web UI:
|
||||
# "fused OR overlap repo:sgl-project/sglang"
|
||||
# "triton OR cutedsl OR cuda overlap repo:sgl-project/sglang"
|
||||
@@ -177,4 +193,5 @@ git -C "$VLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|all
|
||||
# "fused OR overlap repo:vllm-project/vllm"
|
||||
# "triton OR cuda overlap repo:vllm-project/vllm"
|
||||
# "multi-stream OR aux_stream overlap repo:vllm-project/vllm"
|
||||
# "fused OR overlap OR comm_fusion repo:lightseekorg/tokenspeed"
|
||||
```
|
||||
|
||||
+13
-5
@@ -1,7 +1,12 @@
|
||||
# vLLM Torch Compile Fusion Patterns
|
||||
|
||||
Refresh: `2026-05-01`.
|
||||
Source tree: vLLM `origin/main` at `7075df79b`.
|
||||
Refresh: `2026-06-26`.
|
||||
Source tree: vLLM `origin/main` at
|
||||
`abc71548ef029132c3316b902207f254a246d593`; no new LLM compile-fusion pass was
|
||||
added after `2317682f9` in this refresh. The mainline `#40392` MLA RoPE +
|
||||
KV-cache cat fusion is already included below. Recent post-`#46735` vLLM
|
||||
changes include runtime / frontend work such as `#44800` and `#46799`, but they
|
||||
do not add a new LLM compile-fusion pass to this inventory.
|
||||
|
||||
Use this file when the fuse-pattern table reports split kernels in a trace and
|
||||
you need to decide whether the shape is already covered by vLLM's
|
||||
@@ -17,7 +22,7 @@ vLLM registers these passes from
|
||||
| --- | --- | --- |
|
||||
| `enable_sp` | `SequenceParallelismPass` | all-reduce around residual/norm blocks becomes reduce-scatter, local work, and all-gather |
|
||||
| `fuse_gemm_comms` | `AsyncTPPass` | GEMM plus reduce-scatter / all-gather overlap through symmetric-memory collectives |
|
||||
| `fuse_allreduce_rms` | `AllReduceFusionPass` | all-reduce followed by RMSNorm, optional residual add, optional FP8 / NVFP4 quant |
|
||||
| `fuse_allreduce_rms` | `AllReduceFusionPass` or ROCm AITER variant | all-reduce followed by RMSNorm, optional residual add, optional FP8 / NVFP4 quant; current pass ordering runs AITER add-RMSNorm-pad before this fusion when available |
|
||||
| `fuse_minimax_qk_norm` | `MiniMaxQKNormPass` | MiniMax Q/K all-reduce plus RMSNorm decode path |
|
||||
| `fuse_norm_quant` | `RMSNormQuantFusionPass` | RMSNorm or fused-add-RMSNorm followed by FP8 / FP4 quant |
|
||||
| `fuse_norm_quant` + AITER | `RocmAiterRMSNormQuantFusionPass` | ROCm AITER RMSNorm / fused-add-RMSNorm followed by AITER or vLLM quant |
|
||||
@@ -26,6 +31,7 @@ vLLM registers these passes from
|
||||
| `fuse_act_padding` + AITER | `RocmAiterTritonAddRMSNormPadFusionPass` | AITER fused-add-RMSNorm followed by padding into the next layout |
|
||||
| `fuse_mla_dual_rms_norm` + AITER | `MLADualRMSNormFusionPass` | MLA paired Q and KV RMSNorms become `fused_mla_dual_rms_norm` |
|
||||
| `fuse_rope_kvcache` | `RopeKVCacheFusionPass` | RoPE plus paged KV-cache update, after split cleanup passes |
|
||||
| `fuse_rope_kvcache_cat_mla` | `MLARoPEKVCacheCatFusionPass` | MLA RoPE on `q_pe` / `k_pe` plus unified MLA KV-cache update through a fused concat/cache op |
|
||||
| `fuse_attn_quant` | `AttnQuantFusionPass` | attention output followed by FP8 / NVFP4 quant |
|
||||
| `fuse_attn_quant` | `MLAAttnQuantFusionPass` | MLA attention output followed by FP8 / NVFP4 / FP8 group quant |
|
||||
| `enable_qk_norm_rope_fusion` | `QKNormRoPEFusionPass` | Q/K RMSNorm plus RoPE on packed QKV tensors |
|
||||
@@ -43,6 +49,7 @@ vLLM registers these passes from
|
||||
| `fusion/rocm_aiter_fusion.py` | `MLADualRMSNormPattern` | MLA Q branch and KV branch each run RMSNorm | `torch.ops.vllm.fused_mla_dual_rms_norm` backed by AITER fused QK RMSNorm |
|
||||
| `fusion/qk_norm_rope_fusion.py` | `QkNormRopePattern` | Q/K RMSNorm, split/getitem reshapes, then RoPE | `_C.fused_qk_norm_rope` |
|
||||
| `fusion/rope_kvcache_fusion.py` | `RopeReshapeKVCachePattern` | RoPE output followed by reshape/cache update | `vllm.fused_rope_and_unified_kv_cache_update` |
|
||||
| `fusion/mla_rope_kvcache_cat_fusion.py` | `MLARoPEKVCacheCatPattern` | MLA RoPE on `q_pe` and `k_pe` flows into `unified_mla_kv_cache_update` | `vllm.fused_rope_unified_mla_kv_cache_update`, backed by `concat_and_cache_mla_rope_fused` |
|
||||
| `fusion/attn_quant_fusion.py` | `AttnFp8StaticQuantPattern`, `AttnNvfp4QuantPattern` | attention output followed by FP8 static quant or NVFP4 quant | backend attention op with fused output quant when supported |
|
||||
| `fusion/mla_attn_quant_fusion.py` | `MLAAttnFp8StaticQuantPattern`, `MLAAttnNvfp4QuantPattern`, `MLAAttnFp8GroupQuantPattern` | MLA attention output followed by static FP8, NVFP4, or FP8 group quant | MLA attention op with fused output quant when supported |
|
||||
| `fusion/minimax_qk_norm_fusion.py` | `MiniMaxQKNormPattern` | MiniMax `forward_qk`: Q/K variance all-reduce divided by TP world size, then RMS apply | `vllm.minimax_qk_norm_fused` / Lamport fused kernel |
|
||||
@@ -56,8 +63,9 @@ vLLM registers these passes from
|
||||
- If the trace shows attention output followed by quant kernels, compare against
|
||||
`AttnQuantFusionPass` or `MLAAttnQuantFusionPass`, not only handwritten
|
||||
attention kernels.
|
||||
- If the trace shows Q/K norm followed by RoPE or cache update, compare both
|
||||
`QKNormRoPEFusionPass` and `RopeKVCacheFusionPass`; they are separate passes.
|
||||
- If the trace shows Q/K norm followed by RoPE or cache update, compare
|
||||
`QKNormRoPEFusionPass`, `RopeKVCacheFusionPass`, and the MLA-specific
|
||||
`MLARoPEKVCacheCatFusionPass`; they are separate passes.
|
||||
- If the trace is a TP decode trace with visible collectives, check whether
|
||||
`enable_sp` and `fuse_gemm_comms` would transform the same region into
|
||||
sequence-parallel or AsyncTP overlap.
|
||||
|
||||
@@ -34,8 +34,8 @@ def build_triage_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="analyze_llm_torch_profile.py",
|
||||
description=(
|
||||
"Compact LLM torch-profiler triage entrypoint for SGLang, vLLM, and "
|
||||
"TensorRT-LLM. "
|
||||
"Compact LLM torch-profiler triage entrypoint for SGLang, vLLM, "
|
||||
"TensorRT-LLM, and TokenSpeed. "
|
||||
"This prints three tables: kernel mapping, overlap opportunities, "
|
||||
"and fuse opportunities. "
|
||||
"Use either a single trace/profile input or a mapping+formal two-trace pair."
|
||||
@@ -45,7 +45,17 @@ def build_triage_parser() -> argparse.ArgumentParser:
|
||||
"--framework",
|
||||
type=str,
|
||||
default="auto",
|
||||
choices=["auto", "sglang", "vllm", "trtllm", "tllm", "tensorrt-llm"],
|
||||
choices=[
|
||||
"auto",
|
||||
"sglang",
|
||||
"vllm",
|
||||
"trtllm",
|
||||
"tllm",
|
||||
"tensorrt-llm",
|
||||
"tokenspeed",
|
||||
"token-speed",
|
||||
"ts",
|
||||
],
|
||||
help=(
|
||||
"Serving framework. Use auto to detect from trace contents, path hints, "
|
||||
"or URL features."
|
||||
@@ -64,7 +74,9 @@ def build_triage_parser() -> argparse.ArgumentParser:
|
||||
help=(
|
||||
"Running server URL for single-trace triage. SGLang supports direct "
|
||||
"capture via sglang.profiler. vLLM and TensorRT-LLM require a server-side "
|
||||
"torch-profiler output path exposed via --output-dir."
|
||||
"torch-profiler output path exposed via --output-dir. TokenSpeed live "
|
||||
"capture uses the server's /start_profile and /stop_profile endpoints "
|
||||
"when they are available."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
@@ -74,7 +86,8 @@ def build_triage_parser() -> argparse.ArgumentParser:
|
||||
help=(
|
||||
"Trace output dir when using --url. For vLLM this should match the "
|
||||
"server's torch_profiler_dir. For TensorRT-LLM it should match the "
|
||||
"directory or file path configured by TLLM_TORCH_PROFILE_TRACE."
|
||||
"directory or file path configured by TLLM_TORCH_PROFILE_TRACE. "
|
||||
"For TokenSpeed this is passed as start_profile.output_dir."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
@@ -83,7 +96,8 @@ def build_triage_parser() -> argparse.ArgumentParser:
|
||||
default="triage-trace",
|
||||
help=(
|
||||
"Profile prefix when generating a trace from --url. SGLang uses it "
|
||||
"directly; vLLM and TensorRT-LLM may ignore it on the HTTP profiler path."
|
||||
"directly; TokenSpeed maps it to profile_id; vLLM and TensorRT-LLM may "
|
||||
"ignore it on the HTTP profiler path."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Backwards-compatibility shim for the unified LLM torch-profiler entrypoint.
|
||||
|
||||
The real implementation now lives in ``analyze_llm_torch_profile`` because this
|
||||
skill covers SGLang, vLLM, and TensorRT-LLM. Older scripts and runbooks that
|
||||
skill covers SGLang, vLLM, TensorRT-LLM, and TokenSpeed. Older scripts and runbooks that
|
||||
still invoke ``analyze_sglang_torch_profile.py`` keep working by forwarding to
|
||||
that module.
|
||||
"""
|
||||
|
||||
@@ -31,7 +31,7 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument(
|
||||
"--framework",
|
||||
required=True,
|
||||
choices=("sglang", "vllm", "trtllm"),
|
||||
choices=("sglang", "vllm", "trtllm", "tokenspeed"),
|
||||
help="Serving framework.",
|
||||
)
|
||||
parser.add_argument(
|
||||
@@ -42,7 +42,7 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
default=None,
|
||||
help="OpenAI model id. Auto-discovered for vLLM and TensorRT-LLM when omitted.",
|
||||
help="OpenAI model id. Auto-discovered for vLLM, TensorRT-LLM, and TokenSpeed when omitted.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--requests",
|
||||
@@ -153,7 +153,7 @@ def openai_request(
|
||||
def run_probe(args: argparse.Namespace) -> Dict[str, Any]:
|
||||
prompts = args.prompt or list(DEFAULT_PROMPTS)
|
||||
model = args.model
|
||||
if args.framework in {"vllm", "trtllm"} and not model:
|
||||
if args.framework in {"vllm", "trtllm", "tokenspeed"} and not model:
|
||||
model = discover_openai_model(args.url, timeout=args.timeout)
|
||||
|
||||
latencies: List[float] = []
|
||||
|
||||
@@ -22,6 +22,7 @@ FRAMEWORK_LABELS = {
|
||||
"sglang": "SGLang",
|
||||
"vllm": "vLLM",
|
||||
"trtllm": "TensorRT-LLM",
|
||||
"tokenspeed": "TokenSpeed",
|
||||
}
|
||||
TRACE_FILE_PATTERNS = (
|
||||
"*.trace.json",
|
||||
@@ -89,6 +90,9 @@ def canonicalize_framework(value: object) -> str:
|
||||
"trtllm": "trtllm",
|
||||
"tensorrt-llm": "trtllm",
|
||||
"tensorrtllm": "trtllm",
|
||||
"tokenspeed": "tokenspeed",
|
||||
"token-speed": "tokenspeed",
|
||||
"ts": "tokenspeed",
|
||||
}
|
||||
return aliases.get(lowered, "auto")
|
||||
|
||||
@@ -105,6 +109,8 @@ def _normalize_repo_relative_path_cached(text: str) -> str:
|
||||
("python/sglang/", "python/sglang/"),
|
||||
("sgl_kernel/", "sgl_kernel/"),
|
||||
("vllm/", "vllm/"),
|
||||
("python/tokenspeed/", "python/tokenspeed/"),
|
||||
("tokenspeed/", "tokenspeed/"),
|
||||
("tensorrt_llm/", "tensorrt_llm/"),
|
||||
("tensorrt-llm/", "tensorrt_llm/"),
|
||||
):
|
||||
@@ -312,6 +318,8 @@ def detect_framework_from_text(text: object) -> Optional[str]:
|
||||
lowered = normalize_text(text).lower()
|
||||
if not lowered:
|
||||
return None
|
||||
if any(token in lowered for token in ("tokenspeed", "token-speed", "/ts/")):
|
||||
return "tokenspeed"
|
||||
if any(
|
||||
token in lowered
|
||||
for token in (
|
||||
@@ -333,6 +341,22 @@ def detect_framework_from_server_args(server_args: Optional[dict]) -> Optional[s
|
||||
if not isinstance(server_args, dict) or not server_args:
|
||||
return None
|
||||
lowered_keys = {normalize_text(key).lower() for key in server_args}
|
||||
text = json.dumps(server_args, sort_keys=True)
|
||||
if any(token in text.lower() for token in ("tokenspeed", "token-speed")):
|
||||
return "tokenspeed"
|
||||
if lowered_keys & {
|
||||
"attn_tp_size",
|
||||
"dense_tp_size",
|
||||
"moe_tp_size",
|
||||
"enable_mla_l1_5_cache",
|
||||
"mla_chunk_multiplier",
|
||||
"comm_fusion_max_num_tokens",
|
||||
"enable_allreduce_fusion",
|
||||
}:
|
||||
return "tokenspeed"
|
||||
text_hint = detect_framework_from_text(text)
|
||||
if text_hint:
|
||||
return text_hint
|
||||
if lowered_keys & {
|
||||
"attention_backend",
|
||||
"sampling_backend",
|
||||
@@ -342,7 +366,7 @@ def detect_framework_from_server_args(server_args: Optional[dict]) -> Optional[s
|
||||
"schedule_policy",
|
||||
}:
|
||||
return "sglang"
|
||||
return detect_framework_from_text(json.dumps(server_args, sort_keys=True))
|
||||
return None
|
||||
|
||||
|
||||
def detect_framework_from_trace(trace: object) -> Optional[str]:
|
||||
@@ -402,6 +426,9 @@ def detect_framework_from_url(
|
||||
or "decode" in server_info
|
||||
):
|
||||
return "sglang"
|
||||
readiness = try_get_json(url.rstrip("/") + "/readiness", timeout=5.0)
|
||||
if readiness is not None:
|
||||
return "tokenspeed"
|
||||
models = try_get_json(url.rstrip("/") + "/v1/models")
|
||||
if isinstance(models, dict) and isinstance(models.get("data"), list):
|
||||
return "vllm"
|
||||
@@ -794,9 +821,11 @@ def wait_for_profiler_artifact(path: Path, timeout_s: float = 60.0) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
def start_remote_profiler(url: str, framework: str) -> None:
|
||||
def start_remote_profiler(
|
||||
url: str, framework: str, payload: Optional[dict] = None
|
||||
) -> None:
|
||||
try:
|
||||
post_json(url.rstrip("/") + "/start_profile", timeout=60.0)
|
||||
post_json(url.rstrip("/") + "/start_profile", payload=payload, timeout=60.0)
|
||||
except Exception as exc:
|
||||
if framework == "vllm":
|
||||
raise RuntimeError(
|
||||
@@ -808,11 +837,40 @@ def start_remote_profiler(url: str, framework: str) -> None:
|
||||
raise RuntimeError(
|
||||
"TensorRT-LLM live torch profiling requires "
|
||||
"a server build that exposes POST /start_profile plus the env vars "
|
||||
"TLLM_PROFILE_START_STOP=1 and TLLM_TORCH_PROFILE_TRACE=/shared/path."
|
||||
"TLLM_PROFILE_START_STOP=<start>-<stop> and "
|
||||
"TLLM_TORCH_PROFILE_TRACE=/shared/path."
|
||||
) from exc
|
||||
if framework == "tokenspeed":
|
||||
raise RuntimeError(
|
||||
"TokenSpeed live torch profiling requires a server build that "
|
||||
"exposes POST /start_profile and POST /stop_profile. The helper "
|
||||
"passes output_dir, activities, and profile_id in the start payload."
|
||||
) from exc
|
||||
raise
|
||||
|
||||
|
||||
def build_remote_profiler_start_payload(
|
||||
framework: str,
|
||||
output_path: Path,
|
||||
profile_prefix: Optional[str],
|
||||
stage: Optional[str],
|
||||
) -> Optional[dict]:
|
||||
if framework != "tokenspeed":
|
||||
return None
|
||||
|
||||
profile_id = profile_prefix or "triage-trace"
|
||||
if stage:
|
||||
profile_id = f"{profile_id}-{stage}"
|
||||
|
||||
return {
|
||||
"output_dir": str(output_path),
|
||||
"activities": ["CPU", "GPU"],
|
||||
"with_stack": True,
|
||||
"record_shapes": False,
|
||||
"profile_id": profile_id,
|
||||
}
|
||||
|
||||
|
||||
def stop_remote_profiler(url: str, framework: str) -> None:
|
||||
try:
|
||||
post_json(url.rstrip("/") + "/stop_profile", timeout=300.0)
|
||||
@@ -829,6 +887,7 @@ def run_remote_profiler(
|
||||
framework: str,
|
||||
probe_plan: ProbePlan,
|
||||
probe_delay: float,
|
||||
profile_prefix: Optional[str] = None,
|
||||
stage: Optional[str] = None,
|
||||
) -> Path:
|
||||
framework = canonicalize_framework(framework)
|
||||
@@ -843,7 +902,11 @@ def run_remote_profiler(
|
||||
if output_path.exists()
|
||||
else set()
|
||||
)
|
||||
model = discover_openai_model(url) if framework in {"vllm", "trtllm"} else None
|
||||
model = (
|
||||
discover_openai_model(url)
|
||||
if framework in {"vllm", "trtllm", "tokenspeed"}
|
||||
else None
|
||||
)
|
||||
if probe_plan.warmup_requests > 0:
|
||||
send_probe_requests(
|
||||
url=url,
|
||||
@@ -854,13 +917,18 @@ def run_remote_profiler(
|
||||
model=model,
|
||||
)
|
||||
|
||||
start_remote_profiler(url, framework)
|
||||
start_payload = build_remote_profiler_start_payload(
|
||||
framework=framework,
|
||||
output_path=output_path,
|
||||
profile_prefix=profile_prefix,
|
||||
stage=stage,
|
||||
)
|
||||
start_remote_profiler(url, framework, payload=start_payload)
|
||||
stop_error: Optional[BaseException] = None
|
||||
try:
|
||||
if probe_plan.capture_requests > 0:
|
||||
# `sglang.profiler` performs its own startup work before it reaches
|
||||
# POST /start_profile. A very short delay can send probes too early
|
||||
# and miss the profiling window entirely.
|
||||
# Server-side profilers may do setup work after POST /start_profile.
|
||||
# A very short delay can send probes too early and miss the window.
|
||||
time.sleep(max(5.0, probe_delay))
|
||||
send_probe_requests(
|
||||
url=url,
|
||||
@@ -1064,14 +1132,14 @@ def run_profiler(
|
||||
if profile_by_stage:
|
||||
raise ValueError(
|
||||
"--profile-by-stage is only supported for SGLang live capture. "
|
||||
"Disable it when profiling vLLM or TensorRT-LLM."
|
||||
"Disable it when profiling vLLM, TensorRT-LLM, or TokenSpeed."
|
||||
)
|
||||
if merge_profiles:
|
||||
raise ValueError(
|
||||
"--merge-profiles is only supported for SGLang live capture. "
|
||||
"Disable it when profiling vLLM or TensorRT-LLM."
|
||||
"Disable it when profiling vLLM, TensorRT-LLM, or TokenSpeed."
|
||||
)
|
||||
if profile_prefix:
|
||||
if profile_prefix and resolved_framework in {"vllm", "trtllm"}:
|
||||
print(
|
||||
f"Note: {framework_display_name(resolved_framework)} ignores "
|
||||
"--profile-prefix on the HTTP profiler control path.",
|
||||
@@ -1093,6 +1161,7 @@ def run_profiler(
|
||||
warmup_steps=warmup_steps,
|
||||
),
|
||||
probe_delay=probe_delay,
|
||||
profile_prefix=profile_prefix,
|
||||
)
|
||||
output_root = ensure_remote_profiler_output_path(output_dir, resolved_framework)
|
||||
for stage in stages:
|
||||
@@ -1116,6 +1185,7 @@ def run_profiler(
|
||||
warmup_steps=warmup_steps,
|
||||
),
|
||||
probe_delay=probe_delay,
|
||||
profile_prefix=profile_prefix,
|
||||
stage=stage,
|
||||
)
|
||||
return output_root
|
||||
|
||||
@@ -12,9 +12,10 @@ FRAMEWORK_LABELS = {
|
||||
"sglang": "SGLang",
|
||||
"vllm": "vLLM",
|
||||
"trtllm": "TensorRT-LLM",
|
||||
"tokenspeed": "TokenSpeed",
|
||||
}
|
||||
|
||||
FRAMEWORK_ORDER = {"sglang": 0, "vllm": 1, "trtllm": 2}
|
||||
FRAMEWORK_ORDER = {"sglang": 0, "vllm": 1, "trtllm": 2, "tokenspeed": 3}
|
||||
|
||||
|
||||
def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
|
||||
@@ -75,6 +76,8 @@ def framework_key_from_path(path: Path) -> str:
|
||||
return "vllm"
|
||||
if "trtllm" in lowered or "tensorrt" in lowered:
|
||||
return "trtllm"
|
||||
if "tokenspeed" in lowered or "token-speed" in lowered:
|
||||
return "tokenspeed"
|
||||
return "other"
|
||||
|
||||
|
||||
|
||||
@@ -530,16 +530,16 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
likely_share=0.5,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="DSA fused metadata copy for graph replay",
|
||||
pattern="NSA fused metadata copy for graph replay",
|
||||
candidate_path="python/sglang/jit_kernel/fused_metadata_copy.py",
|
||||
active_keywords=(
|
||||
"fused_metadata_copy",
|
||||
"fused_metadata_copy_multi",
|
||||
"fused_dsa_cache_seqlens",
|
||||
"fused_nsa_cache_seqlens",
|
||||
"fused_flashmla_metadata",
|
||||
),
|
||||
rationale_hint=(
|
||||
"DSA replay metadata copies are already fused into one-kernel" " families."
|
||||
"NSA replay metadata copies are already fused into one-kernel" " families."
|
||||
),
|
||||
min_share=0.02,
|
||||
likely_share=0.2,
|
||||
@@ -744,23 +744,23 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
likely_share=1.5,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="DSA fused top-k transform / page-table build",
|
||||
candidate_path="python/sglang/srt/layers/attention/dsa_backend.py",
|
||||
pattern="NSA fused top-k transform / page-table build",
|
||||
candidate_path="python/sglang/srt/layers/attention/nsa_backend.py",
|
||||
active_keywords=(
|
||||
"fast_topk_transform_fused",
|
||||
"fast_topk_transform_ragged_fused",
|
||||
),
|
||||
rationale_hint=(
|
||||
"DSA top-k metadata preparation already has fused transform kernels."
|
||||
"NSA top-k metadata preparation already has fused transform kernels."
|
||||
),
|
||||
min_share=0.05,
|
||||
likely_share=0.3,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="DSA fused quantize + indexed K-cache store",
|
||||
pattern="NSA fused quantize + indexed K-cache store",
|
||||
candidate_path=(
|
||||
"python/sglang/jit_kernel/fused_store_index_cache.py"
|
||||
"<br>python/sglang/srt/layers/attention/dsa/dsa_indexer.py"
|
||||
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
||||
),
|
||||
active_keywords=("fused_store_index_k_cache",),
|
||||
split_groups=(
|
||||
@@ -768,7 +768,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
("index_k", "cache", "store"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"DSA already has a fused quantize-and-indexed-store kernel family."
|
||||
"NSA already has a fused quantize-and-indexed-store kernel family."
|
||||
),
|
||||
min_share=0.2,
|
||||
likely_share=1.0,
|
||||
@@ -855,6 +855,180 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
likely_share=1.0,
|
||||
priority=90,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="SGLang LTX2 fused Ada values",
|
||||
candidate_path=(
|
||||
"PR #29390"
|
||||
"<br>python/sglang/jit_kernel/diffusion/triton/ltx2_ada_values.py"
|
||||
"<br>python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py"
|
||||
),
|
||||
active_keywords=(
|
||||
"ltx2_ada_values9",
|
||||
"ltx2_ada_values",
|
||||
"LTX2TransformerBlock",
|
||||
),
|
||||
split_groups=(
|
||||
("scale_shift_table", "timestep", "reshape"),
|
||||
("get_ada_values", "ada", "adaln"),
|
||||
("slice", "split", "unbind"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"SGLang mainline fuses LTX-2.3 Ada value materialization for"
|
||||
" video/audio streams; split Ada table add/reshape/slice ladders"
|
||||
" should be checked against this diffusion Triton kernel first."
|
||||
),
|
||||
origin="upstream",
|
||||
model_include=("ltx", "ltx-2", "ltx2"),
|
||||
min_share=0.2,
|
||||
likely_share=1.0,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="SGLang LTX2 residual-gate add CUDA fast path",
|
||||
candidate_path=(
|
||||
"PR #29361"
|
||||
"<br>python/sglang/jit_kernel/diffusion/residual_gate_add.py"
|
||||
"<br>python/sglang/jit_kernel/csrc/diffusion/residual_gate_add.cuh"
|
||||
"<br>python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py"
|
||||
),
|
||||
active_keywords=(
|
||||
"diffusion_residual_gate_add",
|
||||
"residual_gate_add",
|
||||
"_ltx2_residual_gate_add",
|
||||
),
|
||||
split_groups=(
|
||||
("add", "mul", "gate"),
|
||||
("residual", "update", "gate"),
|
||||
("hidden_states", "attn_hidden_states", "gate"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"SGLang mainline fuses LTX2 residual + update * gate sites into"
|
||||
" a CUDA custom op; split add/mul gate ladders should be checked"
|
||||
" against this path before proposing a new diffusion elementwise"
|
||||
" fusion."
|
||||
),
|
||||
origin="upstream",
|
||||
model_include=("ltx", "ltx-2", "ltx2"),
|
||||
min_share=0.2,
|
||||
likely_share=1.0,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="TokenSpeed CuTe DSL MLA prefill / decode",
|
||||
candidate_path=(
|
||||
"python/tokenspeed/runtime/layers/attention/backends/tokenspeed_mla.py"
|
||||
"<br>tokenspeed-mla/python/tokenspeed_mla/mla_decode.py"
|
||||
"<br>tokenspeed-mla/python/tokenspeed_mla/mla_prefill.py"
|
||||
"<br>tokenspeed-kernel/python/tokenspeed_kernel/ops/attention/"
|
||||
"tokenspeed_mla/__init__.py"
|
||||
),
|
||||
active_keywords=(
|
||||
"tokenspeed_mla_decode",
|
||||
"tokenspeed_mla_prefill",
|
||||
"BlackwellMultiHeadLatentAttentionForward",
|
||||
),
|
||||
split_groups=(
|
||||
("mla", "flashmla", "attention", "fmha"),
|
||||
("prefill", "decode", "verify"),
|
||||
("fp8", "kv_cache", "page_table"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"TokenSpeed ships Blackwell CuTe DSL MLA prefill/decode kernels;"
|
||||
" split MLA support kernels should be checked against backend"
|
||||
" selection before being called novel."
|
||||
),
|
||||
origin="upstream",
|
||||
model_include=("deepseek", "kimi", "qwen3.5", "qwen3_5"),
|
||||
min_share=0.4,
|
||||
likely_share=2.0,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="TokenSpeed MLA KV pack + FP8 quantize",
|
||||
candidate_path=(
|
||||
"tokenspeed-mla/python/tokenspeed_mla/mla_kv_pack_quantize_fp8.py"
|
||||
"<br>tokenspeed-kernel/python/tokenspeed_kernel/ops/attention/"
|
||||
"tokenspeed_mla/__init__.py"
|
||||
),
|
||||
active_keywords=(
|
||||
"_mla_kv_pack_quantize_fp8_kernel",
|
||||
"mla_kv_pack_quantize_fp8",
|
||||
),
|
||||
split_groups=(
|
||||
("k_nope", "k_pe", "cat", "concat", "pack"),
|
||||
("quant", "fp8", "float8"),
|
||||
("v", "kv", "cache"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"TokenSpeed fuses MLA K/V pack, concat, and FP8 quantization into"
|
||||
" one Triton kernel for chunked prefill."
|
||||
),
|
||||
origin="upstream",
|
||||
model_include=("deepseek", "kimi", "qwen3.5", "qwen3_5"),
|
||||
min_share=0.2,
|
||||
likely_share=1.0,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="TokenSpeed fused top-k + top-p sampling",
|
||||
candidate_path=(
|
||||
"tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/" # codespell:ignore thirdparty
|
||||
"fused_topk_topp.py"
|
||||
"<br>tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/" # codespell:ignore thirdparty
|
||||
"csrc/fused_topk_topp/fused_topk_topp.cu"
|
||||
),
|
||||
active_keywords=("fused_topk_topp", "fused_topk_topp_renorm"),
|
||||
split_groups=(
|
||||
("topk", "top_k"),
|
||||
("topp", "top_p"),
|
||||
("sampling", "renorm", "softmax"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"TokenSpeed has a fused top-k/top-p renormalization path for"
|
||||
" decode sampling."
|
||||
),
|
||||
origin="upstream",
|
||||
min_share=0.1,
|
||||
likely_share=0.8,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="TokenSpeed persistent lm_head GEMM",
|
||||
candidate_path=(
|
||||
"tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/" # codespell:ignore thirdparty
|
||||
"lm_head_gemm.py"
|
||||
"<br>tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/" # codespell:ignore thirdparty
|
||||
"csrc/lm_head_gemm.cu"
|
||||
),
|
||||
active_keywords=("lm_head_gemm",),
|
||||
split_groups=(
|
||||
("lm_head", "logits", "vocab"),
|
||||
("gemm", "matmul", "linear"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"TokenSpeed has a shape-gated persistent lm_head GEMM path; visible"
|
||||
" lm_head matmul ladders should be compared against it."
|
||||
),
|
||||
origin="upstream",
|
||||
model_include=("kimi", "qwen"),
|
||||
min_share=0.2,
|
||||
likely_share=1.0,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="TokenSpeed NVFP4 GEMM + SwiGLU + quant",
|
||||
candidate_path=(
|
||||
"tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cute_dsl/" # codespell:ignore thirdparty
|
||||
"nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
),
|
||||
active_keywords=("nvfp4_gemm_swiglu_nvfp4_quant",),
|
||||
split_groups=(
|
||||
("gemm", "nvfp4", "fp4"),
|
||||
("swiglu", "silu", "activation", "mul"),
|
||||
("quant", "scale", "sfc"),
|
||||
),
|
||||
rationale_hint=(
|
||||
"TokenSpeed's CuTe DSL kernel fuses NVFP4 GEMM, SwiGLU, and"
|
||||
" optional output quantization in one expert-style path."
|
||||
),
|
||||
origin="upstream",
|
||||
min_share=0.3,
|
||||
likely_share=1.5,
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="vLLM-origin Attention + Quantization",
|
||||
candidate_path=(
|
||||
@@ -1235,6 +1409,8 @@ def source_location_priority(location: str) -> int:
|
||||
return 290 - penalty
|
||||
if text.startswith("vllm/"):
|
||||
return 285 - penalty
|
||||
if text.startswith("python/tokenspeed/") or text.startswith("tokenspeed/"):
|
||||
return 283 - penalty
|
||||
if text.startswith("tensorrt_llm/"):
|
||||
return 280 - penalty
|
||||
if text.startswith("sgl_kernel/"):
|
||||
@@ -1254,6 +1430,8 @@ def is_preferred_source_location(location: str) -> bool:
|
||||
text.startswith("python/sglang/")
|
||||
or text.startswith("sglang/")
|
||||
or text.startswith("vllm/")
|
||||
or text.startswith("python/tokenspeed/")
|
||||
or text.startswith("tokenspeed/")
|
||||
or text.startswith("tensorrt_llm/")
|
||||
or text.startswith("sgl_kernel/")
|
||||
)
|
||||
@@ -1316,6 +1494,10 @@ def frame_priority(frame_name: str) -> int:
|
||||
return 290 - penalty
|
||||
if normalized_text.startswith("vllm/"):
|
||||
return 285 - penalty
|
||||
if normalized_text.startswith("python/tokenspeed/") or normalized_text.startswith(
|
||||
"tokenspeed/"
|
||||
):
|
||||
return 283 - penalty
|
||||
if normalized_text.startswith("tensorrt_llm/"):
|
||||
return 280 - penalty
|
||||
if normalized_text.startswith("sgl_kernel/"):
|
||||
@@ -1329,6 +1511,8 @@ def frame_priority(frame_name: str) -> int:
|
||||
return 120
|
||||
if "/vllm/" in raw_text:
|
||||
return 118
|
||||
if "/tokenspeed/" in raw_text or "/TokenSpeed/" in raw_text:
|
||||
return 117
|
||||
if "/TensorRT-LLM/" in raw_text or "/tensorrt_llm/" in raw_text:
|
||||
return 116
|
||||
return 100
|
||||
@@ -1336,6 +1520,10 @@ def frame_priority(frame_name: str) -> int:
|
||||
return 110
|
||||
if ".py(" in raw_text and "/vllm/" in raw_text:
|
||||
return 108
|
||||
if ".py(" in raw_text and (
|
||||
"/tokenspeed/" in raw_text or "/TokenSpeed/" in raw_text
|
||||
):
|
||||
return 107
|
||||
if ".py(" in raw_text and (
|
||||
"/TensorRT-LLM/" in raw_text or "/tensorrt_llm/" in raw_text
|
||||
):
|
||||
@@ -2438,6 +2626,8 @@ def fusion_framework_hints(spec: FusionPatternSpec) -> set[str]:
|
||||
hints: set[str] = set()
|
||||
if "vllm/" in text:
|
||||
hints.add("vllm")
|
||||
if any(token in text for token in ("tokenspeed/", "tokenspeed-", "tokenspeed_")):
|
||||
hints.add("tokenspeed")
|
||||
if "tensorrt_llm/" in text:
|
||||
hints.add("trtllm")
|
||||
if any(token in text for token in ("python/sglang/", "sgl-kernel/", "sgl_kernel/")):
|
||||
|
||||
@@ -416,6 +416,10 @@ def is_meaningful_python_scope(name: str) -> bool:
|
||||
return True
|
||||
if normalized.startswith("vllm/"):
|
||||
return True
|
||||
if normalized.startswith("python/tokenspeed/") or normalized.startswith(
|
||||
"tokenspeed/"
|
||||
):
|
||||
return True
|
||||
if normalized.startswith("tensorrt_llm/"):
|
||||
return True
|
||||
if normalized.startswith("sgl_kernel/"):
|
||||
@@ -697,6 +701,8 @@ def choose_best_scope(scope_chain: Sequence[str]) -> Optional[str]:
|
||||
score += 48.0
|
||||
elif scope.startswith("vllm/"):
|
||||
score += 46.0
|
||||
elif scope.startswith("python/tokenspeed/") or scope.startswith("tokenspeed/"):
|
||||
score += 45.0
|
||||
elif scope.startswith("tensorrt_llm/"):
|
||||
score += 44.0
|
||||
elif scope.startswith("sgl_kernel/"):
|
||||
@@ -744,6 +750,10 @@ def source_scope_priority(scope: Optional[str]) -> int:
|
||||
return 290 - penalty
|
||||
if normalized.startswith("vllm/"):
|
||||
return 285 - penalty
|
||||
if normalized.startswith("python/tokenspeed/") or normalized.startswith(
|
||||
"tokenspeed/"
|
||||
):
|
||||
return 283 - penalty
|
||||
if normalized.startswith("tensorrt_llm/"):
|
||||
return 280 - penalty
|
||||
if normalized.startswith("sgl_kernel/"):
|
||||
|
||||
+1
@@ -58,6 +58,7 @@ Always rule out these existing families first:
|
||||
- SANA packed self-attention Q/K/V and cross-attention K/V GEMMs
|
||||
- fused diffusion `QK norm + RoPE`
|
||||
- LTX2 split RoPE
|
||||
- LTX2 residual-gate add
|
||||
- varlen USP attention pack/scatter
|
||||
- NVFP4 / Nunchaku packed QKV
|
||||
- Nunchaku fused GELU MLP
|
||||
|
||||
+17
-1
@@ -17,14 +17,18 @@ framework-specific optimization workflow.
|
||||
- `python/sglang/jit_kernel/diffusion/triton/rmsnorm_onepass.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/rotary.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/ltx2_rotary.py`
|
||||
- `python/sglang/jit_kernel/diffusion/residual_gate_add.py`
|
||||
- `python/sglang/jit_kernel/csrc/diffusion/residual_gate_add.cuh`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/varlen_pack_pad.py`
|
||||
- `python/sglang/jit_kernel/diffusion/cutedsl/scale_residual_norm_scale_shift.py`
|
||||
- `test/registered/jit/diffusion/test_qwen_image_modulation.py`
|
||||
- `test/registered/jit/diffusion/test_group_norm_silu.py`
|
||||
- `test/registered/jit/diffusion/test_residual_gate_add.py`
|
||||
- `test/registered/jit/diffusion/test_varlen_pack_pad.py`
|
||||
- `test/registered/jit/diffusion/test_varlen_uspattn_equivalence.py`
|
||||
- `test/registered/jit/benchmark/diffusion/bench_qwen_image_modulation.py`
|
||||
- `test/registered/jit/benchmark/diffusion/bench_group_norm_silu.py`
|
||||
- `test/registered/jit/benchmark/diffusion/bench_residual_gate_add.py`
|
||||
- `python/sglang/jit_kernel/norm.py`
|
||||
- `python/sglang/multimodal_gen/runtime/platforms/cuda.py`
|
||||
- `python/sglang/multimodal_gen/runtime/layers/attention/selector.py`
|
||||
@@ -87,7 +91,17 @@ framework-specific optimization workflow.
|
||||
- Constraints: `cos` and `sin` shapes must match `[B, H, S, head_dim / 2]`, and `inner_dim == H * head_dim`.
|
||||
- Workflow rule: if LTX-2 traces show a large split-RoPE PyTorch chain, check whether the LTX2-specific Triton path was disabled by shape or dtype before proposing a new RoPE kernel.
|
||||
|
||||
8. HunyuanVideo / LTX upsampler GroupNorm + SiLU fusion
|
||||
8. LTX2 residual-gate add fusion
|
||||
- Kernel: `diffusion_residual_gate_add`
|
||||
- Locations: `diffusion/residual_gate_add.py`, `csrc/diffusion/residual_gate_add.cuh`, `runtime/models/dits/ltx_2.py`
|
||||
- Use case: `residual + update * gate` in LTX2 self-attention, prompt cross-attention, audio/video cross-attention, and feed-forward residual updates.
|
||||
- Constraints: `residual`, `update`, and `gate` must be CUDA tensors on the same device, contiguous, same dtype (`fp16`, `bf16`, or `fp32`), with `update.shape == residual.shape`; `gate` can match `residual` or be row-broadcast with the last dimension matching.
|
||||
- Behavior: `_ltx2_residual_gate_add(...)` uses the CUDA custom op while guards pass. On a runtime exception outside `torch.compile`, it logs once, disables the fast path for the process, and falls back to `residual + update * gate`.
|
||||
- Validation: `test/registered/jit/diffusion/test_residual_gate_add.py`.
|
||||
- Microbench: `test/registered/jit/benchmark/diffusion/bench_residual_gate_add.py`.
|
||||
- Workflow rule: if LTX2 traces show repeated elementwise `mul` + `add` ladders around attention or MLP residuals, check whether this existing CUDA path was disabled by shape, dtype, contiguity, or a prior runtime failure before proposing another elementwise fusion.
|
||||
|
||||
9. HunyuanVideo / LTX upsampler GroupNorm + SiLU fusion
|
||||
- Kernel: `triton_group_norm_silu`
|
||||
- Locations: `diffusion/group_norm_silu.py`, `triton/group_norm_silu.py`, `runtime/models/vaes/hunyuanvae.py`, `runtime/models/upsampler/latent_upsampler.py`
|
||||
- Use case: `activation(group_norm(x))` when the activation is non-inplace `nn.SiLU` and the GroupNorm is affine.
|
||||
@@ -178,6 +192,7 @@ framework-specific optimization workflow.
|
||||
- QK norm: `apply_qk_norm` used in `flux.py`, `flux_2.py`, `qwen_image.py`, `zimage.py`, `wanvideo.py`, `ltx_2.py`, `hunyuanvideo.py`.
|
||||
- QK norm + RoPE: `apply_qk_norm_rope` in `layernorm.py`; use this path when the model wants fused attention prep instead of separate QK norm and RoPE calls.
|
||||
- LTX2 split RoPE: `apply_ltx2_split_rotary_emb` in `ltx_2.py`.
|
||||
- LTX2 residual-gate add: `_ltx2_residual_gate_add` in `ltx_2.py` wraps the CUDA `diffusion_residual_gate_add` custom op for attention, cross-attention, and MLP residual updates.
|
||||
- Varlen USP attention: `fused_pack_qkv` and `fused_scatter_to_padded` in `attention/layer.py`.
|
||||
- SANA packed projections: `to_qkv` and `to_kv` in `sana.py`.
|
||||
- Nunchaku fused GELU MLP: `_fused_gelu_mlp` in `flux.py` for quantized FLUX-family checkpoints.
|
||||
@@ -207,6 +222,7 @@ relying on any file path, flag, or claim about whether the work has merged.
|
||||
- #18897 dual norm fusion for FLUX-family paths (draft).
|
||||
- #20429 Qwen-Image layernorm and `fuse_scale_shift_gate_select01` work.
|
||||
- #20530 MOVA fused RMSNorm + interleaved RoPE.
|
||||
- #29361 LTX2 residual-gate CUDA fast path for `residual + update * gate`.
|
||||
- VAE and decode-side acceleration:
|
||||
- #22531 LTX2 parallel VAE support and #20927 batched tiled VAE decode (draft).
|
||||
- Attention, communication, and runtime scheduling:
|
||||
|
||||
@@ -285,6 +285,6 @@ about whether the work has merged:
|
||||
- **Offload tuning**: after the first request, the runtime logs peak GPU memory and which components could stay resident. Use this to decide which `--*-cpu-offload` flags to disable.
|
||||
- **Backend selection**: `--backend sglang` (default, auto-detected) enables native optimizations (fused kernels, SP, native Cache-DiT env knobs, etc.). `--backend diffusers` falls back to Diffusers pipelines and is the path that accepts `--cache-dit-config` plus diffusers attention backend names.
|
||||
- **Wan2.2-I2V sizing**: explicit `--width/--height` on `Wan2.2-I2V-A14B` control the target area while preserving the condition-image aspect ratio.
|
||||
- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers GroupNorm+SiLU, Z-Image residual-form modulation, fused diffusion `QK norm + RoPE`, LTX2 split RoPE, varlen USP pack/scatter, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all.
|
||||
- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers GroupNorm+SiLU, Z-Image residual-form modulation, fused diffusion `QK norm + RoPE`, LTX2 split RoPE, LTX2 residual-gate add, varlen USP pack/scatter, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all.
|
||||
- **NVFP4 trace interpretation**: on FLUX.2 NVFP4 and Nunchaku-style checkpoints, packed QKV is expected. SGLang intentionally uses fused projection modules such as `to_qkv` / `to_added_qkv` instead of separate `to_q` / `to_k` / `to_v`, so a split-QKV trace usually means the quantized path did not engage rather than a brand new fusion opportunity.
|
||||
- **Hotspot workflow split**: use `sglang-diffusion-benchmark-profile` to prove and classify a slowdown with perf dumps plus `torch.profiler`; hand concrete kernel work off with the perf/profile evidence attached instead of expanding the benchmark skill.
|
||||
|
||||
Reference in New Issue
Block a user