docs(cookbook): total (input+output) throughput per GPU + percentile latency labels (#30214)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
5f98f62a8a
commit
7c9bb316cf
@@ -49,6 +49,7 @@ the full contract):
|
||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
|
||||
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `accuracyLabels` (below) + `benchmarkCommands.accuracy`. |
|
||||
| `accuracyLabels` | `[key, label, unit][]` | The eval set rendered in the benchmark card and the "⚡ Reproduce" modal — **the engine ships no default**, every config declares its own (e.g. DSv4: GPQA/AIME25/GSM8K; Qwen3.5: GSM8K/MMMU). Required whenever the benchmarks carry accuracy data; without it the accuracy rows silently don't render. Every key used in `benchmarks[].accuracy`, `defaultAccuracy`, and `benchmarkCommands.accuracy` must appear here. |
|
||||
| `latencyPercentile` | `"Mean" \| "P50"` | Optional, **temporary**; the percentile the benchmark TTFT/TPOT values are. **Default `"P50"`** — the card renders `TTFT (<pct>)` / `TPOT (<pct>)`. Set `"Mean"` only for legacy data recorded as Mean (being re-measured to P50). `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU** = `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. |
|
||||
| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the model's HF id (`<hf-org>/<model-slug>`); it prefills the issue template's free-form `model` input when the issue opens. **Don't prune this block** — without it the engine falls back to `deepseek-ai/deepseek-v4` and submissions from your page get mislabeled. |
|
||||
|
||||
## 2.2 Author the 5-dim matrix (`cells[]`)
|
||||
|
||||
@@ -6,7 +6,10 @@
|
||||
// One entry per cell `match` tuple (same 5 keys as config cells). The card stays
|
||||
// "pending" until an entry has a non-null speed metric or accuracy. Speed shape:
|
||||
// speed: [{ workload: {dataset, isl, osl, max_concurrency}, ttft_ms, tpot_ms,
|
||||
// tokens_per_sec_per_gpu }, ...] // interactivity is derived (1000/TPOT)
|
||||
// tokens_per_sec_per_gpu }, ...]
|
||||
// - ttft_ms/tpot_ms are P50 (median); set config.latencyPercentile ("P50" default, or "Mean").
|
||||
// - tokens_per_sec_per_gpu = total (in+out) tok/s/GPU
|
||||
// (= output tok/s ÷ GPUs × (isl+osl)/osl). interactivity is derived = 1000/TPOT (tokens/s/user).
|
||||
// Per-cell `accuracy: { <key>: <pct> }` overrides the config's defaultAccuracy.
|
||||
|
||||
export const benchmarks = [
|
||||
|
||||
@@ -101,7 +101,7 @@ your dispatch prompt, or ask for it.
|
||||
upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
|
||||
Output Examples verbatim); §5 benchmark blocks → transcribe each measured
|
||||
block: deploy command used, bench command (dataset/isl/osl/num-prompts/
|
||||
concurrency), Mean TTFT/TPOT, output tok/s, hardware, version string.
|
||||
concurrency), P50 (median) TTFT/TPOT, output tok/s, hardware, version string.
|
||||
- Inbound-anchor sweep: `grep -rn "<PageName>" docs_new/ --include='*.mdx'` —
|
||||
find links/`#fragments` into this page (`mint broken-links` does NOT check
|
||||
fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
|
||||
@@ -174,7 +174,11 @@ hardware owners sign off on at review.
|
||||
### 4. Benchmarks file
|
||||
One entry per measured block only (cells without entries already render
|
||||
"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
|
||||
output tok/s ÷ (tp × nnodes); TTFT/TPOT take the Mean rows; put the workload's
|
||||
total (in+out) tok/s/GPU = `output tok/s ÷ (tp × nnodes) ×
|
||||
(isl+osl)/osl` — stored directly (the card shows it as-is). TTFT/TPOT
|
||||
take the P50 (median) rows; set `config.latencyPercentile` (default `"P50"`; use
|
||||
`"Mean"` only for legacy Mean-recorded data — temporary, being migrated to P50).
|
||||
Put the workload's
|
||||
`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
|
||||
the benchmarks carry accuracy data** — the engine ships no default eval set
|
||||
(#27842), so missing labels means the accuracy rows silently don't render;
|
||||
|
||||
@@ -118,6 +118,11 @@ than restating.
|
||||
from the `sglang serve` deploy command.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
- **Latency percentile**: `config.latencyPercentile` (default `"P50"`, or `"Mean"`) matches the
|
||||
percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. (`"Mean"` is
|
||||
temporary — legacy data is being re-measured to P50.)
|
||||
- **Throughput convention**: `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU**
|
||||
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
|
||||
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||
column must be produced by the SAME harness — flag a page that, say, measures one
|
||||
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
|
||||
|
||||
@@ -23,6 +23,9 @@
|
||||
// accuracyLabels [key, label, unit][] — the eval set shown in the
|
||||
// benchmark card + "⚡ Reproduce". NO engine default:
|
||||
// required whenever benchmarks carry accuracy data
|
||||
// latencyPercentile optional, TEMPORARY — "Mean" | "P50" (default "P50"); the
|
||||
// percentile the TTFT/TPOT values are, shown as "TTFT (<pct>)".
|
||||
// Legacy "Mean" data is being re-measured to P50; drop once done
|
||||
// multiNodeHints optional — {[hwId]: string[]} prepended as `# ...` lines
|
||||
// dockerImages optional — `docker run` image, keyed by `hw|quant`
|
||||
// then `hw`; falls back to `lmsysorg/sglang:dev`
|
||||
@@ -574,11 +577,14 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
const renderBenchmarkCard = (entry) => {
|
||||
// [key, label, unit, compute?]. Optional compute(measurement) supplies
|
||||
// derived metrics (preferred over measurement[key] when present).
|
||||
const pct = config.latencyPercentile || "P50";
|
||||
const SPEED_LABELS = [
|
||||
["ttft_ms", "TTFT", "ms"],
|
||||
["tpot_ms", "TPOT", "ms"],
|
||||
["tokens_per_sec_per_gpu", "tokens/sec/GPU", ""],
|
||||
["interactivity", "interactivity", "tok/s",
|
||||
["ttft_ms", `TTFT (${pct})`, "ms"],
|
||||
["tpot_ms", `TPOT (${pct})`, "ms"],
|
||||
// throughput per gpu = total(input+output)/elapsed/GPU;
|
||||
// stored directly in the benchmarks file (= output tok/s/GPU × (isl+osl)/osl).
|
||||
["tokens_per_sec_per_gpu", "throughput per gpu", "tok/s"],
|
||||
["interactivity", "interactivity", "tokens/s/user",
|
||||
(m) => (m.tpot_ms != null && m.tpot_ms !== 0)
|
||||
? Math.round((1000 / m.tpot_ms) * 10) / 10
|
||||
: null],
|
||||
@@ -668,7 +674,11 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
])}
|
||||
</div>
|
||||
{legend && (
|
||||
<div style={s.benchLegend}>{legend}</div>
|
||||
<div style={s.benchLegend}>
|
||||
{(Array.isArray(legend) ? legend : [legend]).map((line, i) => (
|
||||
<div key={`legend-${i}`}>{line}</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
@@ -693,7 +703,10 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
});
|
||||
return { title: "Speed", sharedText, colHeaders, rows,
|
||||
colCount: measurements.length,
|
||||
legend: "interactivity = 1000 / TPOT(ms)" };
|
||||
legend: [
|
||||
`throughput per gpu = (input+output tokens)/elapsed/GPU`,
|
||||
`interactivity = 1000/TPOT(ms) (tokens/s/user)`,
|
||||
] };
|
||||
};
|
||||
|
||||
// One row per ACCURACY_LABELS entry with a non-null value; single value column.
|
||||
|
||||
@@ -13,9 +13,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 287.24, tpot_ms: 2.4, tokens_per_sec_per_gpu: 325.11 },
|
||||
ttft_ms: 287.24, tpot_ms: 2.4, tokens_per_sec_per_gpu: 650 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 171.72, tpot_ms: 11.87, tokens_per_sec_per_gpu: 7875.37 },
|
||||
ttft_ms: 171.72, tpot_ms: 11.87, tokens_per_sec_per_gpu: 15751 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -23,9 +23,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 18.9, tpot_ms: 2.08, tokens_per_sec_per_gpu: 471.61 },
|
||||
ttft_ms: 18.9, tpot_ms: 2.08, tokens_per_sec_per_gpu: 943 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 180.55, tpot_ms: 7, tokens_per_sec_per_gpu: 13049.7 },
|
||||
ttft_ms: 180.55, tpot_ms: 7, tokens_per_sec_per_gpu: 26099 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -33,9 +33,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 16.23, tpot_ms: 2.19, tokens_per_sec_per_gpu: 449.06 },
|
||||
ttft_ms: 16.23, tpot_ms: 2.19, tokens_per_sec_per_gpu: 898 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 127.45, tpot_ms: 5.31, tokens_per_sec_per_gpu: 17430.9 },
|
||||
ttft_ms: 127.45, tpot_ms: 5.31, tokens_per_sec_per_gpu: 34862 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -43,9 +43,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 18.8, tpot_ms: 1.65, tokens_per_sec_per_gpu: 590.39 },
|
||||
ttft_ms: 18.8, tpot_ms: 1.65, tokens_per_sec_per_gpu: 1181 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 476.85, tpot_ms: 4.26, tokens_per_sec_per_gpu: 18745.3 },
|
||||
ttft_ms: 476.85, tpot_ms: 4.26, tokens_per_sec_per_gpu: 37491 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -53,9 +53,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 23.74, tpot_ms: 1.77, tokens_per_sec_per_gpu: 546.07 },
|
||||
ttft_ms: 23.74, tpot_ms: 1.77, tokens_per_sec_per_gpu: 1092 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1128.14, tpot_ms: 4.54, tokens_per_sec_per_gpu: 14280.32 },
|
||||
ttft_ms: 1128.14, tpot_ms: 4.54, tokens_per_sec_per_gpu: 28561 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -63,9 +63,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 17.04, tpot_ms: 2.1, tokens_per_sec_per_gpu: 468.74 },
|
||||
ttft_ms: 17.04, tpot_ms: 2.1, tokens_per_sec_per_gpu: 937 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 195.67, tpot_ms: 5.05, tokens_per_sec_per_gpu: 17694.7 },
|
||||
ttft_ms: 195.67, tpot_ms: 5.05, tokens_per_sec_per_gpu: 35389 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -73,9 +73,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 22.01, tpot_ms: 1.54, tokens_per_sec_per_gpu: 630.21 },
|
||||
ttft_ms: 22.01, tpot_ms: 1.54, tokens_per_sec_per_gpu: 1260 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1676.37, tpot_ms: 3.38, tokens_per_sec_per_gpu: 14483.4 },
|
||||
ttft_ms: 1676.37, tpot_ms: 3.38, tokens_per_sec_per_gpu: 28967 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -83,9 +83,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 26.01, tpot_ms: 1.34, tokens_per_sec_per_gpu: 713.36 },
|
||||
ttft_ms: 26.01, tpot_ms: 1.34, tokens_per_sec_per_gpu: 1427 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1604.2, tpot_ms: 3.38, tokens_per_sec_per_gpu: 14852.1 },
|
||||
ttft_ms: 1604.2, tpot_ms: 3.38, tokens_per_sec_per_gpu: 29704 },
|
||||
],
|
||||
},
|
||||
// ====================================================================
|
||||
@@ -96,9 +96,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 48.8, tpot_ms: 2.23, tokens_per_sec_per_gpu: 426.61 },
|
||||
ttft_ms: 48.8, tpot_ms: 2.23, tokens_per_sec_per_gpu: 853 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 119.9, tpot_ms: 11.96, tokens_per_sec_per_gpu: 7913.04 },
|
||||
ttft_ms: 119.9, tpot_ms: 11.96, tokens_per_sec_per_gpu: 15826 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -106,9 +106,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 20.97, tpot_ms: 2.2, tokens_per_sec_per_gpu: 445.43 },
|
||||
ttft_ms: 20.97, tpot_ms: 2.2, tokens_per_sec_per_gpu: 891 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 601.53, tpot_ms: 5.37, tokens_per_sec_per_gpu: 14874 },
|
||||
ttft_ms: 601.53, tpot_ms: 5.37, tokens_per_sec_per_gpu: 29748 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -116,9 +116,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 21.39, tpot_ms: 2.22, tokens_per_sec_per_gpu: 440.08 },
|
||||
ttft_ms: 21.39, tpot_ms: 2.22, tokens_per_sec_per_gpu: 880 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 398.87, tpot_ms: 5.58, tokens_per_sec_per_gpu: 15212.9 },
|
||||
ttft_ms: 398.87, tpot_ms: 5.58, tokens_per_sec_per_gpu: 30426 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -126,9 +126,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 22.51, tpot_ms: 1.72, tokens_per_sec_per_gpu: 564.53 },
|
||||
ttft_ms: 22.51, tpot_ms: 1.72, tokens_per_sec_per_gpu: 1129 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 880.23, tpot_ms: 4.37, tokens_per_sec_per_gpu: 15765.2 },
|
||||
ttft_ms: 880.23, tpot_ms: 4.37, tokens_per_sec_per_gpu: 31530 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -136,9 +136,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 18.74, tpot_ms: 1.74, tokens_per_sec_per_gpu: 561.43 },
|
||||
ttft_ms: 18.74, tpot_ms: 1.74, tokens_per_sec_per_gpu: 1123 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 458.07, tpot_ms: 4.47, tokens_per_sec_per_gpu: 17892.27 },
|
||||
ttft_ms: 458.07, tpot_ms: 4.47, tokens_per_sec_per_gpu: 35785 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -146,9 +146,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 20.85, tpot_ms: 2.09, tokens_per_sec_per_gpu: 468.85 },
|
||||
ttft_ms: 20.85, tpot_ms: 2.09, tokens_per_sec_per_gpu: 938 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 781.82, tpot_ms: 5.23, tokens_per_sec_per_gpu: 14492.3 },
|
||||
ttft_ms: 781.82, tpot_ms: 5.23, tokens_per_sec_per_gpu: 28985 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -156,9 +156,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 20.88, tpot_ms: 1.32, tokens_per_sec_per_gpu: 732.43 },
|
||||
ttft_ms: 20.88, tpot_ms: 1.32, tokens_per_sec_per_gpu: 1465 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1550.27, tpot_ms: 3.26, tokens_per_sec_per_gpu: 15472 },
|
||||
ttft_ms: 1550.27, tpot_ms: 3.26, tokens_per_sec_per_gpu: 30944 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -166,9 +166,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 23.48, tpot_ms: 1.2, tokens_per_sec_per_gpu: 798.74 },
|
||||
ttft_ms: 23.48, tpot_ms: 1.2, tokens_per_sec_per_gpu: 1597 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1544.41, tpot_ms: 3.14, tokens_per_sec_per_gpu: 15617.7 },
|
||||
ttft_ms: 1544.41, tpot_ms: 3.14, tokens_per_sec_per_gpu: 31235 },
|
||||
],
|
||||
},
|
||||
// ====================================================================
|
||||
@@ -179,9 +179,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 124.36, tpot_ms: 2, tokens_per_sec_per_gpu: 436.42 },
|
||||
ttft_ms: 124.36, tpot_ms: 2, tokens_per_sec_per_gpu: 873 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 154.77, tpot_ms: 7.54, tokens_per_sec_per_gpu: 12343.9 },
|
||||
ttft_ms: 154.77, tpot_ms: 7.54, tokens_per_sec_per_gpu: 24688 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -189,9 +189,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 11.22, tpot_ms: 1.19, tokens_per_sec_per_gpu: 818.26 },
|
||||
ttft_ms: 11.22, tpot_ms: 1.19, tokens_per_sec_per_gpu: 1637 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1223.9, tpot_ms: 2.19, tokens_per_sec_per_gpu: 21137.2 },
|
||||
ttft_ms: 1223.9, tpot_ms: 2.19, tokens_per_sec_per_gpu: 42274 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -199,9 +199,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 11.12, tpot_ms: 1.19, tokens_per_sec_per_gpu: 818.62 },
|
||||
ttft_ms: 11.12, tpot_ms: 1.19, tokens_per_sec_per_gpu: 1637 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1230.34, tpot_ms: 2.18, tokens_per_sec_per_gpu: 21121.5 },
|
||||
ttft_ms: 1230.34, tpot_ms: 2.18, tokens_per_sec_per_gpu: 42243 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -209,9 +209,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 12.18, tpot_ms: 0.91, tokens_per_sec_per_gpu: 1065.73 },
|
||||
ttft_ms: 12.18, tpot_ms: 0.91, tokens_per_sec_per_gpu: 2131 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1177.6, tpot_ms: 1.92, tokens_per_sec_per_gpu: 22636.7 },
|
||||
ttft_ms: 1177.6, tpot_ms: 1.92, tokens_per_sec_per_gpu: 45273 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -219,9 +219,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 12.28, tpot_ms: 0.84, tokens_per_sec_per_gpu: 1158.08 },
|
||||
ttft_ms: 12.28, tpot_ms: 0.84, tokens_per_sec_per_gpu: 2316 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1550.61, tpot_ms: 1.93, tokens_per_sec_per_gpu: 19205.62 },
|
||||
ttft_ms: 1550.61, tpot_ms: 1.93, tokens_per_sec_per_gpu: 38411 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -229,9 +229,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 11.98, tpot_ms: 1.19, tokens_per_sec_per_gpu: 817.6 },
|
||||
ttft_ms: 11.98, tpot_ms: 1.19, tokens_per_sec_per_gpu: 1635 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 1367.79, tpot_ms: 2.27, tokens_per_sec_per_gpu: 19794.3 },
|
||||
ttft_ms: 1367.79, tpot_ms: 2.27, tokens_per_sec_per_gpu: 39589 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -239,9 +239,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 11.55, tpot_ms: 1.22, tokens_per_sec_per_gpu: 807.13 },
|
||||
ttft_ms: 11.55, tpot_ms: 1.22, tokens_per_sec_per_gpu: 1614 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 935.24, tpot_ms: 2.34, tokens_per_sec_per_gpu: 23135.8 },
|
||||
ttft_ms: 935.24, tpot_ms: 2.34, tokens_per_sec_per_gpu: 46272 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -249,9 +249,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.0.0.dev1+g631db6c75",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 1, num_prompts: 10 },
|
||||
ttft_ms: 12.09, tpot_ms: 0.92, tokens_per_sec_per_gpu: 1053.2 },
|
||||
ttft_ms: 12.09, tpot_ms: 0.92, tokens_per_sec_per_gpu: 2106 },
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 100, num_prompts: 1000 },
|
||||
ttft_ms: 939.41, tpot_ms: 2.25, tokens_per_sec_per_gpu: 23880.7 },
|
||||
ttft_ms: 939.41, tpot_ms: 2.25, tokens_per_sec_per_gpu: 47761 },
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
@@ -11,6 +11,9 @@
|
||||
export const config = {
|
||||
modelName: "LFM2.5",
|
||||
|
||||
// TTFT/TPOT were recorded as Mean (no percentile restated in the source runs).
|
||||
latencyPercentile: "Mean",
|
||||
|
||||
supportedHardware: ["h100", "h200", "b200"],
|
||||
|
||||
variants: [
|
||||
|
||||
@@ -39,7 +39,7 @@ export const benchmarks = [
|
||||
speed: [
|
||||
// bench_serving --flush-cache, MSA path, tp8; warm steady-state (3-run, identical).
|
||||
{ workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64, num_prompts: 128 },
|
||||
ttft_ms: 1580, tpot_ms: 24.1, tokens_per_sec_per_gpu: 265 },
|
||||
ttft_ms: 1580, tpot_ms: 24.1, tokens_per_sec_per_gpu: 2385 },
|
||||
],
|
||||
accuracy: { gpqa_pct: 89.1, gsm8k_pct: 96.5, mmmu_pro_pct: 72.7 }, // 2026-06-15, sgl-eval --thinking, recommended sampling (temp 1.0/top_p 0.95), tp8. GSM8K full 1319 = 96.51% (greedy 96.89%). GPQA Diamond 198, n-repeats 4 = pass@1[avg-of-4] 89.14% +/-1.73% (pass@4 95.45%, majority@4 93.52%). MMMU-Pro 2026-06-18, sgl-eval "standard (10 options)" test split, full 1730, single-shot 72.66% (thinking, temp 1.0/top_p 0.95).
|
||||
},
|
||||
@@ -51,7 +51,7 @@ export const benchmarks = [
|
||||
speed: [
|
||||
// bench_serving --flush-cache, bf16 Triton path; warm steady-state (3-run, cold-start run-1 excluded).
|
||||
{ workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64, num_prompts: 128 },
|
||||
ttft_ms: 1054, tpot_ms: 70.8, tokens_per_sec_per_gpu: 116 },
|
||||
ttft_ms: 1054, tpot_ms: 70.8, tokens_per_sec_per_gpu: 1044 },
|
||||
],
|
||||
accuracy: { gsm8k_pct: 97.0 }, // #27944, sgl-eval --thinking, full 1319, recommended sampling (temp 1.0/top_p 0.95/top_k 40); stable 97.04% across all 3 runs (std 0.0)
|
||||
},
|
||||
@@ -60,7 +60,7 @@ export const benchmarks = [
|
||||
sglang_version: "PR #27944",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64 },
|
||||
ttft_ms: null, tpot_ms: 32.8, tokens_per_sec_per_gpu: 365 },
|
||||
ttft_ms: null, tpot_ms: 32.8, tokens_per_sec_per_gpu: 3285 },
|
||||
],
|
||||
accuracy: { gsm8k_pct: null }, // TODO: pending sgl-eval re-measure on B300 (legacy few_shot 200: 87.5)
|
||||
},
|
||||
@@ -71,21 +71,21 @@ export const benchmarks = [
|
||||
sglang_version: "PR #27944",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 2048, osl: 256, max_concurrency: 64 },
|
||||
ttft_ms: 4746, tpot_ms: 39.3, tokens_per_sec_per_gpu: 277 },
|
||||
ttft_ms: 4746, tpot_ms: 39.3, tokens_per_sec_per_gpu: 2493 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 256, max_concurrency: 24 },
|
||||
ttft_ms: 3324, tpot_ms: 32.9, tokens_per_sec_per_gpu: 131 },
|
||||
ttft_ms: 3324, tpot_ms: 32.9, tokens_per_sec_per_gpu: 4323 },
|
||||
],
|
||||
accuracy: { gsm8k_pct: null }, // TODO: pending sgl-eval re-measure on GB300 (legacy few_shot 200: 87.5)
|
||||
},
|
||||
// MI355X (gfx950): native MXFP8. Speed: bench_serving 1024/1024 @ conc 64, tp8
|
||||
// -> 1678 output tok/s (3355 total incl. input); 1678 / 8 = ~210 tokens/sec/GPU.
|
||||
// -> 1678 output tok/s (3355 total incl. input); 3355 / 8 = ~420 tokens/sec/GPU (total, in+out).
|
||||
// No TTFT/TPOT reported for this run.
|
||||
{
|
||||
match: { hw: "mi355x", variant: "default", quant: "mxfp8", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "PR #27944",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 1024, osl: 1024, max_concurrency: 64, num_prompts: 640 },
|
||||
ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: 210 },
|
||||
ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: 420 },
|
||||
],
|
||||
accuracy: { gsm8k_pct: null }, // TODO: pending sgl-eval re-measure on MI355X (legacy run_eval 1319: 92.2)
|
||||
},
|
||||
|
||||
@@ -14,6 +14,9 @@
|
||||
export const config = {
|
||||
modelName: "MiniMax-M3",
|
||||
|
||||
// TTFT/TPOT were recorded as Mean (no percentile restated in the source runs).
|
||||
latencyPercentile: "Mean",
|
||||
|
||||
supportedHardware: ["b200", "b300", "gb200", "gb300", "mi300x", "mi325x", "mi350x", "mi355x", "h200"],
|
||||
|
||||
variants: [
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
// DeepSeek-V4 per-cell benchmark numbers, keyed by the same `match` tuple as
|
||||
// deepseek-v4.jsx cells. See _deployment.jsx for the speed/accuracy schema.
|
||||
// Measured on sglang v0.5.12.post1.
|
||||
// tokens_per_sec_per_gpu is total (input+output) tok/s/GPU: fp4/fp8 = measured
|
||||
// output/GPU × (isl+osl)/osl; nvfp4 was measured as total already.
|
||||
export const benchmarks = [
|
||||
// ====================================================================
|
||||
// B200 + FP4
|
||||
@@ -10,9 +12,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 87, tpot_ms: 3.68, tokens_per_sec_per_gpu: 65 },
|
||||
ttft_ms: 87, tpot_ms: 3.68, tokens_per_sec_per_gpu: 585 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 290, tpot_ms: 6.21, tokens_per_sec_per_gpu: 489 },
|
||||
ttft_ms: 290, tpot_ms: 6.21, tokens_per_sec_per_gpu: 4401 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -20,9 +22,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 4228, tpot_ms: 60.98, tokens_per_sec_per_gpu: 225 },
|
||||
ttft_ms: 4228, tpot_ms: 60.98, tokens_per_sec_per_gpu: 2025 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 4628, tpot_ms: 88.25, tokens_per_sec_per_gpu: 643 },
|
||||
ttft_ms: 4628, tpot_ms: 88.25, tokens_per_sec_per_gpu: 5787 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -30,9 +32,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 99949, tpot_ms: 67.46, tokens_per_sec_per_gpu: 939 },
|
||||
ttft_ms: 99949, tpot_ms: 67.46, tokens_per_sec_per_gpu: 8451 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 253310, tpot_ms: 66.11, tokens_per_sec_per_gpu: 964 },
|
||||
ttft_ms: 253310, tpot_ms: 66.11, tokens_per_sec_per_gpu: 8676 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -43,9 +45,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2326, tpot_ms: 69.9, tokens_per_sec_per_gpu: 99 },
|
||||
ttft_ms: 2326, tpot_ms: 69.9, tokens_per_sec_per_gpu: 891 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 7242, tpot_ms: 152.09, tokens_per_sec_per_gpu: 192 },
|
||||
ttft_ms: 7242, tpot_ms: 152.09, tokens_per_sec_per_gpu: 1728 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -59,9 +61,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 88, tpot_ms: 3.67, tokens_per_sec_per_gpu: 66 },
|
||||
ttft_ms: 88, tpot_ms: 3.67, tokens_per_sec_per_gpu: 594 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 266, tpot_ms: 6.06, tokens_per_sec_per_gpu: 495 },
|
||||
ttft_ms: 266, tpot_ms: 6.06, tokens_per_sec_per_gpu: 4455 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -69,9 +71,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2363, tpot_ms: 34.4, tokens_per_sec_per_gpu: 402 },
|
||||
ttft_ms: 2363, tpot_ms: 34.4, tokens_per_sec_per_gpu: 3618 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 2812, tpot_ms: 51.65, tokens_per_sec_per_gpu: 1092 },
|
||||
ttft_ms: 2812, tpot_ms: 51.65, tokens_per_sec_per_gpu: 9828 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -79,9 +81,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 97028, tpot_ms: 65.09, tokens_per_sec_per_gpu: 966 },
|
||||
ttft_ms: 97028, tpot_ms: 65.09, tokens_per_sec_per_gpu: 8694 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 243335, tpot_ms: 63.98, tokens_per_sec_per_gpu: 998 },
|
||||
ttft_ms: 243335, tpot_ms: 63.98, tokens_per_sec_per_gpu: 8982 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -89,9 +91,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 261, tpot_ms: 5.01, tokens_per_sec_per_gpu: 23 },
|
||||
ttft_ms: 261, tpot_ms: 5.01, tokens_per_sec_per_gpu: 207 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 364, tpot_ms: 11.37, tokens_per_sec_per_gpu: 137 },
|
||||
ttft_ms: 364, tpot_ms: 11.37, tokens_per_sec_per_gpu: 1233 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -99,9 +101,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 1866, tpot_ms: 54.48, tokens_per_sec_per_gpu: 139 },
|
||||
ttft_ms: 1866, tpot_ms: 54.48, tokens_per_sec_per_gpu: 1251 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 6325, tpot_ms: 123.95, tokens_per_sec_per_gpu: 237 },
|
||||
ttft_ms: 6325, tpot_ms: 123.95, tokens_per_sec_per_gpu: 2133 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -118,9 +120,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2560, tpot_ms: 39.71, tokens_per_sec_per_gpu: 342 },
|
||||
ttft_ms: 2560, tpot_ms: 39.71, tokens_per_sec_per_gpu: 3078 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 3995, tpot_ms: 82.56, tokens_per_sec_per_gpu: 718 },
|
||||
ttft_ms: 3995, tpot_ms: 82.56, tokens_per_sec_per_gpu: 6462 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -168,9 +170,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 463, tpot_ms: 4.19, tokens_per_sec_per_gpu: 35 },
|
||||
ttft_ms: 463, tpot_ms: 4.19, tokens_per_sec_per_gpu: 315 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 436, tpot_ms: 8.93, tokens_per_sec_per_gpu: 336 },
|
||||
ttft_ms: 436, tpot_ms: 8.93, tokens_per_sec_per_gpu: 3024 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -178,9 +180,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2671, tpot_ms: 45.88, tokens_per_sec_per_gpu: 299 },
|
||||
ttft_ms: 2671, tpot_ms: 45.88, tokens_per_sec_per_gpu: 2691 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 4823, tpot_ms: 94.04, tokens_per_sec_per_gpu: 637 },
|
||||
ttft_ms: 4823, tpot_ms: 94.04, tokens_per_sec_per_gpu: 5733 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -188,9 +190,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 154868, tpot_ms: 104.84, tokens_per_sec_per_gpu: 621 },
|
||||
ttft_ms: 154868, tpot_ms: 104.84, tokens_per_sec_per_gpu: 5589 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 386489, tpot_ms: 103.37, tokens_per_sec_per_gpu: 627 },
|
||||
ttft_ms: 386489, tpot_ms: 103.37, tokens_per_sec_per_gpu: 5643 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -235,9 +237,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 204, tpot_ms: 3.38, tokens_per_sec_per_gpu: 68 },
|
||||
ttft_ms: 204, tpot_ms: 3.38, tokens_per_sec_per_gpu: 612 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 538, tpot_ms: 11.42, tokens_per_sec_per_gpu: 264 },
|
||||
ttft_ms: 538, tpot_ms: 11.42, tokens_per_sec_per_gpu: 2376 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -245,9 +247,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 738, tpot_ms: 36.27, tokens_per_sec_per_gpu: 385 },
|
||||
ttft_ms: 738, tpot_ms: 36.27, tokens_per_sec_per_gpu: 3465 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 39806, tpot_ms: 80.13, tokens_per_sec_per_gpu: 393 },
|
||||
ttft_ms: 39806, tpot_ms: 80.13, tokens_per_sec_per_gpu: 3537 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -255,9 +257,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 195293, tpot_ms: 130.35, tokens_per_sec_per_gpu: 493 },
|
||||
ttft_ms: 195293, tpot_ms: 130.35, tokens_per_sec_per_gpu: 4437 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 502615, tpot_ms: 130.31, tokens_per_sec_per_gpu: 490 },
|
||||
ttft_ms: 502615, tpot_ms: 130.31, tokens_per_sec_per_gpu: 4410 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -277,9 +279,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 193, tpot_ms: 3.38, tokens_per_sec_per_gpu: 67 },
|
||||
ttft_ms: 193, tpot_ms: 3.38, tokens_per_sec_per_gpu: 603 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 598, tpot_ms: 10.46, tokens_per_sec_per_gpu: 308 },
|
||||
ttft_ms: 598, tpot_ms: 10.46, tokens_per_sec_per_gpu: 2772 },
|
||||
],
|
||||
},
|
||||
{
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
export const config = {
|
||||
modelName: "DeepSeek-V4",
|
||||
|
||||
latencyPercentile: "Mean", // temporary; re-measure to P50
|
||||
|
||||
supportedHardware: [
|
||||
"h100", "h200", "b200", "b300", "gb200", "gb300",
|
||||
"rtx6000",
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
// B200 BF16 (tp8): 91.88% · perf A/B (cc1, cc128).
|
||||
// B200 FP8 (tp8): 93.78% — with `--fp8-gemm-backend triton` (DeepGEMM UE8M0 workaround; ~19% slower).
|
||||
// B200 NVFP4 (tp8): 89.38%.
|
||||
// (perf tokens_per_sec_per_gpu = measured output tok/s ÷ 8 GPUs; TTFT = median.)
|
||||
// (perf tokens_per_sec_per_gpu = total (in+out) tok/s/GPU = measured output tok/s ÷ 8 × (isl+osl)/osl; TTFT = median.)
|
||||
//
|
||||
// sglang_version = the build the numbers ran on (PR #28400 + #28604, +#28649 for FP8 load,
|
||||
// +#28662/triton-workaround for Blackwell FP8). H200 numbers taken on a main build @ 3f668733.
|
||||
@@ -23,12 +23,12 @@ export const benchmarks = [
|
||||
verified: true,
|
||||
sglang_version: "main @ 3f668733 (#28400 + #28604)",
|
||||
speed: [
|
||||
// cc=1: median TTFT 81.89 ms, median TPOT 8.91 ms, output 109.96 tok/s (÷8 ≈ 13.7/GPU).
|
||||
// cc=1: median TTFT 81.89 ms, median TPOT 8.91 ms, output 109.96 tok/s (÷8 ≈ 13.7 output/GPU → 69 total/GPU).
|
||||
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 81.9, tpot_ms: 8.91, tokens_per_sec_per_gpu: 13.7 },
|
||||
// cc=128: median TTFT 200.11 ms (mean 1221), median TPOT 52.09 ms, output 2266 tok/s (÷8 ≈ 283/GPU).
|
||||
ttft_ms: 81.9, tpot_ms: 8.91, tokens_per_sec_per_gpu: 69 },
|
||||
// cc=128: median TTFT 200.11 ms (mean 1221), median TPOT 52.09 ms, output 2266 tok/s (÷8 ≈ 283 output/GPU → 1415 total/GPU).
|
||||
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 128 },
|
||||
ttft_ms: 200.1, tpot_ms: 52.1, tokens_per_sec_per_gpu: 283 },
|
||||
ttft_ms: 200.1, tpot_ms: 52.1, tokens_per_sec_per_gpu: 1415 },
|
||||
],
|
||||
accuracy: { gsm8k_pct: 93.02 },
|
||||
},
|
||||
@@ -48,9 +48,9 @@ export const benchmarks = [
|
||||
sglang_version: "PR #28400 + #28604",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 108, tpot_ms: 9.0, tokens_per_sec_per_gpu: 13.6 },
|
||||
ttft_ms: 108, tpot_ms: 9.0, tokens_per_sec_per_gpu: 68 },
|
||||
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 128 },
|
||||
ttft_ms: 170, tpot_ms: 43.3, tokens_per_sec_per_gpu: 331 },
|
||||
ttft_ms: 170, tpot_ms: 43.3, tokens_per_sec_per_gpu: 1655 },
|
||||
],
|
||||
accuracy: { gsm8k_pct: 91.88 },
|
||||
},
|
||||
|
||||
@@ -15,9 +15,9 @@ export const benchmarks = [
|
||||
sglang_version: "main @ 09ca4fc",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 757, tpot_ms: 3.22, tokens_per_sec_per_gpu: 32 },
|
||||
ttft_ms: 757, tpot_ms: 3.22, tokens_per_sec_per_gpu: 288 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 3188, tpot_ms: 9.12, tokens_per_sec_per_gpu: 164 },
|
||||
ttft_ms: 3188, tpot_ms: 9.12, tokens_per_sec_per_gpu: 1476 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -27,9 +27,9 @@ export const benchmarks = [
|
||||
sglang_version: "main @ 09ca4fc",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 5742, tpot_ms: 17.65, tokens_per_sec_per_gpu: 342 },
|
||||
ttft_ms: 5742, tpot_ms: 17.65, tokens_per_sec_per_gpu: 3078 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 18744, tpot_ms: 32.61, tokens_per_sec_per_gpu: 558 },
|
||||
ttft_ms: 18744, tpot_ms: 32.61, tokens_per_sec_per_gpu: 5022 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -40,7 +40,7 @@ export const benchmarks = [
|
||||
sglang_version: "main @ 09ca4fc",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 177620, tpot_ms: 47.99, tokens_per_sec_per_gpu: 451 },
|
||||
ttft_ms: 177620, tpot_ms: 47.99, tokens_per_sec_per_gpu: 4059 },
|
||||
],
|
||||
},
|
||||
// ---- GB300 + FP8 ---- (4-GPU single node, TP4; real weights, --random-range-ratio 1.0, flush-cache every run)
|
||||
@@ -52,9 +52,9 @@ export const benchmarks = [
|
||||
sglang_version: "main @ 09ca4fc",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 374, tpot_ms: 4.55, tokens_per_sec_per_gpu: 51 },
|
||||
ttft_ms: 374, tpot_ms: 4.55, tokens_per_sec_per_gpu: 459 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 3719, tpot_ms: 11.5, tokens_per_sec_per_gpu: 264 },
|
||||
ttft_ms: 3719, tpot_ms: 11.5, tokens_per_sec_per_gpu: 2376 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -64,9 +64,9 @@ export const benchmarks = [
|
||||
sglang_version: "main @ 09ca4fc",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 7429, tpot_ms: 25.21, tokens_per_sec_per_gpu: 493 },
|
||||
ttft_ms: 7429, tpot_ms: 25.21, tokens_per_sec_per_gpu: 4437 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 27488, tpot_ms: 48.43, tokens_per_sec_per_gpu: 756 },
|
||||
ttft_ms: 27488, tpot_ms: 48.43, tokens_per_sec_per_gpu: 6804 },
|
||||
],
|
||||
},
|
||||
// GB300 HT: drop-flags (mfs/cgbs/mrr dropped) + env SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512.
|
||||
@@ -81,7 +81,7 @@ export const benchmarks = [
|
||||
sglang_version: "main @ 09ca4fc",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 231101, tpot_ms: 86.01, tokens_per_sec_per_gpu: 671 },
|
||||
ttft_ms: 231101, tpot_ms: 86.01, tokens_per_sec_per_gpu: 6039 },
|
||||
],
|
||||
},
|
||||
// ---- B300 + FP8 ---- (8-GPU single node, TP8; serve recipe in glm-5.2.jsx; benchmark pending re-measurement)
|
||||
@@ -104,16 +104,16 @@ export const benchmarks = [
|
||||
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
|
||||
// ---- B200 + NVFP4 ---- (8-GPU single node, TP8; nvidia/GLM-5.2-NVFP4 via --quantization modelopt_fp4,
|
||||
// measured on the lmsysorg/sglang:dev-glm52-nvfp4 preview image, flush-cache every run.
|
||||
// ttft_ms/tpot_ms are P50; tokens_per_sec_per_gpu = output tok/s/GPU.
|
||||
// ttft_ms/tpot_ms are P50; tokens_per_sec_per_gpu = total (in+out) tok/s/GPU (output/GPU × (isl+osl)/osl).
|
||||
// balanced & high-throughput add DP-Attention (dp8); low-latency uses MTP 5-1-6, balanced MTP 2-1-3.)
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "dev-glm52-nvfp4",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 295, tpot_ms: 1.85, tokens_per_sec_per_gpu: 58.6 },
|
||||
ttft_ms: 295, tpot_ms: 1.85, tokens_per_sec_per_gpu: 527 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 2491, tpot_ms: 5.43, tokens_per_sec_per_gpu: 254.3 },
|
||||
ttft_ms: 2491, tpot_ms: 5.43, tokens_per_sec_per_gpu: 2289 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -121,9 +121,9 @@ export const benchmarks = [
|
||||
sglang_version: "dev-glm52-nvfp4",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 5837, tpot_ms: 12.70, tokens_per_sec_per_gpu: 418.9 },
|
||||
ttft_ms: 5837, tpot_ms: 12.70, tokens_per_sec_per_gpu: 3770 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 16736, tpot_ms: 30.00, tokens_per_sec_per_gpu: 593.7 },
|
||||
ttft_ms: 16736, tpot_ms: 30.00, tokens_per_sec_per_gpu: 5343 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -131,12 +131,12 @@ export const benchmarks = [
|
||||
sglang_version: "dev-glm52-nvfp4",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 130174, tpot_ms: 67.12, tokens_per_sec_per_gpu: 589.4 },
|
||||
ttft_ms: 130174, tpot_ms: 67.12, tokens_per_sec_per_gpu: 5305 },
|
||||
],
|
||||
},
|
||||
// ---- B300 + NVFP4 ---- (8-GPU single node, TP8; nvidia/GLM-5.2-NVFP4 via --quantization modelopt_fp4,
|
||||
// measured on the lmsysorg/sglang:dev-glm52-nvfp4 preview image, flush-cache every run.
|
||||
// tokens_per_sec_per_gpu = total server output tok/s / 8 GPUs (410→51, 1793→224, 1220→153, 1641→205, 3439→430).
|
||||
// tokens_per_sec_per_gpu = total (in+out) tok/s/GPU (measured output/GPU 51/224/153/205/430 × (isl+osl)/osl).
|
||||
// aime25 overrides the variant default (87.7 → 89.58, measured on this NVFP4 build); gsm8k inherits the default.)
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||
@@ -144,9 +144,9 @@ export const benchmarks = [
|
||||
accuracy: { aime25_pct: 89.58 },
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 196, tpot_ms: 1.86, tokens_per_sec_per_gpu: 51 },
|
||||
ttft_ms: 196, tpot_ms: 1.86, tokens_per_sec_per_gpu: 459 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 274, tpot_ms: 6.95, tokens_per_sec_per_gpu: 224 },
|
||||
ttft_ms: 274, tpot_ms: 6.95, tokens_per_sec_per_gpu: 2016 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -155,9 +155,9 @@ export const benchmarks = [
|
||||
accuracy: { aime25_pct: 89.58 },
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 680, tpot_ms: 48.9, tokens_per_sec_per_gpu: 153 },
|
||||
ttft_ms: 680, tpot_ms: 48.9, tokens_per_sec_per_gpu: 1377 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 3010, tpot_ms: 149, tokens_per_sec_per_gpu: 205 },
|
||||
ttft_ms: 3010, tpot_ms: 149, tokens_per_sec_per_gpu: 1845 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -166,7 +166,7 @@ export const benchmarks = [
|
||||
accuracy: { aime25_pct: 89.58 },
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 6370, tpot_ms: 280, tokens_per_sec_per_gpu: 430 },
|
||||
ttft_ms: 6370, tpot_ms: 280, tokens_per_sec_per_gpu: 3870 },
|
||||
],
|
||||
},
|
||||
// ---- MI355X + FP8 ---- gfx950, TP8, DSA tilelang, NO MTP (disabled on AMD).
|
||||
@@ -177,9 +177,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 634, tpot_ms: 13.56, tokens_per_sec_per_gpu: 9 },
|
||||
ttft_ms: 634, tpot_ms: 13.56, tokens_per_sec_per_gpu: 81 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 5411, tpot_ms: 23.60, tokens_per_sec_per_gpu: 69 },
|
||||
ttft_ms: 5411, tpot_ms: 23.60, tokens_per_sec_per_gpu: 621 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -187,9 +187,9 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 19526, tpot_ms: 46.50, tokens_per_sec_per_gpu: 122 },
|
||||
ttft_ms: 19526, tpot_ms: 46.50, tokens_per_sec_per_gpu: 1098 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 117866, tpot_ms: 56.12, tokens_per_sec_per_gpu: 116 },
|
||||
ttft_ms: 117866, tpot_ms: 56.12, tokens_per_sec_per_gpu: 1044 },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -197,7 +197,7 @@ export const benchmarks = [
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 432058, tpot_ms: 106.44, tokens_per_sec_per_gpu: 141 },
|
||||
ttft_ms: 432058, tpot_ms: 106.44, tokens_per_sec_per_gpu: 1269 },
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
Reference in New Issue
Block a user