docs(cookbook): total (input+output) throughput per GPU + percentile latency labels (#30214)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-07-06 00:39:48 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 5f98f62a8a
commit 7c9bb316cf
13 changed files with 168 additions and 132 deletions
@@ -101,7 +101,7 @@ your dispatch prompt, or ask for it.
upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
Output Examples verbatim); §5 benchmark blocks → transcribe each measured
block: deploy command used, bench command (dataset/isl/osl/num-prompts/
concurrency), Mean TTFT/TPOT, output tok/s, hardware, version string.
concurrency), P50 (median) TTFT/TPOT, output tok/s, hardware, version string.
- Inbound-anchor sweep: `grep -rn "<PageName>" docs_new/ --include='*.mdx'` —
find links/`#fragments` into this page (`mint broken-links` does NOT check
fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
@@ -174,7 +174,11 @@ hardware owners sign off on at review.
### 4. Benchmarks file
One entry per measured block only (cells without entries already render
"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
output tok/s ÷ (tp × nnodes); TTFT/TPOT take the Mean rows; put the workload's
total (in+out) tok/s/GPU = `output tok/s ÷ (tp × nnodes) ×
(isl+osl)/osl` — stored directly (the card shows it as-is). TTFT/TPOT
take the P50 (median) rows; set `config.latencyPercentile` (default `"P50"`; use
`"Mean"` only for legacy Mean-recorded data — temporary, being migrated to P50).
Put the workload's
`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
the benchmarks carry accuracy data** — the engine ships no default eval set
(#27842), so missing labels means the accuracy rows silently don't render;