docs(cookbook): re-benchmark DeepSeek-V4 on sglang 0.5.15 (#31363)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
co-authored by
Claude Opus 4.8
Zijie Xia
parent
9057db9417
commit
4a55fdba0b
@@ -115,12 +115,17 @@ than restating.
|
||||
- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
|
||||
that only released FP8/FP4 is a factual bug.
|
||||
- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
|
||||
from the `sglang serve` deploy command.
|
||||
from the `sglang serve` deploy command, and should carry `--flush-cache`: bench_serving's
|
||||
`random` prompts are deterministic, so a warm rerun hits the radix cache and inflates
|
||||
throughput — speed numbers are measured cache-cold.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
- **Latency percentile**: `config.latencyPercentile` (default `"P50"`, or `"Mean"`) matches the
|
||||
percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. (`"Mean"` is
|
||||
temporary — legacy data is being re-measured to P50.)
|
||||
percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. A benchmarks
|
||||
entry may carry its own `latencyPercentile` to override the page value per cell
|
||||
(entry → config → `"P50"`): on a P50 page, kept legacy Mean cells must set it — a
|
||||
`sglang_version` tag alone doesn't convey the percentile. (`"Mean"` is temporary — legacy
|
||||
data is being re-measured to P50.)
|
||||
- **Throughput convention**: `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU**
|
||||
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
|
||||
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||
|
||||
Reference in New Issue
Block a user