[Benchmark] Add agentic rollout simulator and offline explorer (#40034)

2026-09-18 10:06:13 -07:00
parent 4e0b56c811
commit 50a7de47d5
13 changed files with 2815 additions and 0 deletions
@@ -0,0 +1,12 @@
+Status: implemented
+
+# Rollout Simulator and Explorer
+
+**TL;DR:** A portable client makes growing-conversation cache experiments repeatable.
+
+- [feature-00-session-hicache-latency.md](feature-00-session-hicache-latency.md): workload, measurements, and validation contract.
+- [evidence/feature-00-session-hicache-latency.md](evidence/feature-00-session-hicache-latency.md): exact validation commands and measured results.
+
+## Boundaries
+
+This spec covers the standalone benchmark, not the serving engine.
@@ -0,0 +1,52 @@
+Status: implemented
+
+# Validation evidence
+
+## Packaging checks — 2026-09-17
+
+17 CPU tests pass: all three request modes and both stream formats, default DP routing, verified-single-worker omitted ranks, strict multiworker ranks, context/output validation, aborted/truncated streams, timeouts and cleanup. Measurement tests cover occupancy, per-label resets, missing/idle counters, token-weighted denominators, separate exporters, old/interrupted recordings, partial failed requests and safe self-contained HTML.
+
+~~~bash
+uv run --no-project --with aiohttp --with transformers --with prometheus-client \
+  python -m unittest discover -s benchmark/agentic-rollout/tests -v
+~~~
+
+Viewers were rebuilt from the two saved Qwen runs below, each with 16 conversations and 96 requests. Both tabs, all 16 lanes, zoom, request selection, legends and static plots were checked. Embedded-asset tests pass, and the browser renders with external resource loading blocked by the page policy. Direct file-URL navigation was blocked by the browser automation policy, so that path was not manually verified. Packaging does not rerun GPU inference. Raw recordings, generated HTML and machine-specific orchestration stay outside this PR.
+
+## Saved Qwen comparison — 2026-09-17
+
+One H200, Qwen/Qwen3-1.7B, a downstream SGLang serving build at revision 41da06adca698c0032f75000445087f0922dbe43. These saved runs are not GPU validation against current upstream main. Ordinary sessions, session-aware cache, HiCache ratio 1.6, write-through, direct I/O, layer-first layout and resolved FA3 attention. One fresh-server run per page size.
+
+Launch command, with executable and port normalized for portability:
+
+~~~bash
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-1.7B --host 127.0.0.1 --port 30000 \
+  --enable-session-radix-cache --enable-hierarchical-cache \
+  --hicache-ratio 1.6 --hicache-write-policy write_through \
+  --hicache-io-backend direct --hicache-mem-layout layer_first \
+  --mem-fraction-static 0.5 --max-total-tokens 32768 --context-length 8192 \
+  --cuda-graph-max-bs-decode 8 --enable-metrics --stream-interval 1 \
+  --page-size "$PAGE"
+
+uv run benchmark/agentic-rollout/simulate.py \
+  --tokenizer Qwen/Qwen3-1.7B --mode ordinary \
+  --conversations 16 --concurrency 16 --turns 6 \
+  --initial-tokens 1024 --tool-tokens 256 --output-tokens 128 \
+  --tool-delay 1 3 --start-spread 1 --seed 101 \
+  --disable-dp-sticky-routing --output-dir "results/page$PAGE"
+~~~
+
+The recorded client disabled routing hints because this single-worker server omitted its response rank. The packaged client now accepts that omission with default sticky routing only for verified DP=1; this adjustment is CPU-tested, not GPU-rerun.
+
+| Measurement | Page 1 | Page 64 |
+| --- | ---: | ---: |
+| Successful requests | 96/96 | 96/96 |
+| GPU / CPU capacity (tokens) | 32,768 / 52,429 | 32,768 / 52,480 |
+| Duration | 18.54 s | 14.96 s |
+| Output throughput | 662.8 tokens/s | 821.1 tokens/s |
+| Turns 4–5 TTFT p95 | 2,502.49 ms | 20.73 ms |
+| Turns 4–5 mean time per output token | 10.56 ms | 2.33 ms |
+| Last observed eviction / restore counters | 92,200 / 76,088 | 66,112 / 49,984 |
+
+Limitations: one short run per setting, no reverse-order repeat. Client inputs, delays and lengths matched, but server-generated random seeds differed; 85/96 output hashes matched. This validates the client and exposes cache pressure; it does not establish a general page-size speedup or a kernel-level cause. Counters are last scraped values and may miss the tail.
@@ -0,0 +1,27 @@
+Status: implemented
+
+# Synthetic session workload
+
+**TL;DR:** A portable HTTP client compares full-history requests, ordinary sessions, and streaming sessions using independent synthetic conversations. Server launch settings control feature ablations.
+
+```text
+generate -> seeded tool delay -> append synthetic result -> repeat
+finish configured turns -> close
+```
+
+The client runs exactly `--conversations` conversations, each with `--turns` sequential generations. Each conversation uses fixed input/output lengths and a distinct prefix. Actual output token IDs remain in its history. Conversations run concurrently, each with sequential turns; tool waits release concurrency slots. Client-side DP sticky routing is enabled by default: every turn of conversation `i` routes to rank `i % dp_size`. `--disable-dp-sticky-routing` omits the routing hint. All modes stream responses for timing.
+
+The client records resolved server settings, its arguments/version, exact request timings, streamed token-count increments, output hashes, session events, and one-second raw metrics. Request errors stop the run and trigger bounded cleanup; evidence survives. Metrics errors remain visible. Missing measurements remain unavailable.
+
+Separate plots show time and turn number, TTFT p50/p95, observed output throughput, average token timing, token-weighted GPU/CPU/total cache hit, running/queued requests, occupancy, and available transfer rates. Independent metrics exporters remain separate. Several tokens arriving together are not individual token-gap samples.
+
+Cache-pressure claims require observed restores and measured capacity. Public smoke success does not establish slowdown reproduction. Exact commands and measured results belong in [evidence](evidence/feature-00-session-hicache-latency.md).
+
+## Boundaries
+
+Synthetic text and simulated tool delays only. No trainer, training-stack dependency, real tool execution, engine changes, server orchestration, or private deployment details in the shared client. Features are tested only when supported by the chosen server; no silent compatibility fallback.
+
+
+## Explorer packaging (2026-09-17)
+
+Keep the client, plotter, shared Prometheus parser, viewer assets and tests in benchmark/agentic-rollout/. The explorer embeds all assets/data without a frontend dependency. Timeline contains only sampling, tool call and client wait using exact timestamps; missing old phases remain unavailable. Engine Metrics retains 15 panels with L1/L2 in KV Usage, KV Tokens and Cache Hit Rate. Exporters remain separately selectable. Missing/reset/idle counter intervals are gaps. Preserve failed-run status and partial measurements. Verify with CPU tests and both saved 16-conversation Qwen recordings; no new GPU run is required.