Files
sglang/rust/sglang-bench

sglang-bench

The serving benchmark client, ported from python/sglang/benchmark/serving.py.

Why

The Python client drives every request and parses every response stream on one asyncio thread, and it builds a fresh aiohttp session, and so a fresh connection, per request. Both bite as the request count grows: past some point the client, not the server, sets the measured throughput, and the benchmark reports the client's limit. Here each request is a Tokio task on a shared pooled HTTP client, so stream parsing spreads across --worker-threads cores and connections are reused.

What that is worth was measured on one host against a mock server, 96 output tokens per request at concurrency 256, two repeats each:

Requests Python Rust
500 1.8 s, 27.1k tok/s 1.7 s, 28.9k tok/s
2500 106.8 s, 2.2k tok/s 13.2 s, 18.2k tok/s

So: no difference at 500 requests, and 8x at 2500, with the cliff somewhere between. Mean TTFT follows the same shape, 196 ms against 153 ms at 500 requests and 446 ms against 144 ms at 2500.

Treat those figures as a direction, not a specification. The mock is a Python ThreadingHTTPServer sharing the host's CPUs with the client, so it caps both clients and punishes the Python one's connection churn harder than a real server might. An earlier run of a different workload on the same setup showed 2x rather than 8x. The client-side CPU drawn per request is comparable between the two, which says the Python client here is blocked rather than compute-bound, so where your own cliff falls depends on your server. Measure it on the server you care about before quoting a number.

Usage

Python loads it as the sglang.srt.rust_extensions._bench extension module, the same way the Rust server is loaded. Two entry points:

# Its own command line. Imports no model tooling, so it starts fast.
python -m sglang.benchmark.rust_client \
  --backend sglang --dataset-name random \
  --num-prompts 20000 --random-input-len 1024 --random-output-len 512 \
  --max-concurrency 512

# Or from the existing entry point, for a caller already on that command line.
python -m sglang.benchmark.serving --rust-client \
  --backend sglang --dataset-name random --num-prompts 20000

The Rust client owns the flag definitions, so --help on the first form is authoritative. Flag names, defaults, the printed report, and the result .jsonl keys are the Python client's, so an existing command line runs unchanged and anything reading the result file keeps working. --worker-threads and --disable-retokenize are the additions; through serving.py the first is spelled --rust-worker-threads.

In process:

from sglang.benchmark.rust_client import run_rust_benchmark

result = run_rust_benchmark(args)  # args as `serving.py` parses them

--rust-client refuses a configuration the Rust client does not implement rather than measuring something else; serving.py reports it as a usage error.

Building

A wheel build picks the crate up from [package.metadata.sglang], as it does for the other extensions. In a source tree the loader compiles and caches it on first use, which costs about a minute once and roughly 8 seconds of workspace fingerprinting per run after that; a wheel's bundled module skips both. To stage it by hand:

cd rust && cargo build --release -p sglang-bench
cp target/release/libsglang_bench.so \
  ../python/sglang/srt/rust_extensions/_bench.cpython-312-aarch64-linux-gnu.so

Covered

  • Backends: sglang / sglang-native (/generate), sglang-oai, vllm, lmdeploy (/v1/completions), sglang-oai-chat, vllm-chat, lmdeploy-chat (/v1/chat/completions), streaming and not.
  • Protocols: --protocol http (default) and --protocol grpc, the Rust server's sglang.api.v1 service (below).
  • Datasets: random, random-ids, sharegpt.
  • Poisson request pacing (--request-rate), a concurrency cap, warmup, cache flush, profiler start/stop, the readiness probe, --extra-request-body, custom headers, and API-key auth.
  • The full metric set, including the per-second peak-throughput and peak-concurrency windows, with NumPy's percentile interpolation and population standard deviation so the numbers line up with the Python script's.

Token ids on both ends

--tokenize-prompt sends input_ids instead of text. On the response side tokens are counted from meta_info.completion_tokens, with the output_ids a frame carries as the fallback, so a server launched with --skip-tokenizer-init, which returns ids and an empty text, measures the same way a decoding server does. The Python client cannot benchmark such a server: it counts a token only where it sees text.

With --dataset-name random-ids --tokenize-prompt --vocab-size N the client loads no tokenizer at all and needs no model files; the retokenized totals then report 0, since there is no text to re-encode. That combination takes the tokenizer and detokenizer out of the measurement on both ends, which is the datapath sglang-server is built around.

gRPC

--protocol grpc sends the same backend's requests over the Rust server's sglang.api.v1 service (Generate, GetModelInfo, GetServerInfo) instead of HTTP. It is a transport choice, not a backend: only the native API has a gRPC service, so it pairs with --backend sglang, and the default port is the server's HTTP port plus 10000, matching --grpc-port. The request is the same schema the /generate route decodes, built once and deserialized into the generated type, and each streamed frame is fed to the same accounting as an SSE frame through its JSON form, so an HTTP run and a gRPC run differ only in transport and are directly comparable. The result line carries a protocol key, the one key the Python client does not write, so the two kinds of run stay apart in a shared file. Two differences to know about: all request streams share one HTTP/2 connection, and cache flush and the profiler are HTTP-only controls, so a gRPC run reports and skips them.

Not covered

Flags for these are absent rather than accepted and ignored, because a silently dropped flag would report a benchmark nobody ran.

  • Backends: trt, gserver, truss, and the embedding backends.
  • Datasets: image, mmmu, mooncake, agentic-trace, generated-shared-prefix, custom, openai, longbench-v2, speed-bench, and trace-timestamp replay.
  • Multi-turn replay, LoRA request distributions, --cache-report, --plot-throughput, PD-separated profiling, and the per-stage profiler options.

Reach for the Python script for any of those. Both write the same result shape, so a mixed set of runs stays comparable.

Parity

Counted quantities match exactly. Against one mock server with --random-range-ratio 1.0 (which fixes every length, making the two clients' different random draws irrelevant), both clients report identical completed, total_input_tokens, total_output_tokens, total_output_tokens_retokenized and accept_length, and their result files have identical key sets.

Sampling does not match: the two use different random number generators, so the same --seed draws different prompts. A run reproduces itself, and distributions match, but individual prompts do not.

Latency figures are not expected to match either, and that is the point: the Python client's TTFT on the run above is 2.4x the Rust client's on the same server, because its single parse thread is queueing.

Tests

cargo test -p sglang-bench covers the wire parsers against recorded frame sequences, the statistics against NumPy's definitions, dataset sampling, and argument and config validation. CI runs them through the workspace cargo test in test/registered/rust/test_run_rust_tests.py.

test/registered/unit/benchmark/test_rust_client.py covers the Python boundary (config translation and the refusals) with the extension mocked, so it needs no compiled artifact.