[SKILL] Upgrade sglang profile and auto_benchmark skills (#24250)
This commit is contained in:
Xiaoyu Zhang
@@ -0,0 +1,291 @@
|
||||
---
|
||||
name: sglang-prod-incident-triage
|
||||
description: Replay-first debug flow for SGLang serving problems. Use when a live or recent server shows health-check failures, latency or throughput regressions, queue growth, timeouts, distributed stalls, crash dumps, wrong outputs after deploys, or PD/EP/HiCache issues, and the job is to turn the problem into a replay plus the right next debug tool.
|
||||
---
|
||||
|
||||
# SGLang Serving Debug
|
||||
|
||||
## Overview
|
||||
|
||||
Use this skill to turn a live serving problem into a debug path you can replay.
|
||||
|
||||
Use one loop:
|
||||
|
||||
- collect a baseline bundle
|
||||
- save the failing request or crash dump
|
||||
- replay on a clean target
|
||||
- only then switch tools
|
||||
|
||||
Do not start with profiling.
|
||||
|
||||
This skill should work with more focused skills instead of re-implementing them:
|
||||
|
||||
- `debug-cuda-crash` when replay plus coredump points to a CUDA crash path
|
||||
- `debug-distributed-hang` when the problem is clearly a TP/PP/DP/EP hang
|
||||
- `llm-torch-profiler-analysis` when the issue is already narrowed to a
|
||||
compute-side path
|
||||
|
||||
Three examples are included:
|
||||
|
||||
- TTFT spike with low queue time
|
||||
- replay-first CUDA crash flow
|
||||
- request-shaped distributed hang flow
|
||||
|
||||
## Output Contract
|
||||
|
||||
Return:
|
||||
|
||||
- problem class
|
||||
- what was checked
|
||||
- strongest signal so far
|
||||
- current best guess
|
||||
- what was ruled out
|
||||
- next step
|
||||
- production risk
|
||||
|
||||
## When To Use It
|
||||
|
||||
- `/health` or `/health_generate` is unhealthy
|
||||
- latency or throughput regressed under serving load
|
||||
- queue size grows while health still looks green
|
||||
- one request class times out or hangs
|
||||
- the server crashes only after some requests
|
||||
- outputs changed after a deploy, topology change, or weight switch
|
||||
- one older commit is known-good and a newer commit is known-bad
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Collect a baseline bundle
|
||||
|
||||
If a live server is reachable, collect a read-only bundle before anything more
|
||||
intrusive:
|
||||
|
||||
```bash
|
||||
python3 scripts/incident_artifact_tool.py collect-bundle \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--outdir /tmp/incident_bundle
|
||||
|
||||
python3 scripts/incident_artifact_tool.py summarize-bundle \
|
||||
/tmp/incident_bundle
|
||||
```
|
||||
|
||||
If the server is protected:
|
||||
|
||||
```bash
|
||||
python3 scripts/incident_artifact_tool.py collect-bundle \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--token "$SGLANG_BEARER_TOKEN" \
|
||||
--outdir /tmp/incident_bundle
|
||||
```
|
||||
|
||||
The bundle script collects:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/model_info`
|
||||
- `/server_info`
|
||||
- `/v1/loads?include=all`
|
||||
- `/v1/loads?include=core,queues,disagg,spec`
|
||||
- `/metrics`
|
||||
- `/hicache/storage-backend` on a best-effort basis
|
||||
|
||||
Use the summary for a quick read on:
|
||||
|
||||
- health vs. active health state
|
||||
- topology and runtime flags
|
||||
- point-in-time queue and token usage
|
||||
- TTFT / E2E / queue-time heuristics from Prometheus metrics
|
||||
|
||||
If the summary says the bundle was captured while the server was idle, recollect
|
||||
it during traffic or move quickly to dump plus replay.
|
||||
|
||||
If no live server is reachable, start from the best dump or log already available:
|
||||
|
||||
- crash dump
|
||||
- request dump
|
||||
- logs
|
||||
- CUDA coredump
|
||||
- OTel trace
|
||||
- torch profile
|
||||
|
||||
### 2. Save the failing request
|
||||
|
||||
Read [references/decision-tree.md](references/decision-tree.md) only if the
|
||||
problem class is still unclear:
|
||||
|
||||
- server down or unhealthy
|
||||
- latency or throughput regression
|
||||
- wrong output or behavior regression
|
||||
- intermittent timeout or hang
|
||||
|
||||
Then preserve the request payload that actually triggers the problem:
|
||||
|
||||
- crash path: use `--crash-dump-folder`
|
||||
- non-crash path: enable request dump or save the exact trigger request
|
||||
|
||||
Do not jump straight from a live symptom to low-level debugging without first
|
||||
saving something you can replay.
|
||||
|
||||
### 3. Replay on a clean target
|
||||
|
||||
Read [references/endpoints-and-signals.md](references/endpoints-and-signals.md)
|
||||
when you need help reading the baseline bundle or the replay target.
|
||||
|
||||
Read [references/replay-trace-profile.md](references/replay-trace-profile.md)
|
||||
when you need the replay, trace, profile, or bisect paths.
|
||||
|
||||
Standard order:
|
||||
|
||||
1. collect baseline bundle
|
||||
2. capture request dump or crash dump
|
||||
3. restart a clean debug target if needed
|
||||
4. replay the same issue
|
||||
5. collect replay-time logs and dumps
|
||||
|
||||
### 4. Only go deeper after replay
|
||||
|
||||
#### Replay
|
||||
|
||||
Use replay when:
|
||||
|
||||
- a crash dump exists
|
||||
- a request dump exists
|
||||
- the problem depends on request shape or workload mix
|
||||
|
||||
If a crash dump exists, summarize it first:
|
||||
|
||||
```bash
|
||||
python3 scripts/incident_artifact_tool.py summarize-dump \
|
||||
--input-file /path/to/crash_dump.pkl
|
||||
```
|
||||
|
||||
Then replay:
|
||||
|
||||
```bash
|
||||
python3 /path/to/sglang/scripts/playground/replay_request_dump.py \
|
||||
--input-file /path/to/crash_dump.pkl \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--parallel 128
|
||||
```
|
||||
|
||||
If `safe_pickle_load` blocks a locally captured trusted dump, use:
|
||||
|
||||
```bash
|
||||
python3 scripts/replay_trusted_request_dump.py \
|
||||
--input-file /path/to/request_dump.pkl \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--parallel 1
|
||||
```
|
||||
|
||||
If replay indicates a CUDA crash path, restart the same build with coredumps
|
||||
enabled before reproducing again:
|
||||
|
||||
```bash
|
||||
SGLANG_CUDA_COREDUMP=1 \
|
||||
SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
|
||||
python -m sglang.launch_server \
|
||||
--model-path ... \
|
||||
--crash-dump-folder /tmp/sglang_crash_dump \
|
||||
...
|
||||
```
|
||||
|
||||
Then inspect the generated coredump:
|
||||
|
||||
```bash
|
||||
cuda-gdb "$(which python3)" \
|
||||
-ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"
|
||||
```
|
||||
|
||||
For a replay-first crash example, read
|
||||
[references/case-studies.md](references/case-studies.md).
|
||||
|
||||
#### OTel trace
|
||||
|
||||
Use tracing when:
|
||||
|
||||
- request-stage timing is unclear
|
||||
- router vs. worker attribution is unclear
|
||||
- PD prefill/decode transfer may be implicated
|
||||
|
||||
If tracing was enabled at startup, you can change the level without restart:
|
||||
|
||||
```bash
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=1"
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=2"
|
||||
```
|
||||
|
||||
#### Torch profile
|
||||
|
||||
Use profiling when:
|
||||
|
||||
- the issue is already narrowed to compute-side ownership
|
||||
- replay already reproduces the problem
|
||||
- metrics and loads do not explain the regression
|
||||
|
||||
At that point, switch to `llm-torch-profiler-analysis`. Do not duplicate
|
||||
its profiling workflow here.
|
||||
|
||||
For a low-noise latency example, read
|
||||
[references/case-studies.md](references/case-studies.md).
|
||||
|
||||
#### Distributed hang
|
||||
|
||||
If this looks like a collective stall, save the failing request, replay it on a
|
||||
clean target, collect the replay-time bundle and stacks, then switch to
|
||||
`debug-distributed-hang`.
|
||||
|
||||
For an example of that flow, read
|
||||
[references/case-studies.md](references/case-studies.md).
|
||||
|
||||
#### Regression between two commits
|
||||
|
||||
If one commit is known-good and another is known-bad, build a deterministic
|
||||
harness before doing deeper manual debugging:
|
||||
|
||||
1. choose a stable reproducer: request replay, benchmark command, or correctness check
|
||||
2. make the harness return `0` on good behavior and non-zero on bad behavior
|
||||
3. run `git bisect start <bad> <good>`
|
||||
4. run `git bisect run <harness>`
|
||||
5. return here only after a candidate commit is isolated
|
||||
|
||||
Prefer replay-backed bisect when the regression depends on request shape or
|
||||
long-running serving state.
|
||||
|
||||
### 6. Switch tools when the boundary is clear
|
||||
|
||||
Switch tools once the fault class is clear:
|
||||
|
||||
- `llm-torch-profiler-analysis` for kernel and overlap attribution
|
||||
- `debug-distributed-hang` for collective or rank-divergence hangs
|
||||
- `debug-cuda-crash` for CUDA crash reproduction and kernel API logging
|
||||
|
||||
Do not switch tools before collecting the first bundle unless the user already has
|
||||
decisive logs or dumps.
|
||||
|
||||
## References
|
||||
|
||||
Load only what the current step needs:
|
||||
|
||||
- [references/decision-tree.md](references/decision-tree.md)
|
||||
- problem classes, tool switch points, return shape
|
||||
- [references/endpoints-and-signals.md](references/endpoints-and-signals.md)
|
||||
- endpoint behavior, auth notes, field reading
|
||||
- [references/replay-trace-profile.md](references/replay-trace-profile.md)
|
||||
- request dump, crash dump, replay, trace, profiler step, bisect
|
||||
- [references/case-studies.md](references/case-studies.md)
|
||||
- compact examples for replay-first CUDA crash, latency, and distributed-hang triage
|
||||
|
||||
## Scripts
|
||||
|
||||
- [scripts/incident_artifact_tool.py](scripts/incident_artifact_tool.py)
|
||||
- collect a read-only live bundle
|
||||
- summarize a collected bundle into a compact debug note
|
||||
- summarize a trusted request dump or crash dump before replay
|
||||
- [scripts/replay_trusted_request_dump.py](scripts/replay_trusted_request_dump.py)
|
||||
- replay a trusted request dump when `safe_pickle_load` blocks stock replay
|
||||
|
||||
If a live bundle was collected, include its path.
|
||||
|
||||
If replay, trace, or profiling was chosen, say why bundle plus dump were not enough.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Case Studies
|
||||
|
||||
Use these examples only after the live bundle and request dump point toward the
|
||||
same class of failure. They are patterns for how to reason from replayable
|
||||
evidence, not recipes to copy blindly.
|
||||
|
||||
## CUDA Crash: Upstream Top-K Corruption, Downstream MoE OOB
|
||||
|
||||
Use when a replayed CUDA crash lands in a MoE align or shared-memory kernel but
|
||||
the suspicious data was produced by an earlier routing kernel.
|
||||
|
||||
Shape that made the original case useful:
|
||||
|
||||
- model family: Qwen3 MoE
|
||||
- visible crash: `moe_align_block_size_kernel`
|
||||
- likely producer: `topkGatingSoftmax` / MoE top-k routing
|
||||
- evidence path: crash dump -> replay -> CUDA coredump -> walk one kernel
|
||||
upstream from the visible fault
|
||||
|
||||
Triage loop:
|
||||
|
||||
```text
|
||||
summarize crash dump
|
||||
-> replay the exact request
|
||||
-> enable CUDA coredump on the replay target
|
||||
-> identify the failing kernel
|
||||
-> inspect the immediately preceding producer kernel and tensors
|
||||
```
|
||||
|
||||
Key lesson: a consumer kernel can be the first one to fault even when the bad
|
||||
index was produced earlier. Preserve the request shape before changing prompts.
|
||||
|
||||
## Latency: TTFT Spike With Low Queue Time
|
||||
|
||||
Use when `/health` and `/health_generate` are green, queue depth is low, but TTFT
|
||||
is still high.
|
||||
|
||||
Signals from the original case:
|
||||
|
||||
- `waiting=0`
|
||||
- average queue time was tiny
|
||||
- TTFT was high
|
||||
- scheduler stage timing pointed to prefill forward time
|
||||
|
||||
Triage loop:
|
||||
|
||||
```text
|
||||
collect live bundle
|
||||
-> save the slow request
|
||||
-> replay the same request on a clean target
|
||||
-> profile only after replay reproduces compute-side ownership
|
||||
```
|
||||
|
||||
Key lesson: rule out queue pressure with `/v1/loads`, `/metrics`, and stage
|
||||
timing before opening a profiler trace.
|
||||
|
||||
## Distributed Hang: Request-Shaped TP Collective Mismatch
|
||||
|
||||
Use when one request hangs, ranks stop making progress differently, and the
|
||||
failure looks like a generic serving stall until replay isolates it.
|
||||
|
||||
Shape that made the original case useful:
|
||||
|
||||
- a prompt tokenized to a specific extend length
|
||||
- one TP rank skipped a logits `all_gather`
|
||||
- the peer rank still entered the real collective
|
||||
- the request never returned
|
||||
|
||||
Triage loop:
|
||||
|
||||
```text
|
||||
collect healthy bundle
|
||||
-> save the trigger request
|
||||
-> replay on a clean target
|
||||
-> collect rank stacks and replay-time bundle
|
||||
-> switch to debug-distributed-hang
|
||||
```
|
||||
|
||||
Key lesson: once the symptom looks like rank divergence or a collective mismatch,
|
||||
do not keep profiling kernels. Preserve the replay and move to distributed-hang
|
||||
debugging.
|
||||
@@ -0,0 +1,197 @@
|
||||
# SGLang First Checks
|
||||
|
||||
Use this reference when the problem class is still unclear and you need a fast
|
||||
starting point.
|
||||
|
||||
## Default Order
|
||||
|
||||
1. classify the symptom
|
||||
2. collect the fastest useful signal
|
||||
3. save the failing request or dump
|
||||
4. replay before you profile
|
||||
|
||||
Do not start with `torch.profiler` unless the issue is already clearly
|
||||
compute-side.
|
||||
|
||||
If one commit is known-good and another is known-bad, turn the problem into a
|
||||
stable `git bisect run <harness>` first.
|
||||
|
||||
## Problem Classes
|
||||
|
||||
### Server down or unhealthy
|
||||
|
||||
Check:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/server_info`
|
||||
- recent stderr/stdout
|
||||
- crash dump status if `--crash-dump-folder` is enabled
|
||||
|
||||
Likely directions:
|
||||
|
||||
- startup or weight-load failure
|
||||
- deadlock or blocked scheduler
|
||||
- CUDA crash or OOM
|
||||
- auth or routing mismatch
|
||||
|
||||
### High latency or low throughput
|
||||
|
||||
Check:
|
||||
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
- `/server_info`
|
||||
- the exact request shape or benchmark command
|
||||
|
||||
Likely directions:
|
||||
|
||||
- queueing or capacity pressure
|
||||
- cache hit rate collapse
|
||||
- PD or EP topology mismatch
|
||||
- speculative decoding disabled or ineffective
|
||||
- kernel or backend regression
|
||||
|
||||
### Wrong output or behavior regression
|
||||
|
||||
Check:
|
||||
|
||||
- exact request and expected output
|
||||
- `/model_info`
|
||||
- `/server_info`
|
||||
- current weights or recent config change
|
||||
|
||||
Likely directions:
|
||||
|
||||
- wrong weights or wrong revision
|
||||
- chat template, parser, or tool config drift
|
||||
- multimodal preprocessing drift
|
||||
- quantization or kernel correctness bug
|
||||
|
||||
### Timeout or hang
|
||||
|
||||
Check:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/v1/loads?include=all`
|
||||
- request dumps if enabled
|
||||
- per-rank logs
|
||||
- OTel trace if already enabled
|
||||
|
||||
Likely directions:
|
||||
|
||||
- distributed divergence or collective hang
|
||||
- queue starvation or retraction storm
|
||||
- PD transfer stall
|
||||
- storage or HiCache backend stall
|
||||
|
||||
## Quick Paths
|
||||
|
||||
### TTFT spike
|
||||
|
||||
Start with:
|
||||
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
- `/server_info`
|
||||
|
||||
Watch for:
|
||||
|
||||
- `num_waiting_reqs` growth
|
||||
- `token_usage` saturation
|
||||
- `cache_hit_rate` drop
|
||||
- PD queue buildup
|
||||
|
||||
If queue pressure does not explain the slowdown, save the slow request and
|
||||
replay it.
|
||||
|
||||
### Throughput collapse
|
||||
|
||||
Start with:
|
||||
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
- benchmark reproduction if available
|
||||
|
||||
Watch for:
|
||||
|
||||
- low `gen_throughput`
|
||||
- queue growth
|
||||
- low cache hit rate
|
||||
- speculative metrics collapse
|
||||
- PD transfer or decode prealloc queues backing up
|
||||
|
||||
### Crash after some requests
|
||||
|
||||
Start with:
|
||||
|
||||
- crash dump folder
|
||||
- stderr/stdout
|
||||
- request dump folder if available
|
||||
|
||||
Then replay the crash dump or recent request dump.
|
||||
|
||||
### Regression between two commits
|
||||
|
||||
Start with:
|
||||
|
||||
- known-good commit
|
||||
- known-bad commit
|
||||
- one stable pass/fail harness
|
||||
|
||||
Best move:
|
||||
|
||||
- `git bisect run <harness>`
|
||||
|
||||
### One request class fails
|
||||
|
||||
Start with:
|
||||
|
||||
- exact request payload
|
||||
- request dump if available
|
||||
- smallest reproduction request
|
||||
|
||||
Typical categories:
|
||||
|
||||
- multimodal edge case
|
||||
- parser or structured output bug
|
||||
- model-specific kernel path
|
||||
- tool-call formatting issue
|
||||
|
||||
## When To Switch Tools
|
||||
|
||||
### Use replay when
|
||||
|
||||
- a crash dump or request dump already exists
|
||||
- the issue depends on request shape or workload mix
|
||||
- you need one stable reproducer before going deeper
|
||||
|
||||
### Use OTel trace when
|
||||
|
||||
- request-stage timing is unclear
|
||||
- router vs. worker ownership is unclear
|
||||
- PD boundaries may be involved
|
||||
|
||||
### Use torch profiler when
|
||||
|
||||
- replay already reproduces the issue
|
||||
- queueing and routing are mostly ruled out
|
||||
- you need kernel-level attribution
|
||||
|
||||
At that point, switch to `llm-torch-profiler-analysis`.
|
||||
|
||||
### Use lower-level debug paths when
|
||||
|
||||
- replay plus trace still leave ambiguity
|
||||
- the problem looks like a specific crash, hang, or correctness bug
|
||||
|
||||
## What To Return
|
||||
|
||||
- problem class
|
||||
- what was checked
|
||||
- strongest signal so far
|
||||
- current best guess
|
||||
- what was ruled out
|
||||
- next step
|
||||
- production risk
|
||||
@@ -0,0 +1,218 @@
|
||||
# SGLang Endpoints and Signals
|
||||
|
||||
Use this reference when checking a live server.
|
||||
|
||||
## Auth
|
||||
|
||||
Most read endpoints are public unless the server is protected by `api_key` or
|
||||
`admin_api_key`.
|
||||
|
||||
Use:
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer <token>" ...
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- normal protected endpoints require `api_key`
|
||||
- admin endpoints require `admin_api_key`
|
||||
- some HiCache endpoints fail if `admin_api_key` is not configured at all
|
||||
- `/health` and metrics-style health checks are usually still exposed
|
||||
|
||||
## Core Endpoints
|
||||
|
||||
### `/health`
|
||||
|
||||
Cheap liveness check.
|
||||
|
||||
- `200`: process is alive enough to answer health
|
||||
- `503`: starting, shutting down, or unhealthy
|
||||
|
||||
`/health` alone is not enough for latency or hang diagnosis.
|
||||
|
||||
### `/health_generate`
|
||||
|
||||
Active health check.
|
||||
|
||||
- exercises a real generate or embedding path
|
||||
- catches stuck schedulers or broken worker paths that `/health` can miss
|
||||
|
||||
Use this when requests time out but `/health` is still green.
|
||||
|
||||
### `/model_info`
|
||||
|
||||
Use for model identity:
|
||||
|
||||
- `model_path`
|
||||
- `tokenizer_path`
|
||||
- `is_generation`
|
||||
- `weight_version`
|
||||
- multimodal flags
|
||||
- model type or architectures
|
||||
|
||||
This is the first check for wrong-output or wrong-weight problems.
|
||||
|
||||
### `/server_info`
|
||||
|
||||
Use for runtime shape:
|
||||
|
||||
- serialized `server_args`
|
||||
- scheduler info
|
||||
- per-DP `internal_states`
|
||||
- SGLang version
|
||||
|
||||
This is usually the single best live snapshot.
|
||||
|
||||
## Load And Capacity
|
||||
|
||||
### `/v1/loads?include=all`
|
||||
|
||||
Best structured load endpoint for a first pass.
|
||||
|
||||
Useful fields:
|
||||
|
||||
- `num_running_reqs`
|
||||
- `num_waiting_reqs`
|
||||
- `num_total_tokens`
|
||||
- `num_used_tokens`
|
||||
- `token_usage`
|
||||
- `gen_throughput`
|
||||
- `cache_hit_rate`
|
||||
- `memory`
|
||||
- `speculative`
|
||||
- `disaggregation`
|
||||
- `queues`
|
||||
|
||||
Useful queries:
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:30000/v1/loads
|
||||
curl -s "http://127.0.0.1:30000/v1/loads?include=all"
|
||||
curl -s "http://127.0.0.1:30000/v1/loads?include=core,queues,disagg"
|
||||
curl -s "http://127.0.0.1:30000/v1/loads?format=prometheus"
|
||||
```
|
||||
|
||||
What to look for:
|
||||
|
||||
- high `num_waiting_reqs` with low compute throughput usually means queueing or capacity pressure
|
||||
- `token_usage` near `1.0` usually means KV or token-capacity pressure
|
||||
- low `cache_hit_rate` after a deploy can explain TTFT regressions
|
||||
- PD queue fields often explain transfer or prealloc bottlenecks hidden by plain queue size
|
||||
|
||||
### `/metrics`
|
||||
|
||||
Prometheus endpoint. Use it when you need trends rather than one live snapshot.
|
||||
|
||||
High-value metrics:
|
||||
|
||||
- `sglang:time_to_first_token_seconds`
|
||||
- `sglang:time_per_output_token_seconds`
|
||||
- `sglang:e2e_request_latency_seconds`
|
||||
- `sglang:num_running_reqs`
|
||||
- `sglang:num_queue_reqs`
|
||||
- `sglang:num_used_tokens`
|
||||
- `sglang:cache_hit_rate`
|
||||
- `sglang:gen_throughput`
|
||||
- `sglang:token_usage`
|
||||
|
||||
## Request Capture
|
||||
|
||||
### `/configure_logging`
|
||||
|
||||
Used by `python -m sglang.srt.managers.configure_logging`.
|
||||
|
||||
Main use:
|
||||
|
||||
- enable request logging
|
||||
- set request logging level
|
||||
- enable request dump folder
|
||||
- set request dump threshold
|
||||
|
||||
Typical payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"log_requests": true,
|
||||
"log_requests_level": 3,
|
||||
"dump_requests_folder": "/tmp/sglang_request_dump",
|
||||
"dump_requests_threshold": 100
|
||||
}
|
||||
```
|
||||
|
||||
Use this when the problem is ongoing and you need the next failing request
|
||||
without restarting the service.
|
||||
|
||||
## HiCache
|
||||
|
||||
### `GET /hicache/storage-backend`
|
||||
|
||||
Returns tokenizer-side HiCache storage status:
|
||||
|
||||
- `hicache_storage_backend`
|
||||
- `hicache_storage_backend_extra_config`
|
||||
- `hicache_storage_prefetch_policy`
|
||||
- `hicache_write_policy`
|
||||
|
||||
Use this when long-context or PD problems may involve storage-backed KV reuse.
|
||||
|
||||
### `PUT /hicache/storage-backend`
|
||||
### `DELETE /hicache/storage-backend`
|
||||
|
||||
Runtime attach or detach. These are operational actions, not passive checks.
|
||||
|
||||
## Profiling And Tracing Controls
|
||||
|
||||
### `/start_profile`
|
||||
### `/stop_profile`
|
||||
|
||||
Use only after the problem is already narrowed down.
|
||||
|
||||
### `/set_trace_level?level=N`
|
||||
|
||||
Changes trace verbosity when tracing was enabled at startup.
|
||||
|
||||
Levels:
|
||||
|
||||
- `0`: disabled
|
||||
- `1`: important slices
|
||||
- `2`: all slices except nested ones
|
||||
- `3`: all slices
|
||||
|
||||
## Quick Reads By Problem Type
|
||||
|
||||
### TTFT spike
|
||||
|
||||
Read:
|
||||
|
||||
- `/server_info`
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
|
||||
Compare:
|
||||
|
||||
- queue size
|
||||
- token usage
|
||||
- cache hit rate
|
||||
- PD disaggregation queues
|
||||
|
||||
### Hang or timeout
|
||||
|
||||
Read:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/server_info`
|
||||
- `/v1/loads?include=all`
|
||||
|
||||
If tracing is already enabled, look at trace data before heavier profiling.
|
||||
|
||||
### Wrong model behavior
|
||||
|
||||
Read:
|
||||
|
||||
- `/model_info`
|
||||
- `/server_info`
|
||||
- exact request payload and parser or template config
|
||||
|
||||
Do not jump to kernel profiling until config drift is ruled out.
|
||||
@@ -0,0 +1,236 @@
|
||||
# Replay, Trace, Profile, and Bisect
|
||||
|
||||
Use this reference after the first live checks. The goal is to turn the problem
|
||||
into something repeatable.
|
||||
|
||||
## Save Requests
|
||||
|
||||
### Request dump
|
||||
|
||||
```bash
|
||||
python3 -m sglang.srt.managers.configure_logging \
|
||||
--url http://127.0.0.1:30000 \
|
||||
--dump-requests-folder /tmp/sglang_request_dump \
|
||||
--dump-requests-threshold 100
|
||||
```
|
||||
|
||||
Use this when:
|
||||
|
||||
- the problem is intermittent
|
||||
- you need the real request shape
|
||||
- you do not want to restart the server
|
||||
|
||||
### Crash dump
|
||||
|
||||
If the server already runs with:
|
||||
|
||||
```bash
|
||||
--crash-dump-folder /tmp/crash_dump
|
||||
```
|
||||
|
||||
SGLang saves recent requests before a crash. Treat that dump as the best
|
||||
starting point.
|
||||
|
||||
Summarize it first:
|
||||
|
||||
```bash
|
||||
python3 scripts/incident_artifact_tool.py summarize-dump \
|
||||
--input-file /path/to/crash_dump.pkl
|
||||
```
|
||||
|
||||
Current crash-dump tests show at least:
|
||||
|
||||
- `server_args`
|
||||
- `requests`
|
||||
- `launch_command`
|
||||
|
||||
## Replay
|
||||
|
||||
Use the stock replay tool:
|
||||
|
||||
```bash
|
||||
python3 scripts/playground/replay_request_dump.py \
|
||||
--input-file /path/to/crash_dump.pkl \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--parallel 128
|
||||
```
|
||||
|
||||
Or replay a folder:
|
||||
|
||||
```bash
|
||||
python3 scripts/playground/replay_request_dump.py \
|
||||
--input-folder /path/to/request_dump_dir \
|
||||
--file-number 10 \
|
||||
--parallel 128
|
||||
```
|
||||
|
||||
If `safe_pickle_load` blocks a locally captured trusted dump, use:
|
||||
|
||||
```bash
|
||||
python3 scripts/replay_trusted_request_dump.py \
|
||||
--input-file /path/to/request_dump.pkl \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--parallel 1
|
||||
```
|
||||
|
||||
If that happens, the allowlist is the problem, not the dump.
|
||||
|
||||
Use replay before profiling when:
|
||||
|
||||
- the issue depends on workload mix
|
||||
- it only appears after some number of requests
|
||||
- you need to compare two builds on the same traffic
|
||||
|
||||
## CUDA Restart-And-Replay
|
||||
|
||||
If replay points to a CUDA crash path, restart the same build with coredumps:
|
||||
|
||||
```bash
|
||||
SGLANG_CUDA_COREDUMP=1 \
|
||||
SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
|
||||
python -m sglang.launch_server \
|
||||
--model-path ... \
|
||||
--crash-dump-folder /tmp/sglang_crash_dump \
|
||||
...
|
||||
```
|
||||
|
||||
Then inspect the coredump:
|
||||
|
||||
```bash
|
||||
cuda-gdb "$(which python3)" \
|
||||
-ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"
|
||||
```
|
||||
|
||||
Good first commands:
|
||||
|
||||
- `where`
|
||||
- `info cuda kernels`
|
||||
- `x/10i <pc>`
|
||||
|
||||
Use the coredump to find the failing kernel, not automatically the root-cause
|
||||
kernel.
|
||||
|
||||
See:
|
||||
|
||||
- [case-studies.md](case-studies.md)
|
||||
|
||||
## Trace
|
||||
|
||||
Tracing must be enabled at startup:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--enable-trace \
|
||||
--otlp-traces-endpoint localhost:4317 \
|
||||
...
|
||||
```
|
||||
|
||||
Optional router command:
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_router \
|
||||
--enable-trace \
|
||||
--otlp-traces-endpoint localhost:4317 \
|
||||
...
|
||||
```
|
||||
|
||||
Useful environment variables:
|
||||
|
||||
```bash
|
||||
export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500
|
||||
export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64
|
||||
```
|
||||
|
||||
If tracing is already enabled, change the level without restart:
|
||||
|
||||
```bash
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=1"
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=2"
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=3"
|
||||
```
|
||||
|
||||
Use tracing for:
|
||||
|
||||
- router vs. worker delay
|
||||
- tokenizer / scheduler / detokenizer timing
|
||||
- PD transfer timing
|
||||
- request timing across processes
|
||||
|
||||
If you already have OTEL JSON or JSONL, convert it for timeline inspection:
|
||||
|
||||
```bash
|
||||
python3 scripts/convert_otel_2_perfetto.py \
|
||||
--input /tmp/otel_trace.json \
|
||||
--output /tmp/sglang_trace_perfetto.json
|
||||
```
|
||||
|
||||
## Torch Profiler
|
||||
|
||||
Switch to `llm-torch-profiler-analysis` when:
|
||||
|
||||
- replay already reproduces the issue
|
||||
- metrics and loads do not explain it
|
||||
- the problem now looks compute-side
|
||||
|
||||
This skill should decide when to profile, not duplicate the profiler workflow.
|
||||
|
||||
## Bisect
|
||||
|
||||
If one commit is known-good and a newer commit is known-bad:
|
||||
|
||||
1. build a deterministic harness from the problem
|
||||
2. prefer replay-based harnesses when the failure depends on request mix
|
||||
3. use `git bisect run <harness>`
|
||||
4. only then go back to trace or profile if needed
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
git bisect start <bad> <good>
|
||||
git bisect run bash ./repro_or_check.sh
|
||||
```
|
||||
|
||||
## Common Paths
|
||||
|
||||
### Crash
|
||||
|
||||
1. crash dump
|
||||
2. summarize dump
|
||||
3. replay
|
||||
4. CUDA coredump plus `cuda-gdb`
|
||||
5. `debug-cuda-crash` or narrower instrumentation
|
||||
|
||||
### TTFT regression
|
||||
|
||||
1. baseline metrics and loads
|
||||
2. request dump
|
||||
3. replay the slow request
|
||||
4. trace if stage ownership is unclear
|
||||
5. `llm-torch-profiler-analysis` if it still looks compute-side
|
||||
|
||||
See:
|
||||
|
||||
- [case-studies.md](case-studies.md)
|
||||
|
||||
### Distributed hang
|
||||
|
||||
1. healthy baseline bundle
|
||||
2. save the trigger request
|
||||
3. replay on a clean target
|
||||
4. collect replay-time bundle and stacks
|
||||
5. identify the NCCL or collective path
|
||||
6. switch to `debug-distributed-hang`
|
||||
|
||||
See:
|
||||
|
||||
- [case-studies.md](case-studies.md)
|
||||
|
||||
### Throughput regression after deploy
|
||||
|
||||
1. compare `server_info`
|
||||
2. compare `/metrics` and `/v1/loads`
|
||||
3. replay stable workload
|
||||
4. bisect if one older commit is known-good
|
||||
5. profile only if compute still looks suspicious
|
||||
+735
@@ -0,0 +1,735 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Collect or inspect serving bundles and dumps for SGLang debug."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import pickle
|
||||
import re
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Sequence
|
||||
from urllib import error, parse, request
|
||||
|
||||
METRIC_RE = re.compile(
|
||||
r"^(?P<name>[^{\s]+)(?:\{(?P<labels>[^}]*)\})?\s+(?P<value>[-+]?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)$"
|
||||
)
|
||||
LABEL_RE = re.compile(r'([a-zA-Z_:][a-zA-Z0-9_:]*)="((?:[^"\\]|\\.)*)"')
|
||||
ENDPOINT_SPECS = (
|
||||
("text", "health.txt", "/health"),
|
||||
("text", "health_generate.txt", "/health_generate"),
|
||||
("text", "metrics.txt", "/metrics"),
|
||||
("json", "model_info.json", "/model_info"),
|
||||
("json", "server_info.json", "/server_info"),
|
||||
("json", "loads_all.json", "/v1/loads?include=all"),
|
||||
(
|
||||
"json",
|
||||
"loads_core_queues_disagg.json",
|
||||
"/v1/loads?include=core,queues,disagg,spec",
|
||||
),
|
||||
("json", "hicache_storage_backend.json", "/hicache/storage-backend"),
|
||||
)
|
||||
BUNDLE_NOTES = [
|
||||
"This bundle is read-only. It does not start profiling or change trace level.",
|
||||
"HiCache status may fail if admin_api_key is not configured or the wrong bearer token was used.",
|
||||
"loads_all.json is the best point-in-time load snapshot in this bundle.",
|
||||
"metrics.txt is raw Prometheus text intended for follow-up parsing.",
|
||||
]
|
||||
|
||||
|
||||
def request_text(
|
||||
base_url: str,
|
||||
path: str,
|
||||
token: Optional[str],
|
||||
timeout: float = 10.0,
|
||||
) -> tuple[bool, int, str]:
|
||||
url = parse.urljoin(base_url.rstrip("/") + "/", path.lstrip("/"))
|
||||
req = request.Request(url)
|
||||
if token:
|
||||
req.add_header("Authorization", f"Bearer {token}")
|
||||
try:
|
||||
with request.urlopen(req, timeout=timeout) as resp:
|
||||
body = resp.read().decode("utf-8", errors="replace")
|
||||
return True, resp.status, body
|
||||
except error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
return False, e.code, body
|
||||
except Exception as e: # noqa: BLE001
|
||||
return False, -1, f"{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
def request_endpoint(
|
||||
base_url: str,
|
||||
path: str,
|
||||
token: Optional[str],
|
||||
parse_json: bool,
|
||||
timeout: float = 10.0,
|
||||
) -> Dict[str, Any]:
|
||||
ok, status, body = request_text(base_url, path, token, timeout=timeout)
|
||||
result: Dict[str, Any] = {"ok": ok, "status": status, "path": path}
|
||||
if not ok:
|
||||
result["error"] = body
|
||||
return result
|
||||
if not parse_json:
|
||||
result["text"] = body
|
||||
return result
|
||||
try:
|
||||
result["json"] = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
result["text"] = body
|
||||
result["decode_error"] = "response was not valid JSON"
|
||||
return result
|
||||
|
||||
|
||||
def write_json(path: Path, obj: Dict[str, Any]) -> None:
|
||||
path.write_text(
|
||||
json.dumps(obj, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def write_text(path: Path, text: str) -> None:
|
||||
path.write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
def format_summary_line(filename: str, result: Dict[str, Any]) -> str:
|
||||
if result.get("ok"):
|
||||
return f"{filename}: ok"
|
||||
return (
|
||||
f"{filename}: failed status={result.get('status')} "
|
||||
f"error={result.get('error')}"
|
||||
)
|
||||
|
||||
|
||||
def collect_bundle(
|
||||
base_url: str,
|
||||
token: Optional[str],
|
||||
outdir: Optional[str],
|
||||
timeout: float,
|
||||
) -> Path:
|
||||
timestamp = time.strftime("%Y%m%d_%H%M%S")
|
||||
bundle_dir = Path(outdir or f"./incident_bundle_{timestamp}").resolve()
|
||||
bundle_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
metadata = {
|
||||
"artifact_type": "incident_bundle",
|
||||
"base_url": base_url,
|
||||
"collected_at": timestamp,
|
||||
"token_provided": bool(token),
|
||||
"timeout_seconds": timeout,
|
||||
}
|
||||
write_json(bundle_dir / "metadata.json", metadata)
|
||||
|
||||
summary_lines = []
|
||||
for kind, filename, path in ENDPOINT_SPECS:
|
||||
result = request_endpoint(
|
||||
base_url, path, token, parse_json=(kind == "json"), timeout=timeout
|
||||
)
|
||||
output_path = bundle_dir / filename
|
||||
if kind == "text" and result.get("ok"):
|
||||
write_text(output_path, str(result.get("text", "")))
|
||||
else:
|
||||
write_json(
|
||||
(
|
||||
output_path
|
||||
if kind == "json"
|
||||
else bundle_dir / f"{filename}.error.json"
|
||||
),
|
||||
result,
|
||||
)
|
||||
summary_lines.append(format_summary_line(filename, result))
|
||||
|
||||
write_text(
|
||||
bundle_dir / "SUMMARY.txt",
|
||||
"\n".join(summary_lines + [""] + BUNDLE_NOTES) + "\n",
|
||||
)
|
||||
return bundle_dir
|
||||
|
||||
|
||||
def load_json(path: Path) -> Optional[Dict[str, Any]]:
|
||||
if not path.exists():
|
||||
return None
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def unwrap_result(path: Path) -> Optional[Dict[str, Any]]:
|
||||
obj = load_json(path)
|
||||
if obj is None:
|
||||
return None
|
||||
if isinstance(obj, dict) and "json" in obj:
|
||||
return obj.get("json")
|
||||
return obj
|
||||
|
||||
|
||||
def read_text(path: Path) -> Optional[str]:
|
||||
if not path.exists():
|
||||
return None
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def endpoint_ok(bundle_dir: Path, stem: str) -> bool:
|
||||
return (bundle_dir / f"{stem}.txt").exists() and not (
|
||||
bundle_dir / f"{stem}.txt.error.json"
|
||||
).exists()
|
||||
|
||||
|
||||
def parse_labels(raw: Optional[str]) -> Dict[str, str]:
|
||||
if not raw:
|
||||
return {}
|
||||
labels = {}
|
||||
for key, value in LABEL_RE.findall(raw):
|
||||
labels[key] = bytes(value, "utf-8").decode("unicode_escape")
|
||||
return labels
|
||||
|
||||
|
||||
def parse_metrics(metrics_text: str) -> Dict[str, list[dict[str, Any]]]:
|
||||
series: Dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for line in metrics_text.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
match = METRIC_RE.match(line)
|
||||
if not match:
|
||||
continue
|
||||
series[match.group("name")].append(
|
||||
{
|
||||
"labels": parse_labels(match.group("labels")),
|
||||
"value": float(match.group("value")),
|
||||
}
|
||||
)
|
||||
return series
|
||||
|
||||
|
||||
def metric_sum(metrics: Dict[str, list[dict[str, Any]]], name: str) -> float:
|
||||
return sum(item["value"] for item in metrics.get(name, []))
|
||||
|
||||
|
||||
def safe_div(
|
||||
numerator: Optional[float], denominator: Optional[float]
|
||||
) -> Optional[float]:
|
||||
if numerator is None or denominator in (None, 0):
|
||||
return None
|
||||
return numerator / denominator
|
||||
|
||||
|
||||
def coalesce(*values: Any) -> Any:
|
||||
for value in values:
|
||||
if value is not None:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def fmt_float(value: Optional[float], digits: int = 3) -> str:
|
||||
if value is None or (
|
||||
isinstance(value, float) and (math.isnan(value) or math.isinf(value))
|
||||
):
|
||||
return "n/a"
|
||||
return f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def is_positive_number(value: Any, threshold: float = 0.0) -> bool:
|
||||
return (
|
||||
isinstance(value, (int, float))
|
||||
and not math.isnan(value)
|
||||
and not math.isinf(value)
|
||||
and value > threshold
|
||||
)
|
||||
|
||||
|
||||
def compute_stage_averages(
|
||||
metrics: Dict[str, list[dict[str, Any]]], sum_name: str, count_name: str
|
||||
) -> Dict[str, float]:
|
||||
grouped_sum: Dict[str, float] = defaultdict(float)
|
||||
grouped_count: Dict[str, float] = defaultdict(float)
|
||||
for item in metrics.get(sum_name, []):
|
||||
stage = item["labels"].get("stage", "")
|
||||
rank = item["labels"].get("tp_rank", "")
|
||||
grouped_sum[f"{stage}|{rank}"] += item["value"]
|
||||
for item in metrics.get(count_name, []):
|
||||
stage = item["labels"].get("stage", "")
|
||||
rank = item["labels"].get("tp_rank", "")
|
||||
grouped_count[f"{stage}|{rank}"] += item["value"]
|
||||
|
||||
result: Dict[str, float] = {}
|
||||
for key, total_sum in grouped_sum.items():
|
||||
stage, _rank = key.split("|", 1)
|
||||
avg = safe_div(total_sum, grouped_count.get(key))
|
||||
if avg is None:
|
||||
continue
|
||||
result[stage] = max(result.get(stage, 0.0), avg)
|
||||
return result
|
||||
|
||||
|
||||
def add_signal(signals: list[str], text: str) -> None:
|
||||
if text not in signals:
|
||||
signals.append(text)
|
||||
|
||||
|
||||
def build_bundle_summary(bundle_dir: Path) -> Dict[str, Any]:
|
||||
metadata = load_json(bundle_dir / "metadata.json") or {}
|
||||
model_info = unwrap_result(bundle_dir / "model_info.json") or {}
|
||||
server_info = unwrap_result(bundle_dir / "server_info.json") or {}
|
||||
loads_info = unwrap_result(bundle_dir / "loads_all.json") or {}
|
||||
metrics_text = read_text(bundle_dir / "metrics.txt") or ""
|
||||
metrics = parse_metrics(metrics_text)
|
||||
|
||||
aggregate = loads_info.get("aggregate") or {}
|
||||
loads = loads_info.get("loads") or []
|
||||
load0 = loads[0] if loads else {}
|
||||
internal_states = server_info.get("internal_states") or []
|
||||
runtime_state = internal_states[0] if internal_states else {}
|
||||
memory_usage = runtime_state.get("memory_usage") or load0.get("memory") or {}
|
||||
|
||||
ttft_avg = safe_div(
|
||||
metric_sum(metrics, "sglang:time_to_first_token_seconds_sum"),
|
||||
metric_sum(metrics, "sglang:time_to_first_token_seconds_count"),
|
||||
)
|
||||
e2e_avg = safe_div(
|
||||
metric_sum(metrics, "sglang:e2e_request_latency_seconds_sum"),
|
||||
metric_sum(metrics, "sglang:e2e_request_latency_seconds_count"),
|
||||
)
|
||||
queue_avg = safe_div(
|
||||
metric_sum(metrics, "sglang:queue_time_seconds_sum"),
|
||||
metric_sum(metrics, "sglang:queue_time_seconds_count"),
|
||||
)
|
||||
per_stage_avg = compute_stage_averages(
|
||||
metrics,
|
||||
"sglang:per_stage_req_latency_seconds_sum",
|
||||
"sglang:per_stage_req_latency_seconds_count",
|
||||
)
|
||||
|
||||
summary: Dict[str, Any] = {
|
||||
"artifact_type": "incident_bundle",
|
||||
"bundle_dir": str(bundle_dir),
|
||||
"base_url": metadata.get("base_url"),
|
||||
"collected_at": metadata.get("collected_at"),
|
||||
"health": {
|
||||
"health_ok": endpoint_ok(bundle_dir, "health"),
|
||||
"health_generate_ok": endpoint_ok(bundle_dir, "health_generate"),
|
||||
},
|
||||
"model": {
|
||||
"model_path": model_info.get("model_path") or server_info.get("model_path"),
|
||||
"served_model_name": server_info.get("served_model_name"),
|
||||
"weight_version": model_info.get("weight_version")
|
||||
or server_info.get("weight_version"),
|
||||
"model_type": model_info.get("model_type"),
|
||||
"is_generation": model_info.get("is_generation"),
|
||||
},
|
||||
"topology": {
|
||||
"tp_size": server_info.get("tp_size"),
|
||||
"dp_size": server_info.get("dp_size"),
|
||||
"pp_size": server_info.get("pp_size"),
|
||||
"ep_size": server_info.get("ep_size"),
|
||||
"disaggregation_mode": server_info.get("disaggregation_mode"),
|
||||
"attention_backend": server_info.get("attention_backend"),
|
||||
"sampling_backend": server_info.get("sampling_backend"),
|
||||
"schedule_policy": server_info.get("schedule_policy"),
|
||||
"enable_trace": server_info.get("enable_trace"),
|
||||
"enable_metrics": server_info.get("enable_metrics"),
|
||||
},
|
||||
"capacity": {
|
||||
"max_total_num_tokens": server_info.get("max_total_num_tokens"),
|
||||
"max_req_input_len": server_info.get("max_req_input_len"),
|
||||
"effective_max_running_requests_per_dp": coalesce(
|
||||
runtime_state.get("effective_max_running_requests_per_dp"),
|
||||
load0.get("max_running_requests"),
|
||||
),
|
||||
"weight_gb": coalesce(
|
||||
memory_usage.get("weight"), memory_usage.get("weight_gb")
|
||||
),
|
||||
"kv_cache_gb": coalesce(
|
||||
memory_usage.get("kvcache"), memory_usage.get("kv_cache_gb")
|
||||
),
|
||||
"graph_gb": coalesce(
|
||||
memory_usage.get("graph"), memory_usage.get("graph_gb")
|
||||
),
|
||||
"token_capacity": memory_usage.get("token_capacity"),
|
||||
},
|
||||
"point_in_time_load": {
|
||||
"running_reqs": coalesce(
|
||||
aggregate.get("total_running_reqs"), load0.get("num_running_reqs")
|
||||
),
|
||||
"waiting_reqs": coalesce(
|
||||
aggregate.get("total_waiting_reqs"), load0.get("num_waiting_reqs")
|
||||
),
|
||||
"total_reqs": coalesce(
|
||||
aggregate.get("total_reqs"), load0.get("num_total_reqs")
|
||||
),
|
||||
"token_usage": coalesce(
|
||||
aggregate.get("avg_token_usage"), load0.get("token_usage")
|
||||
),
|
||||
"avg_throughput": coalesce(
|
||||
aggregate.get("avg_throughput"), load0.get("gen_throughput")
|
||||
),
|
||||
"avg_utilization": coalesce(
|
||||
aggregate.get("avg_utilization"), load0.get("utilization")
|
||||
),
|
||||
"cache_hit_rate": load0.get("cache_hit_rate"),
|
||||
"queues": load0.get("queues"),
|
||||
"disaggregation": load0.get("disaggregation"),
|
||||
},
|
||||
"metrics": {
|
||||
"request_count": metric_sum(metrics, "sglang:num_requests_total"),
|
||||
"prompt_tokens_total": metric_sum(metrics, "sglang:prompt_tokens_total"),
|
||||
"generation_tokens_total": metric_sum(
|
||||
metrics, "sglang:generation_tokens_total"
|
||||
),
|
||||
"avg_ttft_seconds": ttft_avg,
|
||||
"avg_e2e_seconds": e2e_avg,
|
||||
"avg_queue_time_seconds": queue_avg,
|
||||
"stage_avg_seconds_max_tp_rank": per_stage_avg,
|
||||
},
|
||||
"signals": [],
|
||||
}
|
||||
|
||||
signals = summary["signals"]
|
||||
health = summary["health"]
|
||||
point_in_time_load = summary["point_in_time_load"]
|
||||
running_reqs = point_in_time_load.get("running_reqs")
|
||||
waiting_reqs = point_in_time_load.get("waiting_reqs")
|
||||
|
||||
if health["health_ok"] and not health["health_generate_ok"]:
|
||||
add_signal(
|
||||
signals,
|
||||
"/health is green but /health_generate failed. Suspect runtime or scheduler path, not just HTTP liveness.",
|
||||
)
|
||||
if not health["health_ok"]:
|
||||
add_signal(
|
||||
signals,
|
||||
"/health failed. Start with startup, crash, or global unhealthy paths.",
|
||||
)
|
||||
if is_positive_number(waiting_reqs):
|
||||
add_signal(
|
||||
signals,
|
||||
f"Point-in-time load shows queue buildup: waiting_reqs={waiting_reqs}.",
|
||||
)
|
||||
if (
|
||||
point_in_time_load.get("token_usage") is not None
|
||||
and point_in_time_load["token_usage"] >= 0.9
|
||||
):
|
||||
add_signal(
|
||||
signals,
|
||||
"Token usage is near saturation. KV or token-capacity pressure may explain latency.",
|
||||
)
|
||||
if (
|
||||
ttft_avg is not None
|
||||
and queue_avg is not None
|
||||
and ttft_avg > 2.0
|
||||
and queue_avg < 0.2
|
||||
):
|
||||
add_signal(
|
||||
signals,
|
||||
f"Average TTFT is high ({fmt_float(ttft_avg)}s) while average queue time is low ({fmt_float(queue_avg)}s). This looks more like prefill or request-path work than queue pressure.",
|
||||
)
|
||||
prefill_forward = per_stage_avg.get("prefill_forward")
|
||||
request_process = per_stage_avg.get("request_process")
|
||||
if (
|
||||
prefill_forward is not None
|
||||
and request_process is not None
|
||||
and prefill_forward > max(0.5, request_process * 10)
|
||||
):
|
||||
add_signal(
|
||||
signals,
|
||||
f"Prefill forward dominates quick stage timing: prefill_forward~{fmt_float(prefill_forward)}s vs request_process~{fmt_float(request_process)}s.",
|
||||
)
|
||||
if running_reqs == 0 and waiting_reqs == 0:
|
||||
add_signal(
|
||||
signals,
|
||||
"Bundle snapshot was captured while the server was effectively idle. Reproduce under live traffic or replayed workload if the problem is intermittent.",
|
||||
)
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
def render_bundle_text(summary: Dict[str, Any]) -> str:
|
||||
health = summary["health"]
|
||||
model = summary["model"]
|
||||
topology = summary["topology"]
|
||||
capacity = summary["capacity"]
|
||||
load = summary["point_in_time_load"]
|
||||
metrics = summary["metrics"]
|
||||
stage_avgs = metrics["stage_avg_seconds_max_tp_rank"]
|
||||
|
||||
lines = [
|
||||
f"Bundle: {summary['bundle_dir']}",
|
||||
f"Base URL: {summary.get('base_url') or 'n/a'}",
|
||||
f"Collected At: {summary.get('collected_at') or 'n/a'}",
|
||||
"",
|
||||
f"Health: /health={'ok' if health['health_ok'] else 'failed'} /health_generate={'ok' if health['health_generate_ok'] else 'failed'}",
|
||||
f"Model: {model.get('model_path') or 'n/a'} weight_version={model.get('weight_version') or 'n/a'} type={model.get('model_type') or 'n/a'}",
|
||||
"Topology: "
|
||||
f"tp={topology.get('tp_size')} dp={topology.get('dp_size')} pp={topology.get('pp_size')} ep={topology.get('ep_size')} "
|
||||
f"disagg={topology.get('disaggregation_mode')} trace={topology.get('enable_trace')} metrics={topology.get('enable_metrics')}",
|
||||
"Capacity: "
|
||||
f"max_total_tokens={capacity.get('max_total_num_tokens')} "
|
||||
f"max_running_reqs={capacity.get('effective_max_running_requests_per_dp')} "
|
||||
f"weight_gb={fmt_float(capacity.get('weight_gb'))} "
|
||||
f"kv_cache_gb={fmt_float(capacity.get('kv_cache_gb'))} "
|
||||
f"graph_gb={fmt_float(capacity.get('graph_gb'))}",
|
||||
"Point-in-time load: "
|
||||
f"running={load.get('running_reqs')} waiting={load.get('waiting_reqs')} total={load.get('total_reqs')} "
|
||||
f"token_usage={fmt_float(load.get('token_usage'))} throughput={fmt_float(load.get('avg_throughput'))} "
|
||||
f"cache_hit_rate={fmt_float(load.get('cache_hit_rate'))}",
|
||||
"Metrics: "
|
||||
f"requests={fmt_float(metrics.get('request_count'), 0)} "
|
||||
f"prompt_tokens={fmt_float(metrics.get('prompt_tokens_total'), 0)} "
|
||||
f"generation_tokens={fmt_float(metrics.get('generation_tokens_total'), 0)} "
|
||||
f"avg_ttft_s={fmt_float(metrics.get('avg_ttft_seconds'))} "
|
||||
f"avg_e2e_s={fmt_float(metrics.get('avg_e2e_seconds'))} "
|
||||
f"avg_queue_s={fmt_float(metrics.get('avg_queue_time_seconds'))}",
|
||||
]
|
||||
|
||||
if stage_avgs:
|
||||
stage_parts = [
|
||||
f"{name}={fmt_float(value)}s" for name, value in sorted(stage_avgs.items())
|
||||
]
|
||||
lines.append("Stage Averages (max across TP ranks): " + ", ".join(stage_parts))
|
||||
|
||||
queues = load.get("queues") or {}
|
||||
if queues:
|
||||
lines.append(
|
||||
"Queues: "
|
||||
+ ", ".join(f"{key}={value}" for key, value in sorted(queues.items()))
|
||||
)
|
||||
|
||||
disagg = load.get("disaggregation") or {}
|
||||
if disagg:
|
||||
lines.append(
|
||||
"Disaggregation: "
|
||||
+ ", ".join(f"{key}={value}" for key, value in sorted(disagg.items()))
|
||||
)
|
||||
|
||||
lines.append("")
|
||||
lines.append("What stands out:")
|
||||
if summary["signals"]:
|
||||
lines.extend(f"- {signal}" for signal in summary["signals"])
|
||||
else:
|
||||
lines.append("- No strong signal from this bundle.")
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def get_field(obj: Any, name: str, default: Any = None) -> Any:
|
||||
if obj is None:
|
||||
return default
|
||||
if isinstance(obj, dict):
|
||||
return obj.get(name, default)
|
||||
return getattr(obj, name, default)
|
||||
|
||||
|
||||
def iter_dump_files(
|
||||
input_file: Optional[str], input_folder: Optional[str]
|
||||
) -> Sequence[Path]:
|
||||
if input_file:
|
||||
return [Path(input_file)]
|
||||
if input_folder:
|
||||
return [Path(p) for p in sorted(glob.glob(f"{input_folder}/*.pkl"))]
|
||||
raise SystemExit("Either --input-file or --input-folder must be provided.")
|
||||
|
||||
|
||||
def load_dump_payload(path: Path) -> dict[str, Any]:
|
||||
with path.open("rb") as fh:
|
||||
payload = pickle.load(fh)
|
||||
if isinstance(payload, dict):
|
||||
return payload
|
||||
return {"requests": payload}
|
||||
|
||||
|
||||
def pick_text_preview(req: Any) -> str:
|
||||
candidates = [
|
||||
get_field(req, "origin_input_text"),
|
||||
get_field(req, "text"),
|
||||
get_field(req, "prompt"),
|
||||
]
|
||||
for value in candidates:
|
||||
if isinstance(value, str) and value:
|
||||
return value
|
||||
if isinstance(value, list) and value:
|
||||
first = value[0]
|
||||
if isinstance(first, str) and first:
|
||||
return first
|
||||
return ""
|
||||
|
||||
|
||||
def format_timestamp(ts: Any) -> str:
|
||||
if not isinstance(ts, (int, float)):
|
||||
return "n/a"
|
||||
return datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
def summarize_request(
|
||||
record: tuple[Any, dict[str, Any], Any, Any], idx: int, preview_chars: int
|
||||
) -> list[str]:
|
||||
req, output, start_time, end_time = record
|
||||
preview = pick_text_preview(req).replace("\n", " ").strip()
|
||||
if len(preview) > preview_chars:
|
||||
preview = preview[: preview_chars - 3] + "..."
|
||||
|
||||
output_dict = output if isinstance(output, dict) else {}
|
||||
meta_info = get_field(output_dict, "meta_info", {}) or {}
|
||||
rid = get_field(req, "rid") or get_field(meta_info, "id")
|
||||
stream = bool(get_field(req, "stream", False))
|
||||
prompt_tokens = get_field(meta_info, "prompt_tokens")
|
||||
completion_tokens = get_field(meta_info, "completion_tokens")
|
||||
duration = (
|
||||
end_time - start_time
|
||||
if isinstance(start_time, (int, float)) and isinstance(end_time, (int, float))
|
||||
else None
|
||||
)
|
||||
|
||||
elapsed_str = f"{duration:.3f}" if duration is not None else "n/a"
|
||||
lines = [
|
||||
f"[{idx}] rid={rid or 'n/a'} stream={stream} "
|
||||
f"prompt_tokens={prompt_tokens if prompt_tokens is not None else 'n/a'} "
|
||||
f"completion_tokens={completion_tokens if completion_tokens is not None else 'n/a'} "
|
||||
f"start={format_timestamp(start_time)} elapsed_s={elapsed_str}"
|
||||
]
|
||||
if preview:
|
||||
lines.append(f" text={preview}")
|
||||
return lines
|
||||
|
||||
|
||||
def summarize_dump_file(path: Path, max_requests: int, preview_chars: int) -> str:
|
||||
payload = load_dump_payload(path)
|
||||
requests = payload.get("requests") or []
|
||||
server_args = payload.get("server_args")
|
||||
launch_command = payload.get("launch_command")
|
||||
|
||||
model_path = get_field(server_args, "model_path")
|
||||
tp_size = get_field(server_args, "tp_size")
|
||||
dp_size = get_field(server_args, "dp_size")
|
||||
pp_size = get_field(server_args, "pp_size")
|
||||
host = get_field(server_args, "host")
|
||||
port = get_field(server_args, "port")
|
||||
|
||||
timestamps = [
|
||||
record[2]
|
||||
for record in requests
|
||||
if isinstance(record, tuple)
|
||||
and len(record) >= 4
|
||||
and isinstance(record[2], (int, float))
|
||||
]
|
||||
time_span = (
|
||||
max(timestamps) - min(timestamps)
|
||||
if len(timestamps) >= 2
|
||||
else 0.0 if len(timestamps) == 1 else None
|
||||
)
|
||||
|
||||
lines = [
|
||||
f"File: {path}",
|
||||
"Dump Type: request_or_crash_dump",
|
||||
f"Requests: {len(requests)}",
|
||||
f"Model: {model_path or 'n/a'}",
|
||||
f"Topology: tp={tp_size if tp_size is not None else 'n/a'} "
|
||||
f"dp={dp_size if dp_size is not None else 'n/a'} "
|
||||
f"pp={pp_size if pp_size is not None else 'n/a'}",
|
||||
f"Endpoint: {host or 'n/a'}:{port if port is not None else 'n/a'}",
|
||||
(
|
||||
f"Time span seconds: {time_span:.3f}"
|
||||
if time_span is not None
|
||||
else "Time span seconds: n/a"
|
||||
),
|
||||
]
|
||||
if launch_command:
|
||||
lines.append(f"Launch command: {launch_command}")
|
||||
|
||||
for idx, record in enumerate(requests[:max_requests]):
|
||||
if not isinstance(record, tuple) or len(record) < 4:
|
||||
lines.append(f"[{idx}] Unsupported record shape: {type(record)!r}")
|
||||
continue
|
||||
lines.extend(summarize_request(record, idx, preview_chars))
|
||||
|
||||
if len(requests) > max_requests:
|
||||
lines.append(f"... truncated {len(requests) - max_requests} more requests")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Collect or inspect serving bundles and dumps for SGLang debug."
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
collect_parser = subparsers.add_parser(
|
||||
"collect-bundle", help="Collect a read-only live bundle from a running server"
|
||||
)
|
||||
collect_parser.add_argument("--base-url", required=True)
|
||||
collect_parser.add_argument(
|
||||
"--token",
|
||||
default=os.environ.get("SGLANG_BEARER_TOKEN"),
|
||||
help="Bearer token for protected endpoints. Defaults to $SGLANG_BEARER_TOKEN.",
|
||||
)
|
||||
collect_parser.add_argument("--outdir", default=None)
|
||||
collect_parser.add_argument("--timeout", type=float, default=10.0)
|
||||
|
||||
bundle_parser = subparsers.add_parser(
|
||||
"summarize-bundle", help="Summarize a bundle directory"
|
||||
)
|
||||
bundle_parser.add_argument("bundle_dir")
|
||||
bundle_parser.add_argument("--out", default=None)
|
||||
bundle_parser.add_argument("--json-out", default=None)
|
||||
bundle_parser.add_argument("--stdout-json", action="store_true")
|
||||
|
||||
dump_parser = subparsers.add_parser(
|
||||
"summarize-dump", help="Summarize a trusted request dump or crash dump"
|
||||
)
|
||||
dump_parser.add_argument("--input-file", default=None)
|
||||
dump_parser.add_argument("--input-folder", default=None)
|
||||
dump_parser.add_argument("--max-requests", type=int, default=20)
|
||||
dump_parser.add_argument("--preview-chars", type=int, default=160)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "collect-bundle":
|
||||
bundle_dir = collect_bundle(
|
||||
args.base_url, args.token, args.outdir, args.timeout
|
||||
)
|
||||
print(bundle_dir)
|
||||
return 0
|
||||
|
||||
if args.command == "summarize-bundle":
|
||||
bundle_dir = Path(args.bundle_dir).resolve()
|
||||
if not bundle_dir.is_dir():
|
||||
raise SystemExit(
|
||||
f"bundle_dir does not exist or is not a directory: {bundle_dir}"
|
||||
)
|
||||
summary = build_bundle_summary(bundle_dir)
|
||||
out_text = render_bundle_text(summary)
|
||||
text_path = Path(args.out) if args.out else bundle_dir / "SUMMARY_REPORT.txt"
|
||||
json_path = (
|
||||
Path(args.json_out) if args.json_out else bundle_dir / "SUMMARY_REPORT.json"
|
||||
)
|
||||
text_path.write_text(out_text, encoding="utf-8")
|
||||
json_path.write_text(
|
||||
json.dumps(summary, indent=2, ensure_ascii=False) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
if args.stdout_json:
|
||||
print(json.dumps(summary, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(out_text, end="")
|
||||
return 0
|
||||
|
||||
files = iter_dump_files(args.input_file, args.input_folder)
|
||||
if not files:
|
||||
raise SystemExit("No .pkl files matched the provided input.")
|
||||
for idx, path in enumerate(files):
|
||||
if idx:
|
||||
print()
|
||||
print(
|
||||
summarize_dump_file(
|
||||
path=path,
|
||||
max_requests=args.max_requests,
|
||||
preview_chars=args.preview_chars,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+219
@@ -0,0 +1,219 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Replay a trusted SGLang request dump directly over HTTP.
|
||||
|
||||
Use this only for locally captured or otherwise trusted dump files.
|
||||
It uses plain pickle loading to bypass SafeUnpickler restrictions that may block
|
||||
the stock replay helper on newer SGLang builds.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import pickle
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import asdict, is_dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Sequence
|
||||
|
||||
import requests
|
||||
|
||||
Record = tuple[object, dict[str, Any], float, float]
|
||||
|
||||
|
||||
def normalize_mm_data_item(item: Any) -> Any:
|
||||
if isinstance(item, dict) and "url" in item:
|
||||
return item["url"]
|
||||
return item
|
||||
|
||||
|
||||
def normalize_mm_data(data: Any) -> Any:
|
||||
if data is None:
|
||||
return None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
(
|
||||
[normalize_mm_data_item(item) for item in sublist]
|
||||
if isinstance(sublist, list)
|
||||
else normalize_mm_data_item(sublist)
|
||||
)
|
||||
for sublist in data
|
||||
]
|
||||
return normalize_mm_data_item(data)
|
||||
|
||||
|
||||
def normalize_request_data(json_data: dict[str, Any]) -> dict[str, Any]:
|
||||
for field in ["image_data", "video_data", "audio_data"]:
|
||||
if field in json_data and json_data[field] is not None:
|
||||
json_data[field] = normalize_mm_data(json_data[field])
|
||||
return json_data
|
||||
|
||||
|
||||
def to_plain_dict(obj: Any) -> dict[str, Any]:
|
||||
if obj is None:
|
||||
return {}
|
||||
if isinstance(obj, dict):
|
||||
return dict(obj)
|
||||
if is_dataclass(obj):
|
||||
return asdict(obj)
|
||||
|
||||
model_dump = getattr(obj, "model_dump", None)
|
||||
if callable(model_dump):
|
||||
dumped = model_dump()
|
||||
if isinstance(dumped, dict):
|
||||
return dumped
|
||||
|
||||
dict_method = getattr(obj, "dict", None)
|
||||
if callable(dict_method):
|
||||
dumped = dict_method()
|
||||
if isinstance(dumped, dict):
|
||||
return dumped
|
||||
|
||||
obj_dict = getattr(obj, "__dict__", None)
|
||||
if isinstance(obj_dict, dict):
|
||||
return {
|
||||
key: value for key, value in obj_dict.items() if not key.startswith("_")
|
||||
}
|
||||
|
||||
raise TypeError(f"Unsupported request object type: {type(obj)!r}")
|
||||
|
||||
|
||||
def request_to_json_data(req: Any) -> dict[str, Any]:
|
||||
json_data = normalize_request_data(to_plain_dict(req))
|
||||
sampling_params = json_data.get("sampling_params")
|
||||
if sampling_params is not None and not isinstance(sampling_params, dict):
|
||||
json_data["sampling_params"] = to_plain_dict(sampling_params)
|
||||
return json_data
|
||||
|
||||
|
||||
def load_records(path: Path) -> list[Record]:
|
||||
with path.open("rb") as fh:
|
||||
payload = pickle.load(fh)
|
||||
if isinstance(payload, dict) and "requests" in payload:
|
||||
return payload["requests"]
|
||||
return payload
|
||||
|
||||
|
||||
def iter_files(args: argparse.Namespace) -> Sequence[Path]:
|
||||
if args.input_file:
|
||||
return [Path(args.input_file)]
|
||||
if args.input_folder:
|
||||
return [
|
||||
Path(p)
|
||||
for p in sorted(glob.glob(f"{args.input_folder}/*.pkl"))[: args.file_number]
|
||||
]
|
||||
raise SystemExit("Either --input-file or --input-folder must be provided.")
|
||||
|
||||
|
||||
def run_one_request(
|
||||
record: Record,
|
||||
args: argparse.Namespace,
|
||||
replay_init_time: float,
|
||||
base_time: float,
|
||||
idx: int,
|
||||
) -> None:
|
||||
req, output, start_time, end_time = record
|
||||
relative_start = start_time - base_time
|
||||
delay = max(0.0, (relative_start - (time.time() - replay_init_time)) / args.speed)
|
||||
if delay:
|
||||
time.sleep(delay)
|
||||
|
||||
json_data = request_to_json_data(req)
|
||||
if args.ignore_eos:
|
||||
json_data.setdefault("sampling_params", {})["ignore_eos"] = True
|
||||
completion_tokens = output.get("meta_info", {}).get("completion_tokens")
|
||||
if completion_tokens:
|
||||
json_data["sampling_params"]["max_new_tokens"] = completion_tokens
|
||||
|
||||
t0 = time.time()
|
||||
response = requests.post(
|
||||
f"http://{args.host}:{args.port}/generate",
|
||||
json=json_data,
|
||||
timeout=args.timeout,
|
||||
stream=bool(json_data.get("stream")),
|
||||
)
|
||||
elapsed = time.time() - t0
|
||||
|
||||
if json_data.get("stream"):
|
||||
last = None
|
||||
for chunk in response.iter_lines(decode_unicode=False):
|
||||
decoded = chunk.decode("utf-8")
|
||||
if decoded and decoded.startswith("data:"):
|
||||
if decoded == "data: [DONE]":
|
||||
break
|
||||
last = json.loads(decoded[5:].strip())
|
||||
result = last or {}
|
||||
else:
|
||||
result = response.json()
|
||||
|
||||
meta = result.get("meta_info", {})
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"idx": idx,
|
||||
"status_code": response.status_code,
|
||||
"elapsed_seconds": round(elapsed, 3),
|
||||
"prompt_tokens": meta.get("prompt_tokens"),
|
||||
"completion_tokens": meta.get("completion_tokens"),
|
||||
"rid": meta.get("id"),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Replay a trusted SGLang request dump or crash dump directly over HTTP."
|
||||
)
|
||||
parser.add_argument("--host", default="127.0.0.1")
|
||||
parser.add_argument("--port", type=int, default=30000)
|
||||
parser.add_argument("--input-folder", default=None)
|
||||
parser.add_argument("--input-file", default=None)
|
||||
parser.add_argument("--file-number", type=int, default=1)
|
||||
parser.add_argument("--req-number", type=int, default=1_000_000)
|
||||
parser.add_argument("--req-start", type=int, default=0)
|
||||
parser.add_argument("--parallel", type=int, default=1)
|
||||
parser.add_argument("--ignore-eos", action="store_true")
|
||||
parser.add_argument("--speed", type=float, default=1.0)
|
||||
parser.add_argument("--timeout", type=float, default=120.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
files = iter_files(args)
|
||||
print(f"Replay files: {[str(p) for p in files]}")
|
||||
|
||||
records: list[Record] = []
|
||||
for path in files:
|
||||
records.extend(load_records(path))
|
||||
|
||||
if not records:
|
||||
print("No requests found.")
|
||||
return 0
|
||||
|
||||
records.sort(key=lambda x: x[-2])
|
||||
records = records[args.req_start : args.req_start + args.req_number]
|
||||
print(f"Replay requests: {len(records)}")
|
||||
base_time = records[0][-2]
|
||||
print(
|
||||
"Base time: " + datetime.fromtimestamp(base_time).strftime("%Y-%m-%d %H:%M:%S")
|
||||
)
|
||||
|
||||
replay_init_time = time.time()
|
||||
with ThreadPoolExecutor(max_workers=args.parallel) as executor:
|
||||
futures = []
|
||||
for idx, record in enumerate(records):
|
||||
futures.append(
|
||||
executor.submit(
|
||||
run_one_request, record, args, replay_init_time, base_time, idx
|
||||
)
|
||||
)
|
||||
for future in futures:
|
||||
future.result()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user