[SKILL] Upgrade sglang profile and auto_benchmark skills (#24250)
This commit is contained in:
Xiaoyu Zhang
@@ -0,0 +1,81 @@
|
||||
# Case Studies
|
||||
|
||||
Use these examples only after the live bundle and request dump point toward the
|
||||
same class of failure. They are patterns for how to reason from replayable
|
||||
evidence, not recipes to copy blindly.
|
||||
|
||||
## CUDA Crash: Upstream Top-K Corruption, Downstream MoE OOB
|
||||
|
||||
Use when a replayed CUDA crash lands in a MoE align or shared-memory kernel but
|
||||
the suspicious data was produced by an earlier routing kernel.
|
||||
|
||||
Shape that made the original case useful:
|
||||
|
||||
- model family: Qwen3 MoE
|
||||
- visible crash: `moe_align_block_size_kernel`
|
||||
- likely producer: `topkGatingSoftmax` / MoE top-k routing
|
||||
- evidence path: crash dump -> replay -> CUDA coredump -> walk one kernel
|
||||
upstream from the visible fault
|
||||
|
||||
Triage loop:
|
||||
|
||||
```text
|
||||
summarize crash dump
|
||||
-> replay the exact request
|
||||
-> enable CUDA coredump on the replay target
|
||||
-> identify the failing kernel
|
||||
-> inspect the immediately preceding producer kernel and tensors
|
||||
```
|
||||
|
||||
Key lesson: a consumer kernel can be the first one to fault even when the bad
|
||||
index was produced earlier. Preserve the request shape before changing prompts.
|
||||
|
||||
## Latency: TTFT Spike With Low Queue Time
|
||||
|
||||
Use when `/health` and `/health_generate` are green, queue depth is low, but TTFT
|
||||
is still high.
|
||||
|
||||
Signals from the original case:
|
||||
|
||||
- `waiting=0`
|
||||
- average queue time was tiny
|
||||
- TTFT was high
|
||||
- scheduler stage timing pointed to prefill forward time
|
||||
|
||||
Triage loop:
|
||||
|
||||
```text
|
||||
collect live bundle
|
||||
-> save the slow request
|
||||
-> replay the same request on a clean target
|
||||
-> profile only after replay reproduces compute-side ownership
|
||||
```
|
||||
|
||||
Key lesson: rule out queue pressure with `/v1/loads`, `/metrics`, and stage
|
||||
timing before opening a profiler trace.
|
||||
|
||||
## Distributed Hang: Request-Shaped TP Collective Mismatch
|
||||
|
||||
Use when one request hangs, ranks stop making progress differently, and the
|
||||
failure looks like a generic serving stall until replay isolates it.
|
||||
|
||||
Shape that made the original case useful:
|
||||
|
||||
- a prompt tokenized to a specific extend length
|
||||
- one TP rank skipped a logits `all_gather`
|
||||
- the peer rank still entered the real collective
|
||||
- the request never returned
|
||||
|
||||
Triage loop:
|
||||
|
||||
```text
|
||||
collect healthy bundle
|
||||
-> save the trigger request
|
||||
-> replay on a clean target
|
||||
-> collect rank stacks and replay-time bundle
|
||||
-> switch to debug-distributed-hang
|
||||
```
|
||||
|
||||
Key lesson: once the symptom looks like rank divergence or a collective mismatch,
|
||||
do not keep profiling kernels. Preserve the replay and move to distributed-hang
|
||||
debugging.
|
||||
@@ -0,0 +1,197 @@
|
||||
# SGLang First Checks
|
||||
|
||||
Use this reference when the problem class is still unclear and you need a fast
|
||||
starting point.
|
||||
|
||||
## Default Order
|
||||
|
||||
1. classify the symptom
|
||||
2. collect the fastest useful signal
|
||||
3. save the failing request or dump
|
||||
4. replay before you profile
|
||||
|
||||
Do not start with `torch.profiler` unless the issue is already clearly
|
||||
compute-side.
|
||||
|
||||
If one commit is known-good and another is known-bad, turn the problem into a
|
||||
stable `git bisect run <harness>` first.
|
||||
|
||||
## Problem Classes
|
||||
|
||||
### Server down or unhealthy
|
||||
|
||||
Check:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/server_info`
|
||||
- recent stderr/stdout
|
||||
- crash dump status if `--crash-dump-folder` is enabled
|
||||
|
||||
Likely directions:
|
||||
|
||||
- startup or weight-load failure
|
||||
- deadlock or blocked scheduler
|
||||
- CUDA crash or OOM
|
||||
- auth or routing mismatch
|
||||
|
||||
### High latency or low throughput
|
||||
|
||||
Check:
|
||||
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
- `/server_info`
|
||||
- the exact request shape or benchmark command
|
||||
|
||||
Likely directions:
|
||||
|
||||
- queueing or capacity pressure
|
||||
- cache hit rate collapse
|
||||
- PD or EP topology mismatch
|
||||
- speculative decoding disabled or ineffective
|
||||
- kernel or backend regression
|
||||
|
||||
### Wrong output or behavior regression
|
||||
|
||||
Check:
|
||||
|
||||
- exact request and expected output
|
||||
- `/model_info`
|
||||
- `/server_info`
|
||||
- current weights or recent config change
|
||||
|
||||
Likely directions:
|
||||
|
||||
- wrong weights or wrong revision
|
||||
- chat template, parser, or tool config drift
|
||||
- multimodal preprocessing drift
|
||||
- quantization or kernel correctness bug
|
||||
|
||||
### Timeout or hang
|
||||
|
||||
Check:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/v1/loads?include=all`
|
||||
- request dumps if enabled
|
||||
- per-rank logs
|
||||
- OTel trace if already enabled
|
||||
|
||||
Likely directions:
|
||||
|
||||
- distributed divergence or collective hang
|
||||
- queue starvation or retraction storm
|
||||
- PD transfer stall
|
||||
- storage or HiCache backend stall
|
||||
|
||||
## Quick Paths
|
||||
|
||||
### TTFT spike
|
||||
|
||||
Start with:
|
||||
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
- `/server_info`
|
||||
|
||||
Watch for:
|
||||
|
||||
- `num_waiting_reqs` growth
|
||||
- `token_usage` saturation
|
||||
- `cache_hit_rate` drop
|
||||
- PD queue buildup
|
||||
|
||||
If queue pressure does not explain the slowdown, save the slow request and
|
||||
replay it.
|
||||
|
||||
### Throughput collapse
|
||||
|
||||
Start with:
|
||||
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
- benchmark reproduction if available
|
||||
|
||||
Watch for:
|
||||
|
||||
- low `gen_throughput`
|
||||
- queue growth
|
||||
- low cache hit rate
|
||||
- speculative metrics collapse
|
||||
- PD transfer or decode prealloc queues backing up
|
||||
|
||||
### Crash after some requests
|
||||
|
||||
Start with:
|
||||
|
||||
- crash dump folder
|
||||
- stderr/stdout
|
||||
- request dump folder if available
|
||||
|
||||
Then replay the crash dump or recent request dump.
|
||||
|
||||
### Regression between two commits
|
||||
|
||||
Start with:
|
||||
|
||||
- known-good commit
|
||||
- known-bad commit
|
||||
- one stable pass/fail harness
|
||||
|
||||
Best move:
|
||||
|
||||
- `git bisect run <harness>`
|
||||
|
||||
### One request class fails
|
||||
|
||||
Start with:
|
||||
|
||||
- exact request payload
|
||||
- request dump if available
|
||||
- smallest reproduction request
|
||||
|
||||
Typical categories:
|
||||
|
||||
- multimodal edge case
|
||||
- parser or structured output bug
|
||||
- model-specific kernel path
|
||||
- tool-call formatting issue
|
||||
|
||||
## When To Switch Tools
|
||||
|
||||
### Use replay when
|
||||
|
||||
- a crash dump or request dump already exists
|
||||
- the issue depends on request shape or workload mix
|
||||
- you need one stable reproducer before going deeper
|
||||
|
||||
### Use OTel trace when
|
||||
|
||||
- request-stage timing is unclear
|
||||
- router vs. worker ownership is unclear
|
||||
- PD boundaries may be involved
|
||||
|
||||
### Use torch profiler when
|
||||
|
||||
- replay already reproduces the issue
|
||||
- queueing and routing are mostly ruled out
|
||||
- you need kernel-level attribution
|
||||
|
||||
At that point, switch to `llm-torch-profiler-analysis`.
|
||||
|
||||
### Use lower-level debug paths when
|
||||
|
||||
- replay plus trace still leave ambiguity
|
||||
- the problem looks like a specific crash, hang, or correctness bug
|
||||
|
||||
## What To Return
|
||||
|
||||
- problem class
|
||||
- what was checked
|
||||
- strongest signal so far
|
||||
- current best guess
|
||||
- what was ruled out
|
||||
- next step
|
||||
- production risk
|
||||
@@ -0,0 +1,218 @@
|
||||
# SGLang Endpoints and Signals
|
||||
|
||||
Use this reference when checking a live server.
|
||||
|
||||
## Auth
|
||||
|
||||
Most read endpoints are public unless the server is protected by `api_key` or
|
||||
`admin_api_key`.
|
||||
|
||||
Use:
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer <token>" ...
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- normal protected endpoints require `api_key`
|
||||
- admin endpoints require `admin_api_key`
|
||||
- some HiCache endpoints fail if `admin_api_key` is not configured at all
|
||||
- `/health` and metrics-style health checks are usually still exposed
|
||||
|
||||
## Core Endpoints
|
||||
|
||||
### `/health`
|
||||
|
||||
Cheap liveness check.
|
||||
|
||||
- `200`: process is alive enough to answer health
|
||||
- `503`: starting, shutting down, or unhealthy
|
||||
|
||||
`/health` alone is not enough for latency or hang diagnosis.
|
||||
|
||||
### `/health_generate`
|
||||
|
||||
Active health check.
|
||||
|
||||
- exercises a real generate or embedding path
|
||||
- catches stuck schedulers or broken worker paths that `/health` can miss
|
||||
|
||||
Use this when requests time out but `/health` is still green.
|
||||
|
||||
### `/model_info`
|
||||
|
||||
Use for model identity:
|
||||
|
||||
- `model_path`
|
||||
- `tokenizer_path`
|
||||
- `is_generation`
|
||||
- `weight_version`
|
||||
- multimodal flags
|
||||
- model type or architectures
|
||||
|
||||
This is the first check for wrong-output or wrong-weight problems.
|
||||
|
||||
### `/server_info`
|
||||
|
||||
Use for runtime shape:
|
||||
|
||||
- serialized `server_args`
|
||||
- scheduler info
|
||||
- per-DP `internal_states`
|
||||
- SGLang version
|
||||
|
||||
This is usually the single best live snapshot.
|
||||
|
||||
## Load And Capacity
|
||||
|
||||
### `/v1/loads?include=all`
|
||||
|
||||
Best structured load endpoint for a first pass.
|
||||
|
||||
Useful fields:
|
||||
|
||||
- `num_running_reqs`
|
||||
- `num_waiting_reqs`
|
||||
- `num_total_tokens`
|
||||
- `num_used_tokens`
|
||||
- `token_usage`
|
||||
- `gen_throughput`
|
||||
- `cache_hit_rate`
|
||||
- `memory`
|
||||
- `speculative`
|
||||
- `disaggregation`
|
||||
- `queues`
|
||||
|
||||
Useful queries:
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:30000/v1/loads
|
||||
curl -s "http://127.0.0.1:30000/v1/loads?include=all"
|
||||
curl -s "http://127.0.0.1:30000/v1/loads?include=core,queues,disagg"
|
||||
curl -s "http://127.0.0.1:30000/v1/loads?format=prometheus"
|
||||
```
|
||||
|
||||
What to look for:
|
||||
|
||||
- high `num_waiting_reqs` with low compute throughput usually means queueing or capacity pressure
|
||||
- `token_usage` near `1.0` usually means KV or token-capacity pressure
|
||||
- low `cache_hit_rate` after a deploy can explain TTFT regressions
|
||||
- PD queue fields often explain transfer or prealloc bottlenecks hidden by plain queue size
|
||||
|
||||
### `/metrics`
|
||||
|
||||
Prometheus endpoint. Use it when you need trends rather than one live snapshot.
|
||||
|
||||
High-value metrics:
|
||||
|
||||
- `sglang:time_to_first_token_seconds`
|
||||
- `sglang:time_per_output_token_seconds`
|
||||
- `sglang:e2e_request_latency_seconds`
|
||||
- `sglang:num_running_reqs`
|
||||
- `sglang:num_queue_reqs`
|
||||
- `sglang:num_used_tokens`
|
||||
- `sglang:cache_hit_rate`
|
||||
- `sglang:gen_throughput`
|
||||
- `sglang:token_usage`
|
||||
|
||||
## Request Capture
|
||||
|
||||
### `/configure_logging`
|
||||
|
||||
Used by `python -m sglang.srt.managers.configure_logging`.
|
||||
|
||||
Main use:
|
||||
|
||||
- enable request logging
|
||||
- set request logging level
|
||||
- enable request dump folder
|
||||
- set request dump threshold
|
||||
|
||||
Typical payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"log_requests": true,
|
||||
"log_requests_level": 3,
|
||||
"dump_requests_folder": "/tmp/sglang_request_dump",
|
||||
"dump_requests_threshold": 100
|
||||
}
|
||||
```
|
||||
|
||||
Use this when the problem is ongoing and you need the next failing request
|
||||
without restarting the service.
|
||||
|
||||
## HiCache
|
||||
|
||||
### `GET /hicache/storage-backend`
|
||||
|
||||
Returns tokenizer-side HiCache storage status:
|
||||
|
||||
- `hicache_storage_backend`
|
||||
- `hicache_storage_backend_extra_config`
|
||||
- `hicache_storage_prefetch_policy`
|
||||
- `hicache_write_policy`
|
||||
|
||||
Use this when long-context or PD problems may involve storage-backed KV reuse.
|
||||
|
||||
### `PUT /hicache/storage-backend`
|
||||
### `DELETE /hicache/storage-backend`
|
||||
|
||||
Runtime attach or detach. These are operational actions, not passive checks.
|
||||
|
||||
## Profiling And Tracing Controls
|
||||
|
||||
### `/start_profile`
|
||||
### `/stop_profile`
|
||||
|
||||
Use only after the problem is already narrowed down.
|
||||
|
||||
### `/set_trace_level?level=N`
|
||||
|
||||
Changes trace verbosity when tracing was enabled at startup.
|
||||
|
||||
Levels:
|
||||
|
||||
- `0`: disabled
|
||||
- `1`: important slices
|
||||
- `2`: all slices except nested ones
|
||||
- `3`: all slices
|
||||
|
||||
## Quick Reads By Problem Type
|
||||
|
||||
### TTFT spike
|
||||
|
||||
Read:
|
||||
|
||||
- `/server_info`
|
||||
- `/v1/loads?include=all`
|
||||
- `/metrics`
|
||||
|
||||
Compare:
|
||||
|
||||
- queue size
|
||||
- token usage
|
||||
- cache hit rate
|
||||
- PD disaggregation queues
|
||||
|
||||
### Hang or timeout
|
||||
|
||||
Read:
|
||||
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/server_info`
|
||||
- `/v1/loads?include=all`
|
||||
|
||||
If tracing is already enabled, look at trace data before heavier profiling.
|
||||
|
||||
### Wrong model behavior
|
||||
|
||||
Read:
|
||||
|
||||
- `/model_info`
|
||||
- `/server_info`
|
||||
- exact request payload and parser or template config
|
||||
|
||||
Do not jump to kernel profiling until config drift is ruled out.
|
||||
@@ -0,0 +1,236 @@
|
||||
# Replay, Trace, Profile, and Bisect
|
||||
|
||||
Use this reference after the first live checks. The goal is to turn the problem
|
||||
into something repeatable.
|
||||
|
||||
## Save Requests
|
||||
|
||||
### Request dump
|
||||
|
||||
```bash
|
||||
python3 -m sglang.srt.managers.configure_logging \
|
||||
--url http://127.0.0.1:30000 \
|
||||
--dump-requests-folder /tmp/sglang_request_dump \
|
||||
--dump-requests-threshold 100
|
||||
```
|
||||
|
||||
Use this when:
|
||||
|
||||
- the problem is intermittent
|
||||
- you need the real request shape
|
||||
- you do not want to restart the server
|
||||
|
||||
### Crash dump
|
||||
|
||||
If the server already runs with:
|
||||
|
||||
```bash
|
||||
--crash-dump-folder /tmp/crash_dump
|
||||
```
|
||||
|
||||
SGLang saves recent requests before a crash. Treat that dump as the best
|
||||
starting point.
|
||||
|
||||
Summarize it first:
|
||||
|
||||
```bash
|
||||
python3 scripts/incident_artifact_tool.py summarize-dump \
|
||||
--input-file /path/to/crash_dump.pkl
|
||||
```
|
||||
|
||||
Current crash-dump tests show at least:
|
||||
|
||||
- `server_args`
|
||||
- `requests`
|
||||
- `launch_command`
|
||||
|
||||
## Replay
|
||||
|
||||
Use the stock replay tool:
|
||||
|
||||
```bash
|
||||
python3 scripts/playground/replay_request_dump.py \
|
||||
--input-file /path/to/crash_dump.pkl \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--parallel 128
|
||||
```
|
||||
|
||||
Or replay a folder:
|
||||
|
||||
```bash
|
||||
python3 scripts/playground/replay_request_dump.py \
|
||||
--input-folder /path/to/request_dump_dir \
|
||||
--file-number 10 \
|
||||
--parallel 128
|
||||
```
|
||||
|
||||
If `safe_pickle_load` blocks a locally captured trusted dump, use:
|
||||
|
||||
```bash
|
||||
python3 scripts/replay_trusted_request_dump.py \
|
||||
--input-file /path/to/request_dump.pkl \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--parallel 1
|
||||
```
|
||||
|
||||
If that happens, the allowlist is the problem, not the dump.
|
||||
|
||||
Use replay before profiling when:
|
||||
|
||||
- the issue depends on workload mix
|
||||
- it only appears after some number of requests
|
||||
- you need to compare two builds on the same traffic
|
||||
|
||||
## CUDA Restart-And-Replay
|
||||
|
||||
If replay points to a CUDA crash path, restart the same build with coredumps:
|
||||
|
||||
```bash
|
||||
SGLANG_CUDA_COREDUMP=1 \
|
||||
SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
|
||||
python -m sglang.launch_server \
|
||||
--model-path ... \
|
||||
--crash-dump-folder /tmp/sglang_crash_dump \
|
||||
...
|
||||
```
|
||||
|
||||
Then inspect the coredump:
|
||||
|
||||
```bash
|
||||
cuda-gdb "$(which python3)" \
|
||||
-ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"
|
||||
```
|
||||
|
||||
Good first commands:
|
||||
|
||||
- `where`
|
||||
- `info cuda kernels`
|
||||
- `x/10i <pc>`
|
||||
|
||||
Use the coredump to find the failing kernel, not automatically the root-cause
|
||||
kernel.
|
||||
|
||||
See:
|
||||
|
||||
- [case-studies.md](case-studies.md)
|
||||
|
||||
## Trace
|
||||
|
||||
Tracing must be enabled at startup:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--enable-trace \
|
||||
--otlp-traces-endpoint localhost:4317 \
|
||||
...
|
||||
```
|
||||
|
||||
Optional router command:
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_router \
|
||||
--enable-trace \
|
||||
--otlp-traces-endpoint localhost:4317 \
|
||||
...
|
||||
```
|
||||
|
||||
Useful environment variables:
|
||||
|
||||
```bash
|
||||
export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500
|
||||
export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64
|
||||
```
|
||||
|
||||
If tracing is already enabled, change the level without restart:
|
||||
|
||||
```bash
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=1"
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=2"
|
||||
curl "http://127.0.0.1:30000/set_trace_level?level=3"
|
||||
```
|
||||
|
||||
Use tracing for:
|
||||
|
||||
- router vs. worker delay
|
||||
- tokenizer / scheduler / detokenizer timing
|
||||
- PD transfer timing
|
||||
- request timing across processes
|
||||
|
||||
If you already have OTEL JSON or JSONL, convert it for timeline inspection:
|
||||
|
||||
```bash
|
||||
python3 scripts/convert_otel_2_perfetto.py \
|
||||
--input /tmp/otel_trace.json \
|
||||
--output /tmp/sglang_trace_perfetto.json
|
||||
```
|
||||
|
||||
## Torch Profiler
|
||||
|
||||
Switch to `llm-torch-profiler-analysis` when:
|
||||
|
||||
- replay already reproduces the issue
|
||||
- metrics and loads do not explain it
|
||||
- the problem now looks compute-side
|
||||
|
||||
This skill should decide when to profile, not duplicate the profiler workflow.
|
||||
|
||||
## Bisect
|
||||
|
||||
If one commit is known-good and a newer commit is known-bad:
|
||||
|
||||
1. build a deterministic harness from the problem
|
||||
2. prefer replay-based harnesses when the failure depends on request mix
|
||||
3. use `git bisect run <harness>`
|
||||
4. only then go back to trace or profile if needed
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
git bisect start <bad> <good>
|
||||
git bisect run bash ./repro_or_check.sh
|
||||
```
|
||||
|
||||
## Common Paths
|
||||
|
||||
### Crash
|
||||
|
||||
1. crash dump
|
||||
2. summarize dump
|
||||
3. replay
|
||||
4. CUDA coredump plus `cuda-gdb`
|
||||
5. `debug-cuda-crash` or narrower instrumentation
|
||||
|
||||
### TTFT regression
|
||||
|
||||
1. baseline metrics and loads
|
||||
2. request dump
|
||||
3. replay the slow request
|
||||
4. trace if stage ownership is unclear
|
||||
5. `llm-torch-profiler-analysis` if it still looks compute-side
|
||||
|
||||
See:
|
||||
|
||||
- [case-studies.md](case-studies.md)
|
||||
|
||||
### Distributed hang
|
||||
|
||||
1. healthy baseline bundle
|
||||
2. save the trigger request
|
||||
3. replay on a clean target
|
||||
4. collect replay-time bundle and stacks
|
||||
5. identify the NCCL or collective path
|
||||
6. switch to `debug-distributed-hang`
|
||||
|
||||
See:
|
||||
|
||||
- [case-studies.md](case-studies.md)
|
||||
|
||||
### Throughput regression after deploy
|
||||
|
||||
1. compare `server_info`
|
||||
2. compare `/metrics` and `/v1/loads`
|
||||
3. replay stable workload
|
||||
4. bisect if one older commit is known-good
|
||||
5. profile only if compute still looks suspicious
|
||||
Reference in New Issue
Block a user