minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] fix profiling endpoint (#22982)

Browse Source 
Signed-off-by: Zaire404 <3147879462@qq.com>
This commit is contained in:
Zaire
2026-04-16 12:51:39 -04:00
committed by GitHub
parent 082eaed0a4
commit 71377deda7
1 changed files with 12 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/developer_guide/benchmark_and_profiling.md
+12 -9
View File
@@ -54,7 +54,10 @@ python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile
```
The `SGLANG_TORCH_PROFILER_DIR` environment variable must be set on both the server and client side; otherwise, the trace file will not be generated correctly. A secure way to do this is by setting it in your shell's resource file (e.g., `~/.bashrc` for bash).
For `bench_serving --profile`, the output directory is selected on the client side from `--profile-output-dir` or `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`), then sent in the `/start_profile` request.
If you call `/start_profile` directly and do not provide `output_dir`, the server uses its own `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`).
Setting `SGLANG_TORCH_PROFILER_DIR` on both server and client is still recommended to avoid confusion about where traces are written.
For more details, please refer to [Bench Serving Guide](./bench_serving.md).
@@ -155,7 +158,7 @@ curl -X POST http://127.0.0.1:30000/start_profile \
**Parameters:**
- `output_dir` (optional): Directory where profile traces will be saved. If not specified, uses `SGLANG_TORCH_PROFILER_DIR` environment variable, or `/tmp` as the default
- `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/end_profile`
- `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/stop_profile`
- `start_step` (optional): Step number at which to start profiling (inclusive). Useful for skipping warmup iterations
- `activities` (optional): List of activities to profile, e.g., `["CPU", "GPU"]`. Default is `["CPU", "GPU"]`
- `merge_profiles` (optional): Whether to merge distributed traces. Default is `false`
@@ -179,17 +182,17 @@ curl -X POST http://127.0.0.1:30000/start_profile \
**Continuous profiling (manual stop):**
```bash
# Start profiling without num_steps - must manually stop with /end_profile
# Start profiling without num_steps - must manually stop with /stop_profile
curl -X POST http://127.0.0.1:30000/start_profile
```
#### Using `/end_profile` endpoint
#### Using `/stop_profile` endpoint
The `/end_profile` endpoint stops an ongoing profiling session and saves the trace file.
The `/stop_profile` endpoint stops an ongoing profiling session and saves the trace file.
```bash
# Stop profiling and save traces
curl -X POST http://127.0.0.1:30000/end_profile
curl -X POST http://127.0.0.1:30000/stop_profile
```
This is only needed when you start profiling without specifying `num_steps`. If `num_steps` is specified, profiling will automatically stop after that many steps.
@@ -212,7 +215,7 @@ curl -X POST http://127.0.0.1:30000/start_profile \
python -m sglang.bench_serving --backend sglang --num-prompts 100
# Terminal 2: Stop profiling when done
curl -X POST http://127.0.0.1:30000/end_profile
curl -X POST http://127.0.0.1:30000/stop_profile
```
### Profiler Trace Merger for Distributed Traces
@@ -406,10 +409,10 @@ This method allows you to control exactly when profiling starts/stops via HTTP A
```bash
# Terminal 2: Only needed if num_steps was not specified
curl -X POST http://127.0.0.1:30000/end_profile
curl -X POST http://127.0.0.1:30000/stop_profile
```
The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/end_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead.
The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/stop_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead.
**Method 2: Simpler approach without `/start_profile` API**