Fix inaccuracies and add NPU constraints in ascend_npu_profiling.mdx. (#28283)
This commit is contained in:
@@ -44,12 +44,6 @@ sglang serve \
|
|||||||
--max-running-requests 128
|
--max-running-requests 128
|
||||||
```
|
```
|
||||||
|
|
||||||
<Note>
|
|
||||||
On Ascend NPU, SGLang uses `torch_npu._apply_patches()` to automatically
|
|
||||||
redirect PyTorch Profiler's CUDA activity to NPU, so
|
|
||||||
`activities: ["CPU", "GPU"]` actually captures NPU operator events.
|
|
||||||
</Note>
|
|
||||||
|
|
||||||
**Profiling-related environment variables:**
|
**Profiling-related environment variables:**
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
@@ -79,6 +73,10 @@ redirect PyTorch Profiler's CUDA activity to NPU, so
|
|||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
Priority order: **API parameter > Environment variable > Default value**.
|
||||||
|
</Note>
|
||||||
|
|
||||||
### 2. Collection Methods
|
### 2. Collection Methods
|
||||||
|
|
||||||
SGLang provides four collection methods. The core differences are **whether you
|
SGLang provides four collection methods. The core differences are **whether you
|
||||||
@@ -227,6 +225,12 @@ sending `/stop_profile`.
|
|||||||
shown in the server log as `Profiling done. Traces are saved to: <path>`.
|
shown in the server log as `Profiling done. Traces are saved to: <path>`.
|
||||||
</Note>
|
</Note>
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
On Ascend NPU, SGLang uses `torch_npu._apply_patches()` to automatically
|
||||||
|
redirect PyTorch Profiler's CUDA activity to NPU, so
|
||||||
|
`activities: ["CPU", "GPU"]` actually captures NPU operator events.
|
||||||
|
</Note>
|
||||||
|
|
||||||
**`bench_serving --profile` parameters:**
|
**`bench_serving --profile` parameters:**
|
||||||
|
|
||||||
<table>
|
<table>
|
||||||
@@ -256,7 +260,9 @@ python3 -m sglang.profiler \
|
|||||||
--cpu --gpu &
|
--cpu --gpu &
|
||||||
```
|
```
|
||||||
```bash Command
|
```bash Command
|
||||||
# Terminal 2: Immediately send inference requests to provide steps for profiling
|
# Terminal 2: Wait for "Waiting for N steps" output from Terminal 1, then send requests.
|
||||||
|
# The profiler starts recording once /start_profile is received by the server.
|
||||||
|
# Requests sent before the server receives /start_profile will not be captured.
|
||||||
curl http://127.0.0.1:30000/generate \
|
curl http://127.0.0.1:30000/generate \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}'
|
-d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}'
|
||||||
@@ -352,7 +358,8 @@ set of supported parameters:
|
|||||||
<tr>
|
<tr>
|
||||||
<td><code>activities</code></td>
|
<td><code>activities</code></td>
|
||||||
<td>
|
<td>
|
||||||
Activity types: CPU, GPU, MEM, RPD. On Ascend NPU, primarily CPU and GPU
|
Activity types: CPU, GPU, MEM, RPD. On Ascend NPU, only CPU and GPU are supported.
|
||||||
|
MEM depends on CUDA memory APIs and will be silently ignored. RPD requires ROCm and will cause an error.
|
||||||
</td>
|
</td>
|
||||||
<td><code>["CPU", "GPU"]</code></td>
|
<td><code>["CPU", "GPU"]</code></td>
|
||||||
</tr>
|
</tr>
|
||||||
@@ -426,7 +433,7 @@ Waiting for 10 steps and the trace to be flushed.... (profile_by_stage=False)
|
|||||||
The directory structure is
|
The directory structure is
|
||||||
`<output_dir>/<hostname>_<pid>_<timestamp>_ascend_pt/`. When using Method C
|
`<output_dir>/<hostname>_<pid>_<timestamp>_ascend_pt/`. When using Method C
|
||||||
(`bench_serving --profile`), a timestamp subdirectory is added:
|
(`bench_serving --profile`), a timestamp subdirectory is added:
|
||||||
`<output_dir>/<timestamp>/`. Always check the server log for the exact path:
|
`<output_dir>/<timestamp>/<hostname>_<pid>_<timestamp>_ascend_pt/`. Always check the server log for the exact path:
|
||||||
`Profiling done. Traces are saved to: <path>`.
|
`Profiling done. Traces are saved to: <path>`.
|
||||||
|
|
||||||
### 5. Viewing Results
|
### 5. Viewing Results
|
||||||
@@ -465,12 +472,11 @@ files — **no need to manually call `analyse()`**:
|
|||||||
</Note>
|
</Note>
|
||||||
|
|
||||||
<Note>
|
<Note>
|
||||||
If you need to merge distributed trace files in a multi-node deployment, set
|
In a multi-node deployment, each node generates its own trace files.
|
||||||
`"merge_profiles": true` in the `/start_profile` request. Note: on Ascend NPU,
|
To merge them into a single unified trace, set `"merge_profiles": true` in the `/start_profile` request.
|
||||||
the merger has limited support for the `*_ascend_pt` format — check
|
However, on Ascend NPU, the merge feature does **not** fully support the `*_ascend_pt` format — merged results may be incomplete or incorrect.
|
||||||
`trace_view.json` on each node individually. See
|
It is recommended to view `trace_view.json` on each node individually instead.
|
||||||
[Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling#profiler-trace-merger-for-distributed-traces)
|
For more details, see [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling#profiler-trace-merger-for-distributed-traces).
|
||||||
for details.
|
|
||||||
</Note>
|
</Note>
|
||||||
|
|
||||||
### 6. Re-parsing Raw Data (Optional)
|
### 6. Re-parsing Raw Data (Optional)
|
||||||
@@ -487,6 +493,8 @@ analyse("./sglang_profile/<hostname>_*_ascend_pt/")
|
|||||||
<Note>
|
<Note>
|
||||||
Normally **no need** to manually run `analyse()` — the server already parses
|
Normally **no need** to manually run `analyse()` — the server already parses
|
||||||
data automatically. Only use this for re-parsing or handling interrupted data.
|
data automatically. Only use this for re-parsing or handling interrupted data.
|
||||||
|
Running `analyse()` when ASCEND_PROFILER_OUTPUT already exists will overwrite the original directory.
|
||||||
|
If the original data is still needed, back it up before running `analyse()`.
|
||||||
</Note>
|
</Note>
|
||||||
|
|
||||||
## Best Practices
|
## Best Practices
|
||||||
|
|||||||
Reference in New Issue
Block a user