minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[AMD] Make breakable CUDA graph run on ROCm/HIP (#28173)

Browse Source
This commit is contained in:
Oguz Ulgen
2026-06-19 07:16:00 -07:00
committed by GitHub
parent 5eaae5bacd
commit 3af991fb3e
8 changed files with 36 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/breakable_cuda_graph.mdx
+3 -3
View File
@@ -139,8 +139,8 @@ Some models fork work onto secondary CUDA streams (e.g., for overlapped computat
## Compatibility
- **NVIDIA CUDA only.** Breakable CUDA graph is not supported on ROCm/HIP or other non-CUDA platforms. On unsupported platforms, `--debug-cuda-graph` is automatically disabled with a warning.
- **Requires `cuda-python`.** The `cuda.bindings` package must be installed (`pip install cuda-python`).
- **CUDA and ROCm/HIP.** Breakable CUDA graph runs on both NVIDIA and AMD GPUs. Other platforms (NPU, CPU, MPS, XPU) are unsupported; there `--debug-cuda-graph` is automatically disabled with a warning.
- **Requires `cuda-python` on NVIDIA.** Stream-capture-status queries use the CUDA runtime via `cuda.bindings` (`pip install cuda-python`); the portable `torch.cuda.is_current_stream_capturing()` has proven unreliable on CUDA. On ROCm/HIP — where `cuda-python` is unavailable — the portable `torch.cuda` API (which maps to the HIP runtime) is used instead.
- **Not compatible with memory saver mode.** Cannot be used together with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`.
## Performance
@@ -174,7 +174,7 @@ For typical use cases with a small number of graph breaks, the overhead is negli
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/cuda_utils.py</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA runtime binding utilities</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA runtime binding utilities (NVIDIA stream-capture queries)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/runner_backend/breakable_cuda_graph_backend.py</code></td>