minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[HiSparse]: Add readme docs for HiSparse Feature (#22238)

Browse Source
This commit is contained in:
Zhangheng
2026-04-07 00:39:24 -07:00
committed by GitHub
parent 3148742ddb
commit 3d3a32c0b9
2 changed files with 117 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/basic_usage/deepseek_v32.md
+6
View File
@@ -468,3 +468,9 @@ python -m sglang.launch_server \
```
For the Decode nodes, it is recommended to use the **EP mode**.
## HiSparse: Hierarchical Sparse Attention for DSA (experimental)
HiSparse reduces per-request GPU memory during decode by keeping only a small "hot" KV buffer on GPU while storing complete KV data in CPU pinned memory. A CUDA kernel dynamically swaps in the top-k most relevant KV entries from host memory on each decode step. This enables significantly higher decode concurrency for long-context DSA models.
HiSparse currently requires PD disaggregation mode and is enabled on the decode instance only. For detailed design, configuration, and deployment instructions, see the [HiSparse Guide](../advanced_features/hisparse_guide.md).