hisparse: support NIXL DRAM KV destinations for HiSparse (#27563)

Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
This commit is contained in:
ishandhanani
2026-06-27 22:32:29 +08:00
committed by GitHub
co-authored by Zhangheng Shangming Cai
parent c1b5c7e499
commit b030b1a5f3
12 changed files with 726 additions and 167 deletions
@@ -108,10 +108,15 @@ Pass as a JSON string via `--hisparse-config`:
<td>int</td>
<td>Ratio of logical pool size to device pool size, determining host memory capacity</td>
</tr>
<tr>
<td><code>swap_in_block_size</code></td>
<td>int / 960</td>
<td>CUDA thread-block size for the HiSparse swap-in kernel</td>
</tr>
</tbody>
</table>
Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'`
Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'`
## Deployment
@@ -149,7 +154,7 @@ python3 -m sglang.launch_server \
--dist-init-addr 127.0.0.1:5757 \
--nnodes 1 --node-rank 0 \
--enable-hisparse \
--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'
--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'
```
> **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`). DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend.