[hisparse]: update user guide (#26249)
Co-authored-by: 晟海 <huangtingwei.htw@antgroup.com>
This commit is contained in:
@@ -6,7 +6,7 @@ metatags:
|
||||
|
||||
HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency.
|
||||
|
||||
> **Prerequisites**: HiSparse only works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5). These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only.
|
||||
> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1) and **DeepSeek V4**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only.
|
||||
|
||||
## Why HiSparse?
|
||||
|
||||
@@ -44,6 +44,8 @@ Prefill GPU ──RDMA──▶ Decode Host Pool (CPU pinned memory)
|
||||
swap-in kernel (on-demand top-k)
|
||||
```
|
||||
|
||||
For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4_indexer and C128 KV remain device-to-device transfers.
|
||||
|
||||
## Server Arguments
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
@@ -141,8 +143,7 @@ python3 -m sglang.launch_server \
|
||||
--context-length 81920 \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--mem-fraction-static 0.85 \
|
||||
--kv-cache-dtype bfloat16 \
|
||||
--dsa-decode-backend flashmla_sparse \
|
||||
--disable-radix-cache \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
|
||||
--dist-init-addr 127.0.0.1:5757 \
|
||||
@@ -151,6 +152,8 @@ python3 -m sglang.launch_server \
|
||||
--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'
|
||||
```
|
||||
|
||||
> **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`). DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend.
|
||||
|
||||
### Benchmark
|
||||
|
||||
```bash Command
|
||||
@@ -173,14 +176,12 @@ python3 -m sglang.bench_serving \
|
||||
### Key Notes
|
||||
|
||||
- The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse.
|
||||
- On the decode instance, the following flags are **required** for HiSparse:
|
||||
- `--kv-cache-dtype bfloat16` — currently only bfloat16 KV cache is supported (more dtypes planned).
|
||||
- `--dsa-decode-backend flashmla_sparse` — currently only `flashmla_sparse` backend is supported.
|
||||
- `--enable-hisparse` — enables HiSparse.
|
||||
- `--hisparse-config` — HiSparse configuration (top_k, device_buffer_size, host_to_device_ratio).
|
||||
- `host_to_device_ratio` should be configured based on the host machine's available memory. For example:
|
||||
- **~1 TB** host memory → `host_to_device_ratio: 5`
|
||||
- **~2 TB** host memory → `host_to_device_ratio: 10`
|
||||
- On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse.
|
||||
- For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`.
|
||||
- For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default.
|
||||
- `host_to_device_ratio` should be configured based on the host machine's available memory. For example:
|
||||
- **~1 TB** host memory → `host_to_device_ratio: 5`
|
||||
- **~2 TB** host memory → `host_to_device_ratio: 10`
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
|
||||
Reference in New Issue
Block a user