diff --git a/docs/advanced_features/hisparse_guide.md b/docs/advanced_features/hisparse_guide.md index 941318c86..675bbb05e 100644 --- a/docs/advanced_features/hisparse_guide.md +++ b/docs/advanced_features/hisparse_guide.md @@ -2,7 +2,7 @@ HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency. -> **Prerequisites**: HiSparse only works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5). These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. +> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1) and **DeepSeek V4**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. ## Why HiSparse? @@ -40,6 +40,8 @@ Prefill GPU ──RDMA──▶ Decode Host Pool (CPU pinned memory) swap-in kernel (on-demand top-k) ``` +For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4_indexer and C128 KV remain device-to-device transfers. + ## Server Arguments | Argument | Type / Default | Description | @@ -89,8 +91,7 @@ python3 -m sglang.launch_server \ --context-length 81920 \ --tp-size 8 --dp-size 8 --enable-dp-attention \ --mem-fraction-static 0.85 \ - --kv-cache-dtype bfloat16 \ - --dsa-decode-backend flashmla_sparse \ + --disable-radix-cache \ --disaggregation-mode decode \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --dist-init-addr 127.0.0.1:5757 \ @@ -99,6 +100,8 @@ python3 -m sglang.launch_server \ --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}' ``` +> **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`). DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend. + ### Benchmark ```bash @@ -121,14 +124,12 @@ python3 -m sglang.bench_serving \ ### Key Notes - The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse. -- On the decode instance, the following flags are **required** for HiSparse: - - `--kv-cache-dtype bfloat16` — currently only bfloat16 KV cache is supported (more dtypes planned). - - `--dsa-decode-backend flashmla_sparse` — currently only `flashmla_sparse` backend is supported. - - `--enable-hisparse` — enables HiSparse. - - `--hisparse-config` — HiSparse configuration (top_k, device_buffer_size, host_to_device_ratio). - - `host_to_device_ratio` should be configured based on the host machine's available memory. For example: - - **~1 TB** host memory → `host_to_device_ratio: 5` - - **~2 TB** host memory → `host_to_device_ratio: 10` +- On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse. +- For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`. +- For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default. +- `host_to_device_ratio` should be configured based on the host machine's available memory. For example: + - **~1 TB** host memory → `host_to_device_ratio: 5` + - **~2 TB** host memory → `host_to_device_ratio: 10` ## Acknowledgments diff --git a/docs_new/docs/advanced_features/hisparse_guide.mdx b/docs_new/docs/advanced_features/hisparse_guide.mdx index 2c62b0b0a..78b71288a 100644 --- a/docs_new/docs/advanced_features/hisparse_guide.mdx +++ b/docs_new/docs/advanced_features/hisparse_guide.mdx @@ -6,7 +6,7 @@ metatags: HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency. -> **Prerequisites**: HiSparse only works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5). These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. +> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1) and **DeepSeek V4**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. ## Why HiSparse? @@ -44,6 +44,8 @@ Prefill GPU ──RDMA──▶ Decode Host Pool (CPU pinned memory) swap-in kernel (on-demand top-k) ``` +For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4_indexer and C128 KV remain device-to-device transfers. + ## Server Arguments @@ -141,8 +143,7 @@ python3 -m sglang.launch_server \ --context-length 81920 \ --tp-size 8 --dp-size 8 --enable-dp-attention \ --mem-fraction-static 0.85 \ - --kv-cache-dtype bfloat16 \ - --dsa-decode-backend flashmla_sparse \ + --disable-radix-cache \ --disaggregation-mode decode \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --dist-init-addr 127.0.0.1:5757 \ @@ -151,6 +152,8 @@ python3 -m sglang.launch_server \ --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}' ``` +> **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`). DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend. + ### Benchmark ```bash Command @@ -173,14 +176,12 @@ python3 -m sglang.bench_serving \ ### Key Notes - The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse. -- On the decode instance, the following flags are **required** for HiSparse: - - `--kv-cache-dtype bfloat16` — currently only bfloat16 KV cache is supported (more dtypes planned). - - `--dsa-decode-backend flashmla_sparse` — currently only `flashmla_sparse` backend is supported. - - `--enable-hisparse` — enables HiSparse. - - `--hisparse-config` — HiSparse configuration (top_k, device_buffer_size, host_to_device_ratio). - - `host_to_device_ratio` should be configured based on the host machine's available memory. For example: - - **~1 TB** host memory → `host_to_device_ratio: 5` - - **~2 TB** host memory → `host_to_device_ratio: 10` +- On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse. +- For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`. +- For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default. +- `host_to_device_ratio` should be configured based on the host machine's available memory. For example: + - **~1 TB** host memory → `host_to_device_ratio: 5` + - **~2 TB** host memory → `host_to_device_ratio: 10` ## Acknowledgments