minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

docs: add DeepSeek-V4 compressed state dtype tip (#28613)

Browse Source 
Co-authored-by: zhujunyu <zhujunyu.666@bytedance.com>
This commit is contained in:
Ryan Zzz
2026-06-17 23:22:22 -07:00
committed by GitHub
co-authored by zhujunyu
parent 7976928c57
commit b55cf4382d
1 changed files with 13 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+13
View File
@@ -180,6 +180,19 @@ The generator currently picks values on the **conservative** side (mirroring an
- `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
- MTP runs on the v2 speculative path.
**Compressed attention state dtype**
DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path:
```bash Command
SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
sglang serve \
--model-path deepseek-ai/DeepSeek-V4-Flash \
<other args>
```
This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior.
**EPLB + DeepEP Waterfill (Experimental)**
For recorded/static EPLB reproduction, first record an expert-distribution file by following