From b55cf4382d0d1cbb31f384d7ae04a9b3d3779d51 Mon Sep 17 00:00:00 2001 From: Ryan Zzz <53418069+DaZhUUU@users.noreply.github.com> Date: Thu, 18 Jun 2026 14:22:22 +0800 Subject: [PATCH] docs: add DeepSeek-V4 compressed state dtype tip (#28613) Co-authored-by: zhujunyu --- .../autoregressive/DeepSeek/DeepSeek-V4.mdx | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index f3002061e..be8d11c40 100644 --- a/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -180,6 +180,19 @@ The generator currently picks values on the **conservative** side (mirroring an - `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves. - MTP runs on the v2 speculative path. +**Compressed attention state dtype** + +DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path: + +```bash Command +SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \ +sglang serve \ + --model-path deepseek-ai/DeepSeek-V4-Flash \ + +``` + +This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior. + **EPLB + DeepEP Waterfill (Experimental)** For recorded/static EPLB reproduction, first record an expert-distribution file by following