[diffusion] feat: support quant-videogen prq kv-cache quantization (memory-saving) for causal-dit (#32581)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Mick <mickjagger19@icloud.com>
This commit is contained in:
Haocheng Xi
co-authored by
Claude Opus 4.8
Mick
Claude Opus 4.8
Mick
parent
24c84dfa68
commit
f64328c7f6
@@ -114,7 +114,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
|
||||
For frame interpolation and upscaling, see [Post-Processing](./post_processing).
|
||||
|
||||
### Quantized transformers
|
||||
### Quantization
|
||||
|
||||
For quantized transformer checkpoints, prefer:
|
||||
|
||||
@@ -124,7 +124,13 @@ For quantized transformer checkpoints, prefer:
|
||||
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
|
||||
See [Quantization](../quantization) for supported quantization families and examples.
|
||||
For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}`
|
||||
compresses completed KV-cache chunks independently of transformer weight
|
||||
quantization. It is lossy and disabled by default.
|
||||
|
||||
See [Realtime and Causal Video Models](../realtime_models) for the runtime and
|
||||
model scope, and [Quantization](../quantization) for supported quantization
|
||||
families and examples.
|
||||
|
||||
### Request logging
|
||||
|
||||
|
||||
@@ -35,7 +35,8 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010
|
||||
- [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server
|
||||
- [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers
|
||||
- [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost
|
||||
- [Quantization](/docs/sglang-diffusion/quantization): load quantized transformer checkpoints
|
||||
- [Quantization](/docs/sglang-diffusion/quantization): configure transformer weight and causal KV-cache quantization
|
||||
- [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models): understand session state, causal caches, and realtime-only controls
|
||||
- [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines
|
||||
|
||||
## Additional Documentation
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
title: "Quantization"
|
||||
tag: "approx"
|
||||
metatags:
|
||||
description: "SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep the base model and the quantized transformer override separate."
|
||||
description: "Configure transformer weight quantization and Quant-VideoGen causal KV-cache quantization in SGLang-Diffusion."
|
||||
---
|
||||
|
||||
SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep
|
||||
@@ -17,6 +17,7 @@ Use these paths:
|
||||
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
|
||||
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
- `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models
|
||||
|
||||
Recommended example for pre-quantized checkpoints:
|
||||
|
||||
@@ -99,6 +100,14 @@ backend.
|
||||
<td>None</td>
|
||||
<td>Mixed override repos keep the base model separate; full Qwen Image exports can be loaded directly as <code>--model-path</code>; raw exports such as <code>black-forest-labs/FLUX.2-dev-NVFP4</code> still use the weights-path flow</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>qvg-kv</code></td>
|
||||
<td>Unquantized model with runtime causal KV-cache compression</td>
|
||||
<td><code>--kv-cache-quant {int4,int2}</code></td>
|
||||
<td>LingBot World realtime causal path</td>
|
||||
<td><code>quant-videogen</code></td>
|
||||
<td>CUDA only; compresses completed cache chunks rather than model weights; lossy and disabled by default</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>nunchaku-svdq</code></td>
|
||||
<td>Pre-quantized Nunchaku transformer weights, usually named <code>svdq-{int4\|fp4}_r{rank}-...</code></td>
|
||||
@@ -118,6 +127,102 @@ backend.
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Causal KV-Cache Quantization
|
||||
|
||||
Quant-VideoGen KV-cache quantization targets long-running autoregressive video
|
||||
sessions, where the causal self-attention cache can become comparable to the
|
||||
model weights. It does not change or quantize the checkpoint weights.
|
||||
|
||||
See [Realtime and Causal Video Models](./realtime_models) for the session
|
||||
lifecycle, supported pipelines, and the distinction between realtime and
|
||||
request-based causal generation.
|
||||
|
||||
Install the optional dependency without allowing its stale Torch requirement to
|
||||
replace SGLang's pinned Torch version, then enable int4 compression when serving
|
||||
a supported LingBot World realtime pipeline:
|
||||
|
||||
```bash
|
||||
pip install "sglang[diffusion,diffusion-qvg]"
|
||||
pip install --no-deps quant-videogen==0.1.0
|
||||
|
||||
sglang serve \
|
||||
--model-path robbyant/lingbot-world-fast-diffusers \
|
||||
--pipeline-class-name LingBotWorldCausalDMDPipeline \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 4 \
|
||||
--kv-cache-quant int4 \
|
||||
--dit-cpu-offload false \
|
||||
--text-encoder-cpu-offload false
|
||||
```
|
||||
|
||||
### Storage Policy
|
||||
|
||||
The current chunk is rewritten at every denoising step, so it remains in BF16.
|
||||
The newest `--kv-cache-quant-keep-recent` completed chunks also remain in BF16.
|
||||
Older completed chunks are stable and are packed once with Progressive Residual
|
||||
Quantization (PRQ); their dense BF16 tensors are then released.
|
||||
|
||||
When a transformer layer runs attention, its packed visible chunks are
|
||||
dequantized and concatenated with the recent BF16 chunks. This creates one
|
||||
layer's dense attention view at a time instead of keeping dense windows
|
||||
resident for every transformer layer.
|
||||
|
||||
### How PRQ Works
|
||||
|
||||
For each K or V vector, PRQ uses k-means to select a centroid, then quantizes
|
||||
the remaining error:
|
||||
|
||||
```text
|
||||
x = centroid_1 + residual_1
|
||||
residual_1 = centroid_2 + residual_2
|
||||
...
|
||||
x_hat = centroid_1 + centroid_2 + ... + dequantize(low_bit_residual)
|
||||
```
|
||||
|
||||
Each additional stage applies another centroid lookup to the previous stage's
|
||||
residual. SGLang's default uses one stage, 128 centroids, and an int4 or int2
|
||||
block-quantized residual. More stages or centroids can reduce reconstruction
|
||||
error but add codebook storage and packing work.
|
||||
|
||||
PRQ is the compression algorithm; selecting older completed chunks is the
|
||||
runtime storage policy that makes it practical. Stable chunks are compressed
|
||||
once, while mutable and recent chunks avoid repeated packing and retain higher
|
||||
precision.
|
||||
|
||||
### Quality And Performance
|
||||
|
||||
<Warning>
|
||||
KV-cache quantization is lossy. Disabling it uses the original dense BF16 cache
|
||||
and is bit-exact with the unmodified path. Enabling int4 or int2 reconstructs an
|
||||
approximation of K and V, so fixed-seed generated frames are not expected to be
|
||||
pixel-identical to BF16.
|
||||
</Warning>
|
||||
|
||||
In the initial LingBot measurements, int4 used about 47% of the dense resident
|
||||
KV-cache memory for a 24-frame window and added about 18% per-chunk latency.
|
||||
Int2 used about 37% of the dense resident KV-cache memory but introduces more
|
||||
quantization error. These measurements are configuration-specific; benchmark
|
||||
memory, latency, temporal consistency, identity stability, and motion quality
|
||||
on the intended session length. Start with int4 unless capacity requires int2.
|
||||
|
||||
The current implementation is limited to the LingBot realtime
|
||||
sliding-window-and-sink path, including Ulysses sequence sharding. It does not
|
||||
support LongLive2 pinned sinks, global sinks, or dynamically growing caches.
|
||||
|
||||
### Tuning Options
|
||||
|
||||
| Option | Default | Effect |
|
||||
| --- | ---: | --- |
|
||||
| `--kv-cache-quant {off,int4,int2}` | `off` | Enables QVG KV-cache compression and selects residual precision. |
|
||||
| `--kv-cache-quant-stages` | `1` | Number of progressive centroid-residual stages. |
|
||||
| `--kv-cache-quant-centroids` | `128` | Number of k-means centroids per stage. |
|
||||
| `--kv-cache-quant-block-size` | `64` | Block size used to quantize the final residual. |
|
||||
| `--kv-cache-quant-iters` | `2` | K-means iterations used while packing a chunk. |
|
||||
| `--kv-cache-quant-asymmetric` | disabled | Uses asymmetric residual quantization. |
|
||||
| `--kv-cache-quant-keep-recent` | `1` | Number of newest completed chunks retained in BF16. |
|
||||
| `--kv-cache-quant-sink {0,1}` | `1` | Whether to quantize completed sink chunks. |
|
||||
| `--kv-cache-quant-sink-keep` | `0` | Number of leading sink chunks retained in BF16. |
|
||||
|
||||
## Online Quantization
|
||||
|
||||
Online quantization applies quantization to unquantized models at load time. This is useful for when pre-quantized checkpoints are not available.
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: "Realtime and Causal Video Models"
|
||||
metatags:
|
||||
description: "Deploy session-based realtime and request-based causal video models with SGLang Diffusion."
|
||||
---
|
||||
|
||||
Realtime and causal video pipelines generate video incrementally and reuse state
|
||||
across chunks. This differs from offline diffusion pipelines, which denoise one
|
||||
bounded latent sequence and release all request state when generation finishes.
|
||||
|
||||
## Execution Modes
|
||||
|
||||
SGLang Diffusion exposes two related but distinct modes:
|
||||
|
||||
| Mode | Lifetime | Interface | Examples |
|
||||
| --- | --- | --- | --- |
|
||||
| Realtime session | State persists until the client disconnects or the session ends | `/v1/realtime_video/generate` WebSocket | LingBot World, SANA-WM realtime |
|
||||
| Request-based causal generation | State is reused across chunks within one request, then released | Standard video generation API | LongLive 2.0, batch-streaming SANA-WM |
|
||||
|
||||
The realtime server retains model-specific state such as the causal self-attention
|
||||
KV cache, cross-attention cache, decoder history, and pending control events.
|
||||
State is isolated per session and is not reused by unrelated requests.
|
||||
|
||||
<Note>
|
||||
A causal DiT is not automatically a realtime session model. The pipeline must
|
||||
also register a realtime adapter and implement the WebSocket session lifecycle.
|
||||
</Note>
|
||||
|
||||
## Supported Realtime Pipelines
|
||||
|
||||
| Model family | Pipeline | Live controls | QVG KV-cache quantization |
|
||||
| --- | --- | --- | --- |
|
||||
| LingBot World | `LingBotWorldCausalDMDPipeline` | Camera actions and prompt updates | Supported |
|
||||
| SANA-WM | `SanaWMRealtimePipeline` | Camera actions | Not supported |
|
||||
|
||||
Use the model cookbooks for launch commands, request schemas, and control-token
|
||||
details:
|
||||
|
||||
- [LingBot World](/cookbook/diffusion/LingBot-World/LingBot-World)
|
||||
- [LingBot World 2.0](/cookbook/diffusion/LingBot-World/LingBot-World-2.0)
|
||||
- [SANA-WM](/cookbook/diffusion/SANA-WM/SANA-WM)
|
||||
|
||||
For the complete model list, see
|
||||
[Supported Models and Optimization Compatibility](./compatibility_matrix).
|
||||
|
||||
## Causal Cache Controls
|
||||
|
||||
Realtime requests can override two model defaults:
|
||||
|
||||
- `realtime_causal_sink_size`: amount of stable prefix history retained as an attention sink
|
||||
- `realtime_causal_kv_cache_num_frames`: recent causal history retained in the rolling KV-cache window
|
||||
|
||||
Larger windows preserve more history but increase resident memory and attention
|
||||
work. These fields are request/session controls; supported ranges and defaults
|
||||
remain model-specific.
|
||||
|
||||
For supported LingBot World deployments, the server-level
|
||||
`--kv-cache-quant {off,int4,int2}` option compresses completed cache chunks.
|
||||
It is disabled by default and is lossy when enabled. Start with `int4`; use
|
||||
`int2` only when the additional memory reduction is worth the larger quality
|
||||
risk.
|
||||
|
||||
See [Causal KV-Cache Quantization](./quantization#causal-kv-cache-quantization)
|
||||
for installation, storage policy, tuning options, memory/latency tradeoffs, and
|
||||
current limitations.
|
||||
|
||||
<Warning>
|
||||
QVG KV-cache quantization currently supports only the LingBot realtime
|
||||
sliding-window-and-sink path. It does not apply to SANA-WM realtime, LongLive 2.0
|
||||
pinned sinks, global sinks, or dynamically growing caches.
|
||||
</Warning>
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
- Keep `--kv-cache-quant off` when bit-exact BF16 cache behavior is required.
|
||||
- Benchmark a representative session length. Short clips may not exercise cold-cache packing and can hide both its memory benefit and packing overhead.
|
||||
- Treat sequence parallelism as model-specific. Follow the model cookbook and the [Sequence Parallelism](./ring_sp_performance) guide instead of assuming one mesh is best for every realtime pipeline.
|
||||
- Realtime WebSocket clients must send an initialization message before control events. The exact MessagePack schema and output encoding are documented in each model cookbook.
|
||||
Reference in New Issue
Block a user