docs: sync legacy docs/-only updates into docs_new (Mintlify) (#27308)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
46c58b5c70
commit
c6c1f1a29a
@@ -14,8 +14,10 @@ Use these paths:
|
||||
- `--model-path`: the base or original model
|
||||
- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
|
||||
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
|
||||
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
|
||||
Recommended example:
|
||||
Recommended example for pre-quantized checkpoints:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
@@ -65,7 +67,15 @@ backend.
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>fp8</code></td>
|
||||
<td><code>fp8</code> / <code>mxfp4</code> (online quantization)</td>
|
||||
<td>Unquantized checkpoint (offline via AMD Quark coming soon)</td>
|
||||
<td><code>--quantization {fp8,mxfp4}</code></td>
|
||||
<td>Z-Image-Turbo (validated), others likely work. More support coming soon.</td>
|
||||
<td>MXFP4: <code>aiter</code> on ROCm</td>
|
||||
<td>MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to <code>fp8</code> / <code>mxfp4</code> dynamically.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>fp8</code> (offline quantization)</td>
|
||||
<td>Quantized transformer component folder, or safetensors with <code>quantization_config</code> metadata</td>
|
||||
<td><code>--transformer-path</code> or <code>--transformer-weights-path</code></td>
|
||||
<td>ALL</td>
|
||||
@@ -107,6 +117,58 @@ backend.
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Online Quantization
|
||||
|
||||
Online quantization applies quantization to unquantized models at load time. This is useful for when pre-quantized checkpoints are not available.
|
||||
|
||||
### FP8 Online Quantization
|
||||
|
||||
Apply FP8 quantization to any unquantized model:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization fp8 \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### MXFP4 Online Quantization
|
||||
|
||||
MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.**
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization mxfp4 \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
**Note:** Requires `aiter` package with MXFP4 kernel support
|
||||
|
||||
### Skipping Layers
|
||||
|
||||
By default, online quantization quantizes every linear layer in
|
||||
the transformer. However, `--quantization-ignored-layers` can be used to keep specific layers in their original precision:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization fp8 \
|
||||
--quantization-ignored-layers attention.to_ \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization mxfp4 \
|
||||
--quantization-ignored-layers attention.to_ \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantizd if its prefix contains any of the given patterns.
|
||||
|
||||
## Validated ModelOpt Checkpoints
|
||||
|
||||
This section is the canonical support matrix for the nine diffusion ModelOpt
|
||||
|
||||
Reference in New Issue
Block a user