docs: sync legacy docs/-only updates into docs_new (Mintlify) (#27308)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-06-04 19:45:13 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 46c58b5c70
commit c6c1f1a29a
18 changed files with 650 additions and 29 deletions
@@ -14,8 +14,10 @@ Use these paths:
- `--model-path`: the base or original model
- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
Recommended example:
Recommended example for pre-quantized checkpoints:
```bash
sglang generate \
@@ -65,7 +67,15 @@ backend.
</thead>
<tbody>
<tr>
<td><code>fp8</code></td>
<td><code>fp8</code> / <code>mxfp4</code> (online quantization)</td>
<td>Unquantized checkpoint (offline via AMD Quark coming soon)</td>
<td><code>--quantization &#123;fp8,mxfp4&#125;</code></td>
<td>Z-Image-Turbo (validated), others likely work. More support coming soon.</td>
<td>MXFP4: <code>aiter</code> on ROCm</td>
<td>MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to <code>fp8</code> / <code>mxfp4</code> dynamically.</td>
</tr>
<tr>
<td><code>fp8</code> (offline quantization)</td>
<td>Quantized transformer component folder, or safetensors with <code>quantization_config</code> metadata</td>
<td><code>--transformer-path</code> or <code>--transformer-weights-path</code></td>
<td>ALL</td>
@@ -107,6 +117,58 @@ backend.
</tbody>
</table>
## Online Quantization
Online quantization applies quantization to unquantized models at load time. This is useful for when pre-quantized checkpoints are not available.
### FP8 Online Quantization
Apply FP8 quantization to any unquantized model:
```bash
sglang generate \
--model-path Tongyi-MAI/Z-Image-Turbo \
--quantization fp8 \
--prompt "a beautiful sunset" \
--save-output
```
### MXFP4 Online Quantization
MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.**
```bash
sglang generate \
--model-path Tongyi-MAI/Z-Image-Turbo \
--quantization mxfp4 \
--prompt "a beautiful sunset" \
--save-output
```
**Note:** Requires `aiter` package with MXFP4 kernel support
### Skipping Layers
By default, online quantization quantizes every linear layer in
the transformer. However, `--quantization-ignored-layers` can be used to keep specific layers in their original precision:
```bash
sglang generate \
--model-path Tongyi-MAI/Z-Image-Turbo \
--quantization fp8 \
--quantization-ignored-layers attention.to_ \
--prompt "a beautiful sunset" \
--save-output
sglang generate \
--model-path Tongyi-MAI/Z-Image-Turbo \
--quantization mxfp4 \
--quantization-ignored-layers attention.to_ \
--prompt "a beautiful sunset" \
--save-output
```
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantizd if its prefix contains any of the given patterns.
## Validated ModelOpt Checkpoints
This section is the canonical support matrix for the nine diffusion ModelOpt