docs: sync legacy docs/-only updates into docs_new (Mintlify) (#27308)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
46c58b5c70
commit
c6c1f1a29a
@@ -1203,6 +1203,7 @@
|
||||
"group": "Performance Optimization",
|
||||
"pages": [
|
||||
"docs/sglang-diffusion/performance-optimization",
|
||||
"docs/sglang-diffusion/deployment_cookbook",
|
||||
"docs/sglang-diffusion/ring_sp_performance",
|
||||
"docs/sglang-diffusion/attention_backends",
|
||||
{
|
||||
|
||||
@@ -175,6 +175,13 @@ The following table summarizes quantization method support across NVIDIA and AMD
|
||||
<td>Yes</td>
|
||||
<td>Ascend quantization; Uses CANN kernels</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>mxfp8</code> (diffusion)</td>
|
||||
<td>No</td>
|
||||
<td>No</td>
|
||||
<td>Yes (A2/A3)</td>
|
||||
<td>Ascend NPU only; online MXFP8 quantization for diffusion models (e.g., Wan2.2); requires CANN ≥ 8.0.RC3</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -816,6 +823,36 @@ sglang serve --model-path Qwen/Qwen3-30B-A3B \
|
||||
--quantization quark_mxfp4
|
||||
```
|
||||
|
||||
## Diffusion Model Quantization on Ascend NPU
|
||||
|
||||
SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend A5 NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI.
|
||||
|
||||
**Requirements:** Ascend A5, CANN ≥ 8.0.RC3
|
||||
|
||||
### Online MXFP8
|
||||
|
||||
Pass `--quantization mxfp8` to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization mxfp8 \
|
||||
--num-gpus 4
|
||||
```
|
||||
|
||||
### Offline MXFP8 (ModelSlim)
|
||||
|
||||
Pre-quantize with [msModelSlim](https://gitcode.com/Ascend/msmodelslim) and load the checkpoint directly — the quantization scheme is auto-detected from `quant_model_description.json`:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path /path/to/wan2_2_mxfp8_diffusers \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
For the full quantization + format conversion workflow and a complete list of supported schemes, see [Diffusion Quantization on Ascend NPU](../hardware-platforms/ascend-npus/ascend_npu_quantization#diffusion-model-quantization-on-ascend-npu) and [SGLang-Diffusion Quantization](../sglang-diffusion/quantization#modelslim).
|
||||
|
||||
## Reference
|
||||
|
||||
- [GPTQModel](https://github.com/ModelCloud/GPTQModel)
|
||||
|
||||
@@ -486,6 +486,18 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: float</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prefill-delayer-queue-min-ratio`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Opt-in to the adaptive queue-based delay trigger (independent of the slot-based one). Defers prefill until the waiting queue reaches `min(running_req * ratio, max_prefill_bs)` so small fragments batch into a larger prefill. Unset keeps the original slot-only behavior. Typical: `0.1`–`0.5`.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: float</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prefill-delayer-max-delay-ms`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, prefill is force-released to bound worst-case TTFT. Only consulted when `--prefill-delayer-queue-min-ratio` is set. Typical: `1000`–`5000`.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`5000`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: float</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prefill-delayer-forward-passes-buckets`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Custom buckets for prefill delayer forward passes histogram. 0 and max_delay_passes-1 will be auto-added.</td>
|
||||
@@ -1592,6 +1604,18 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>` None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-deepep-waterfill`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable DeepEP Waterfill: dispatch the shared expert as the 9th routed expert to the least-loaded EP rank. Automatically sets `--moe-a2a-backend deepep`, implicitly enables shared-expert fusion, and supports `--deepep-mode auto`, `normal`, or `low_latency`. Use `auto` or `low_latency` for production decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2. By default, Waterfill uses the static local-batch path; set `SGLANG_DISABLE_STATIC_WATERFILL=1` to force dynamic Waterfill with runtime EP all-reduce.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--elastic-ep-rejoin`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Indicates that this process is a relaunched elastic EP rank that should rejoin an existing process group during rank recovery.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -1632,7 +1656,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mamba-scheduler-strategy`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The strategy to use for mamba scheduler. <code>auto</code> currently defaults to <code>no_buffer</code>. 1. <code>no_buffer</code> does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. <code>extra_buffer</code> supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes <code>2x</code> for non-spec; <code>1+(1/(2+speculative_num_draft_tokens))x</code> for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. <code>extra_buffer</code> is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support (currently only FLA backend), currently only extra_buffer supports branching.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The strategy to use for mamba scheduler. <code>auto</code> currently defaults to <code>no_buffer</code>. 1. <code>no_buffer</code> does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. <code>extra_buffer</code> supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes <code>2x</code> for non-spec; <code>1+(1/(2+speculative_num_draft_tokens))x</code> for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. <code>extra_buffer</code> is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra_buffer supports branching.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`auto`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>auto</code>, <code>no_buffer</code>, <code>extra_buffer</code></td>
|
||||
</tr>
|
||||
@@ -2780,6 +2804,32 @@ Please consult the documentation below and [server_args.py](https://github.com/s
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## For MindStudio-probe(msProbe) dump
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--msprobe-dump-config`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The path of the JSON configuration file for msProbe. If specified, enables msProbe dump.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Deprecated arguments
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
|
||||
@@ -221,7 +221,7 @@ python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>repetition_penalty</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 1.0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values < 1) repetition. Valid range is `[0, 2]`; `1.0` leaves probabilities unchanged.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values < 1) repetition. Valid range is `(0, 2]`; `1.0` leaves probabilities unchanged.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_new_tokens</td>
|
||||
|
||||
@@ -58,6 +58,9 @@ Use `bench_serving` by default unless there are specific needs.
|
||||
python3 -m sglang.bench_one_batch_server --base-url http://127.0.0.1:30000 --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32
|
||||
```
|
||||
|
||||
- Pass `--enable-multi-batch` and set `--batch-size` to a multiple of the server's `--max-running-requests` to stabilize throughput measurements. Surplus requests are queued by the scheduler and promoted batch-by-batch, amortizing per-request prefill and first-step transients into steady-state decode. Under this flag, only `overall_throughput` is authoritative; `input_throughput`, `output_throughput`, `last_ttft`, and ITL include cross-batch queueing in their denominators and should be treated as informational.
|
||||
- Pass `--lora-name <name>` to route every prompt through a pre-loaded LoRA adapter. Requires the server to be launched with `--enable-lora --lora-paths <name>=<path>`.
|
||||
|
||||
**`bench_offline_throughput`** directly instantiates the `Engine` object in-process (no HTTP server) and submits all requests at once via `engine.generate()`. The engine's scheduler handles batching and execution. This measures maximum achievable throughput without any network overhead.
|
||||
|
||||
```bash Command
|
||||
|
||||
@@ -114,10 +114,10 @@ Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com
|
||||
|
||||
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
|
||||
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
|
||||
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
|
||||
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
|
||||
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
|
||||
- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
|
||||
- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
|
||||
- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
|
||||
- `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `<test-spec>` is pytest-style `<file>::<TestClass>[.<test_method>]` (the `::TestClass` and `.<test_method>` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once).
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
@@ -148,9 +148,10 @@ Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob
|
||||
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
|
||||
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
|
||||
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
|
||||
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
|
||||
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible.
|
||||
- Make functions as pure as possible. Avoid in-place modification of arguments.
|
||||
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
|
||||
- In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file.
|
||||
- Keep tests run fast.
|
||||
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
|
||||
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
|
||||
|
||||
@@ -94,10 +94,10 @@ Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com
|
||||
|
||||
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
|
||||
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
|
||||
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
|
||||
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
|
||||
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
|
||||
- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
|
||||
- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
|
||||
- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
|
||||
|
||||
@@ -315,3 +315,132 @@ python3 -m sglang.launch_server \
|
||||
> - GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
|
||||
> - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation.
|
||||
> - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.
|
||||
|
||||
## Diffusion Model Quantization on Ascend NPU
|
||||
|
||||
SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires A5; the ModelSlim W8A8/W4A4 schemes work on A2/A3.
|
||||
|
||||
**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend A5
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Quantization method</th>
|
||||
<th><code>quant_type</code> in JSON</th>
|
||||
<th>Scheme class</th>
|
||||
<th>Mode</th>
|
||||
<th>A2/A3 Supported</th>
|
||||
<th>A5 Supported</th>
|
||||
<th>Trigger</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>MXFP8 (W8A8)</td>
|
||||
<td>—</td>
|
||||
<td><code>MXFP8Config</code></td>
|
||||
<td>Online</td>
|
||||
<td><strong style={{color: 'red'}}>x</strong></td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td><code>--quantization mxfp8</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MXFP8 (W8A8)</td>
|
||||
<td><code>W8A8_MXFP8</code></td>
|
||||
<td><code>ModelSlimMXFP8Scheme</code></td>
|
||||
<td>Offline</td>
|
||||
<td><strong style={{color: 'red'}}>x</strong></td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td>auto-detected from <code>quant_model_description.json</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>W8A8 static</td>
|
||||
<td><code>W8A8</code></td>
|
||||
<td><code>ModelSlimW8A8Int8</code></td>
|
||||
<td>Offline</td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td><strong style={{color: 'orange'}}>TBD</strong></td>
|
||||
<td>auto-detected from <code>quant_model_description.json</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>W8A8 dynamic</td>
|
||||
<td><code>W8A8_DYNAMIC</code></td>
|
||||
<td><code>ModelSlimW8A8Int8</code></td>
|
||||
<td>Offline</td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td><strong style={{color: 'orange'}}>TBD</strong></td>
|
||||
<td>auto-detected from <code>quant_model_description.json</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>W4A4 dynamic</td>
|
||||
<td><code>W4A4_DYNAMIC</code></td>
|
||||
<td><code>ModelSlimW4A4Int4</code></td>
|
||||
<td>Offline</td>
|
||||
<td><strong style={{color: 'green'}}>√</strong></td>
|
||||
<td><strong style={{color: 'orange'}}>TBD</strong></td>
|
||||
<td>auto-detected from <code>quant_model_description.json</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Online MXFP8 Quantization
|
||||
|
||||
Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection.
|
||||
|
||||
```bash Command
|
||||
# Start the diffusion server with online MXFP8 quantization
|
||||
sglang serve \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization mxfp8 \
|
||||
--num-gpus 4
|
||||
```
|
||||
|
||||
```bash Command
|
||||
# One-shot generation
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization mxfp8 \
|
||||
--prompt "a beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### Offline MXFP8 Quantization (ModelSlim)
|
||||
|
||||
For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed.
|
||||
|
||||
**Step 1: Quantize with msModelSlim**
|
||||
|
||||
```bash Command
|
||||
msmodelslim quant \
|
||||
--model_path /path/to/wan2_2_float_weights \
|
||||
--save_path /path/to/wan2_2_mxfp8_weights \
|
||||
--device npu \
|
||||
--model_type Wan2_2 \
|
||||
--quant_type mxfp8 \
|
||||
--trust_remote_code True
|
||||
```
|
||||
|
||||
> Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.
|
||||
|
||||
**Step 2: Convert to Diffusers format**
|
||||
|
||||
msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script:
|
||||
|
||||
```bash Command
|
||||
python python/sglang/multimodal_gen/tools/wan_repack.py \
|
||||
--input-path /path/to/wan2_2_mxfp8_weights \
|
||||
--output-path /path/to/wan2_2_mxfp8_diffusers
|
||||
```
|
||||
|
||||
Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory.
|
||||
|
||||
**Step 3: Run inference**
|
||||
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path /path/to/wan2_2_mxfp8_diffusers \
|
||||
--prompt "a beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).
|
||||
|
||||
@@ -188,6 +188,86 @@ python3 -m sglang_router.launch_router \
|
||||
--prometheus-port 29010
|
||||
```
|
||||
|
||||
#### Running Qwen3-235B-A22B-Instruct-2507-W8A8 with Prefill Context Parallel (CP) on 2 x Atlas 800I A3
|
||||
|
||||
This example enables **Prefill Context Parallel** (`--enable-prefill-context-parallel`) to split the context across CP ranks during prefill, reducing per-device memory pressure and improving TTFT for long sequences. PD disaggregation is required.
|
||||
|
||||
> **Constraints**
|
||||
> - Prefill side must set `--max-running-requests 1` (PCP only supports batch_size=1)
|
||||
> - `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs
|
||||
|
||||
**Prefill node `<PREFILL_HOST_IP>`:**
|
||||
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:23456"
|
||||
export ASCEND_USE_FIA=True
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /mnt/share/weights/Qwen3-235B-A22B-Instruct-2507-W8A8 \
|
||||
--trust-remote-code \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--quantization modelslim \
|
||||
--attention-backend ascend \
|
||||
--skip-server-warmup \
|
||||
--mem-fraction-static 0.7 \
|
||||
--chunked-prefill-size 32768 \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 16 \
|
||||
--enable-prefill-context-parallel \
|
||||
--attn-cp-size 2 \
|
||||
--moe-dp-size 2 \
|
||||
--max-running-requests 1 \
|
||||
--host <PREFILL_HOST_IP> \
|
||||
--port 8000 \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--dist-init-addr <PREFILL_HOST_IP>:6688
|
||||
```
|
||||
|
||||
Key parameters for PCP:
|
||||
|
||||
| Parameter | Value | Description |
|
||||
|-----------|-------|-------------|
|
||||
| `--enable-prefill-context-parallel` | flag | Enable PCP feature |
|
||||
| `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) |
|
||||
| `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` |
|
||||
| `--max-running-requests` | 1 | Required by PCP (batch_size=1 constraint) |
|
||||
|
||||
**Decode node (`<DECODE_HOST_IP>`):**
|
||||
|
||||
```shell Launch Server
|
||||
export ASCEND_MF_STORE_URL="tcp://141.61.39.231:23456"
|
||||
export ASCEND_USE_FIA=True
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /mnt/share/weights/Qwen3-235B-A22B-Instruct-2507-W8A8 \
|
||||
--trust-remote-code \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--quantization modelslim \
|
||||
--attention-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--disable-cuda-graph \
|
||||
--mem-fraction-static 0.7 \
|
||||
--chunked-prefill-size 32768 \
|
||||
--skip-server-warmup \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 8 \
|
||||
--max-running-requests 32 \
|
||||
--host <DECODE_HOST_IP> \
|
||||
--port 8001 \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--dist-init-addr <DECODE_HOST_IP>:6688
|
||||
```
|
||||
|
||||
> **Note:** `ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the Prefill node IP). `ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer. PCP is a Prefill-only feature; the Decode side needs no CP-related flags.
|
||||
|
||||
#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
|
||||
|
||||
@@ -1840,7 +1840,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--offload-mode`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`cpu`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>cpu</code> (DeepSeek only) <br/><code>meta</code> (DeepSeek only) <br/><code>sharded_gpu</code> (DeepSeek only)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>cpu</code> (DeepSeek only) <br/><code>meta</code> (DeepSeek only) <br/><code>sharded_gpu</code> (DeepSeek only, only support tp=1 dp>1)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
|
||||
@@ -142,3 +142,54 @@ python -m sglang.bench_serving -h
|
||||
Additionally, the requests can be formed with
|
||||
[OpenAI Completions API](../basic_usage/openai_api_completions)
|
||||
and sent via the command line (e.g. using `curl`) or via your own script.
|
||||
|
||||
## Prefill-Decode (P/D) Disaggregation on Intel XPU [Experimental]
|
||||
|
||||
SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend.
|
||||
|
||||
**Tested models:**
|
||||
|
||||
| Model | Notes |
|
||||
|:---:|:---:|
|
||||
| [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) |
|
||||
| [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) |
|
||||
|
||||
**Prerequisites:** `pip install nixl sglang-router`
|
||||
|
||||
**Start the prefill server (GPU 0):**
|
||||
|
||||
```bash
|
||||
ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend nixl \
|
||||
--disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
**Start the decode server (GPU 1):**
|
||||
|
||||
```bash
|
||||
ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend nixl \
|
||||
--disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
**Start the router:**
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 \
|
||||
--decode http://127.0.0.1:30001 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
**Send a request:**
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}'
|
||||
```
|
||||
|
||||
> **Note:** `UCX_POSIX_USE_PROC_LINK=n` is required on Intel XPU to avoid UCX shared-memory transport issues.
|
||||
|
||||
@@ -520,6 +520,11 @@ SGLang supports various environment variables that can be used to configure its
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use AITER optimize implementation</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_ROCM_USE_MULTI_STREAM</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Allocate alt CUDA/HIP stream on ROCm/AITER to overlap shared and routed experts in DeepseekV2 MoE. Requires the HIP env <code>GPU_MAX_HW_QUEUES>=5</code> (default <code>4</code>, the cap on HSA/ROCr HW queues HIP creates) so the alt stream gets its own queue instead of serializing with the main stream. Best paired with <code>--deepep-mode low_latency</code> so Mori's AsyncLL kernel offloads dispatch/combine to copy engines and frees CUs.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_MOE_PADDING</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable MoE padding (sets padding size to 128 if value is <code>1</code>, often set to <code>1</code> in Docker builds)</td>
|
||||
|
||||
@@ -55,9 +55,14 @@ This section explains how to configure the request tracing and export the trace
|
||||
0: disable tracing
|
||||
1: Trace important slices
|
||||
2: Trace all slices except nested ones
|
||||
3: Trace all slices
|
||||
3: Trace all slices (default)
|
||||
```
|
||||
The trace level can be dynamically set via HTTP API, for example:
|
||||
**At startup** — set `SGLANG_TRACE_LEVEL` before launching the server:
|
||||
```bash Command
|
||||
SGLANG_TRACE_LEVEL=2 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other options>
|
||||
```
|
||||
|
||||
**At runtime** — dynamically adjust via HTTP API without restarting:
|
||||
```bash Command
|
||||
curl http://0.0.0.0:30000/set_trace_level?level=2
|
||||
```
|
||||
|
||||
@@ -76,10 +76,13 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
|
||||
- `--model-path {MODEL}`: model path or Hugging Face model ID
|
||||
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
|
||||
- `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
|
||||
- `--num-gpus {N}`: number of GPUs to use
|
||||
- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
|
||||
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
|
||||
- `--attention-backend {BACKEND}`: attention backend for native SGLang pipelines
|
||||
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
|
||||
- `--attention-backend-config {CONFIG}`: attention backend configuration
|
||||
@@ -101,6 +104,8 @@ For quantized transformer checkpoints, prefer:
|
||||
- `--model-path` for the base pipeline
|
||||
- `--transformer-path` for a quantized `transformers` transformer component folder
|
||||
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
|
||||
- `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
|
||||
See [Quantization](../quantization) for supported quantization families and examples.
|
||||
|
||||
@@ -158,6 +163,21 @@ HTTP server-only arguments are ignored by `sglang generate`.
|
||||
|
||||
For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../cache_dit).
|
||||
|
||||
### Layerwise Offload
|
||||
|
||||
Use layerwise offload when a large component does not fit comfortably in GPU memory. By default, `--dit-layerwise-offload` only applies to legacy DiT components. Use `--layerwise-offload-components` to select pipeline component names explicitly (`--layerwise-offload-modules` is accepted as an alias):
|
||||
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--dit-layerwise-offload \
|
||||
--layerwise-offload-components transformer text_encoder \
|
||||
--dit-offload-prefetch-size 0 \
|
||||
--prompt "A quiet city street after rain"
|
||||
```
|
||||
|
||||
The values must match keys in the selected pipeline's `pipeline.modules`, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Use `all` to select every layerwise-offloadable component. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency.
|
||||
|
||||
## Serve
|
||||
|
||||
`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.
|
||||
|
||||
@@ -271,13 +271,13 @@ curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
|
||||
The server supports dynamic loading, merging, and unmerging of LoRA adapters.
|
||||
|
||||
**Important Notes:**
|
||||
- Mutual Exclusion: Only one LoRA can be *merged* (active) at a time
|
||||
- Switching: To switch LoRAs, you must first `unmerge` the current one, then `set` the new one
|
||||
- Mutual Exclusion: Only one LoRA configuration can be active per target at a time
|
||||
- Switching: To switch LoRAs, deactivate the current LoRA with `unmerge_lora_weights`, then `set` the new one
|
||||
- Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost
|
||||
|
||||
**Set LoRA Adapter**
|
||||
|
||||
Loads one or more LoRA adapters and merges their weights into the model. Supports both single LoRA (backward compatible) and multiple LoRA adapters.
|
||||
Loads one or more LoRA adapters and applies them to the model. By default, regular weights are statically merged, while FSDP-sharded weights use dynamic LoRA to avoid full-gather memory peaks.
|
||||
|
||||
**Endpoint:** `POST /v1/set_lora`
|
||||
|
||||
@@ -290,6 +290,7 @@ Loads one or more LoRA adapters and merges their weights into the model. Support
|
||||
- `"transformer_2"`: Apply only to transformer_2 (low noise for Wan2.2)
|
||||
- `"critic"`: Apply only to the critic model
|
||||
- `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
|
||||
- `merge_mode` (string, optional): `"auto"` (default server policy), `"merge"` (force static merge), or `"dynamic"` (apply LoRA at forward time)
|
||||
|
||||
**Single LoRA Example:**
|
||||
|
||||
@@ -334,7 +335,7 @@ curl -X POST http://localhost:30010/v1/set_lora \
|
||||
> When using multiple LoRAs:
|
||||
> - All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
|
||||
> - If `target` or `strength` is a single value, it will be applied to all LoRAs
|
||||
> - Multiple LoRAs applied to the same target will be merged in order
|
||||
> - Multiple LoRAs applied to the same target are applied in order
|
||||
|
||||
|
||||
**Merge LoRA Weights**
|
||||
@@ -342,7 +343,7 @@ curl -X POST http://localhost:30010/v1/set_lora \
|
||||
Manually merges the currently set LoRA weights into the base model.
|
||||
|
||||
> [!NOTE]
|
||||
> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.*
|
||||
> With FSDP-sharded weights, manual merge may require a full-gather and can OOM. Use `set_lora` with `merge_mode="auto"` or `"dynamic"` for the lower-peak path.
|
||||
|
||||
**Endpoint:** `POST /v1/merge_lora_weights`
|
||||
|
||||
@@ -398,6 +399,7 @@ curl -sS -X GET "http://localhost:30010/v1/list_loras"
|
||||
"nickname": "lora2",
|
||||
"path": "tarn59/pixel_art_style_lora_z_image_turbo",
|
||||
"merged": true,
|
||||
"mode": "merged",
|
||||
"strength": 1.0
|
||||
}
|
||||
]
|
||||
|
||||
@@ -380,6 +380,36 @@ Optimization columns are abbreviated to keep the matrix readable:
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Cosmos3-Nano (T2V / I2V / T2I)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>nvidia/Cosmos3-Nano</code></td>
|
||||
<td style={{padding: "9px 8px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p · 480p<br />1024×1024 (T2I)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Cosmos3-Super (T2V / I2V / T2I)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>nvidia/Cosmos3-Super</code></td>
|
||||
<td style={{padding: "9px 8px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p · 480p<br />1024×1024 (T2I)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -399,6 +429,11 @@ Optimization columns are abbreviated to keep the matrix readable:
|
||||
- `resident` usually provides the best latency/throughput but uses much more VRAM.
|
||||
- `original` keeps official two-stage semantics without the premerged stage-2 transformer path.
|
||||
- Example (one prior run): `original` `154.67s`, `snapshot` `114.05s`, `resident` `75.71s`; peak VRAM trend is `original < snapshot < resident`.
|
||||
5. Cosmos3 ships in two sizes — `nvidia/Cosmos3-Nano` (8B) and
|
||||
`nvidia/Cosmos3-Super` (32B). Both share the same pipeline; the only
|
||||
difference is transformer depth and width, picked up from
|
||||
`transformer/config.json` at load time. A single checkpoint serves T2V,
|
||||
I2V (`--image-path`), and T2I (`--num-frames 1`).
|
||||
|
||||
### Image Generation Models
|
||||
|
||||
@@ -435,14 +470,6 @@ Optimization columns are abbreviated to keep the matrix readable:
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLUX.2-Klein-9B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>black-forest-labs/FLUX.2-klein-9B</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLUX.2-Klein-Base-4B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>black-forest-labs/FLUX.2-klein-base-4B</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLUX.2-Klein-Base-9B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>black-forest-labs/FLUX.2-klein-base-9B</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Z-Image</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>Tongyi-MAI/Z-Image</code></td>
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: "Deployment Cookbook"
|
||||
description: "Practical defaults for choosing CPU offload, FSDP, CFG parallelism, SP, and TP in SGLang Diffusion."
|
||||
---
|
||||
|
||||
This page gives practical defaults for choosing CPU offload, FSDP, CFG parallelism, SP, and TP.
|
||||
|
||||
## Quick Rule
|
||||
|
||||
Use the simplest setting that fits your memory target:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "40%"}} />
|
||||
<col style={{width: "60%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Goal</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Recommended setting</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fastest single-GPU run when the model fits</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable CPU offload and do not use FSDP.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Lower single-GPU memory usage</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use component CPU offload, or layerwise DiT offload for supported Wan/MOVA models.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Faster multi-GPU Qwen/Wan CFG generation</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use FSDP with CFG parallelism and disable CPU offload.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Sequence length or video-shape scaling</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use SP/Ulysses/Ring when the model benefits from sequence parallelism.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TP compatibility or encoder-heavy paths</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Set TP explicitly; do not treat TP as the default latency optimization.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Base the decision on available memory on the selected GPU(s).
|
||||
|
||||
- For multi-GPU deployment: the least-free selected GPU is the bottleneck. A busy 80GiB GPU can behave like a much smaller GPU.
|
||||
- For single-GPU deployment: FSDP shards DiT weights across multiple GPUs. It is not useful for keeping a single-GPU deployment on one GPU; for that case use CPU offload.
|
||||
|
||||
## Performance Modes
|
||||
|
||||
`--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "82%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Meaning</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`manual`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Keeps performance-related server args under explicit user control.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`auto`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default. Keeps legacy safe offload defaults and uses FSDP/CFG only on validated multi-GPU deployments where FSDP can replace DiT offload.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`speed`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset; may OOM.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`memory`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Favors lower GPU memory. Uses component offload, or Wan/MOVA layerwise DiT offload when supported.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. Text encoder, image encoder, and other component residency still follow the offload policy unless the model marks a high-memory resident path as safe. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk.
|
||||
|
||||
The modes tune residency for native pipeline components declared to the component residency manager. Today this covers the major DiT, text/image encoder, VAE, vocoder, and upsampler components; DiT can use layerwise offload when supported, while text encoders use either resident execution or component CPU offload. Do not assume text-encoder layerwise offload unless a model implements and validates it.
|
||||
|
||||
> [!NOTE]
|
||||
> The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--num-gpus 2 \
|
||||
--performance-mode auto
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--performance-mode memory
|
||||
```
|
||||
|
||||
Explicit flags win over the mode:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--num-gpus 2 \
|
||||
--performance-mode auto \
|
||||
--use-fsdp-inference false
|
||||
```
|
||||
|
||||
In this example, `auto` will not re-enable FSDP. The same applies to parallelism; for example, `--enable-cfg-parallel false` keeps CFG parallelism disabled.
|
||||
|
||||
## Interpreting The Levers
|
||||
|
||||
**No offload** keeps model components resident on GPU. It is usually fastest when memory is sufficient.
|
||||
|
||||
**Component CPU offload** lowers GPU memory by moving large components to CPU. It is simple and robust, but it usually trades latency for memory.
|
||||
|
||||
**Layerwise DiT offload** lowers DiT memory further for supported Wan/MOVA models by moving DiT layers between CPU and GPU. It can be the best single-GPU memory mode, but may increase latency and lower throughput.
|
||||
|
||||
**FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads.
|
||||
|
||||
FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths.
|
||||
|
||||
**CFG parallelism** splits positive and negative CFG branches across GPUs. For Qwen/Wan workloads with normal step counts, this is the most reliable multi-GPU speedup observed so far.
|
||||
|
||||
**SP/Ulysses/Ring** splits sequence work. It can help video workloads, but validated Qwen/Wan runs showed CFG parallelism outperforming SP for latency.
|
||||
|
||||
**TP** is supported for compatibility and some model structures, but current measurements do not make it the default latency path for Qwen/Wan.
|
||||
|
||||
## Current Benchmark Takeaways
|
||||
|
||||
Observed regular-scale trends:
|
||||
|
||||
- Z-Image: single-GPU no-offload was faster than FSDP/SP in the tested setting; keep FSDP off unless memory or parallelism requires it.
|
||||
- Qwen-Image: keep the default non-FSDP path unless a specific FSDP/SP/Ring setting has been benchmarked on the target hardware.
|
||||
- Wan: FSDP can replace DiT offload on validated multi-GPU workloads, while text/image encoders may still need component offload. Keep model-specific precision checks before making FSDP automatic for a path.
|
||||
- Component offload mainly reduced memory; it did not improve latency in the tested no-offload-vs-offload runs.
|
||||
|
||||
Always benchmark with your actual resolution, frame count, step count, and GPU type before locking production defaults.
|
||||
@@ -14,8 +14,10 @@ Use these paths:
|
||||
- `--model-path`: the base or original model
|
||||
- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
|
||||
- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
|
||||
- `--quantization`: apply online quantization to unquantized models at load time (activations are quantized dynamically)
|
||||
- `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`)
|
||||
|
||||
Recommended example:
|
||||
Recommended example for pre-quantized checkpoints:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
@@ -65,7 +67,15 @@ backend.
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>fp8</code></td>
|
||||
<td><code>fp8</code> / <code>mxfp4</code> (online quantization)</td>
|
||||
<td>Unquantized checkpoint (offline via AMD Quark coming soon)</td>
|
||||
<td><code>--quantization {fp8,mxfp4}</code></td>
|
||||
<td>Z-Image-Turbo (validated), others likely work. More support coming soon.</td>
|
||||
<td>MXFP4: <code>aiter</code> on ROCm</td>
|
||||
<td>MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to <code>fp8</code> / <code>mxfp4</code> dynamically.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>fp8</code> (offline quantization)</td>
|
||||
<td>Quantized transformer component folder, or safetensors with <code>quantization_config</code> metadata</td>
|
||||
<td><code>--transformer-path</code> or <code>--transformer-weights-path</code></td>
|
||||
<td>ALL</td>
|
||||
@@ -107,6 +117,58 @@ backend.
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Online Quantization
|
||||
|
||||
Online quantization applies quantization to unquantized models at load time. This is useful for when pre-quantized checkpoints are not available.
|
||||
|
||||
### FP8 Online Quantization
|
||||
|
||||
Apply FP8 quantization to any unquantized model:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization fp8 \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### MXFP4 Online Quantization
|
||||
|
||||
MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.**
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization mxfp4 \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
**Note:** Requires `aiter` package with MXFP4 kernel support
|
||||
|
||||
### Skipping Layers
|
||||
|
||||
By default, online quantization quantizes every linear layer in
|
||||
the transformer. However, `--quantization-ignored-layers` can be used to keep specific layers in their original precision:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization fp8 \
|
||||
--quantization-ignored-layers attention.to_ \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
|
||||
sglang generate \
|
||||
--model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--quantization mxfp4 \
|
||||
--quantization-ignored-layers attention.to_ \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantizd if its prefix contains any of the given patterns.
|
||||
|
||||
## Validated ModelOpt Checkpoints
|
||||
|
||||
This section is the canonical support matrix for the nine diffusion ModelOpt
|
||||
|
||||
Reference in New Issue
Block a user