[diffusion] cli: support component attention backend overrides (#24320)
This commit is contained in:
@@ -78,6 +78,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--attention-backend {BACKEND}`: attention backend for native SGLang pipelines
|
||||
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
|
||||
- `--attention-backend-config {CONFIG}`: attention backend configuration
|
||||
|
||||
### Sampling and output
|
||||
@@ -195,6 +196,28 @@ sglang serve \
|
||||
|
||||
The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.
|
||||
|
||||
## Component Attention Backend Overrides
|
||||
|
||||
Use `--component-attention-backends` when one pipeline component needs a different native attention backend from the global `--attention-backend`.
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Lightricks/LTX-2.3 \
|
||||
--attention-backend fa \
|
||||
--component-attention-backends text_encoder=torch_sdpa
|
||||
```
|
||||
|
||||
The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` only while that component is being constructed.
|
||||
|
||||
You can also pass dotted CLI entries:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--component-attention-backends.text_encoder torch_sdpa \
|
||||
--component-attention-backends.transformer fa
|
||||
```
|
||||
|
||||
## Diffusers Backend
|
||||
|
||||
Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.
|
||||
|
||||
@@ -42,8 +42,9 @@ For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBa
|
||||
The selection order in `runtime/layers/attention/selector.py` is:
|
||||
|
||||
1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
|
||||
2. CLI `--attention-backend` (`ServerArgs.attention_backend`)
|
||||
3. Auto selection (platform capability, dtype, and installed packages)
|
||||
2. Component override from `--component-attention-backends` while that component is being constructed
|
||||
3. CLI `--attention-backend` (`ServerArgs.attention_backend`)
|
||||
4. Auto selection (platform capability, dtype, and installed packages)
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -122,6 +123,20 @@ sglang generate \
|
||||
--attention-backend torch_sdpa
|
||||
```
|
||||
|
||||
### Override one component
|
||||
|
||||
Use component overrides when a specific module needs different attention semantics from the main transformer:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend fa \
|
||||
--component-attention-backends text_encoder=torch_sdpa
|
||||
```
|
||||
|
||||
Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`.
|
||||
|
||||
### Using Sliding Tile Attention (STA)
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user