[diffusion] feat: support multiple task types for pipelines (#38762)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
This commit is contained in:
Yihao Wang
2026-09-28 10:00:45 +08:00
committed by GitHub
co-authored by Mick Qian
parent 6cbdfec453
commit 8b2ca8ecc2
25 changed files with 1036 additions and 155 deletions
@@ -75,6 +75,7 @@ curl -sS "http://localhost:30010/v1/models"
"max_model_len": null,
"num_gpus": 4,
"task_type": "T2V",
"supported_task_types": ["T2V"],
"dit_precision": "bf16",
"vae_precision": "fp16",
"pipeline_name": "WanPipeline",
@@ -96,6 +97,37 @@ curl -sS "http://localhost:30010/v1/models/wan-t2v"
## Endpoints
### Select a request task
`GET /v1/models` reports `task_type` (the configured default) and
`supported_task_types` (the tasks implemented by that configured pipeline).
`GET /model_info` also reports the supported `output_types`. A pipeline may
support image and video output on one server when its implementation and
loaded components provide both paths.
Send the optional SGLang extension `task_type` as a case-insensitive enum name:
| Interface | Task field | Output constraint |
| --- | --- | --- |
| `POST /v1/images/generations` JSON | `"task_type": "t2i"` | Image |
| `POST /v1/images/edits` multipart | `-F task_type=i2i` | Image |
| `POST /v1/videos` JSON or multipart | `"task_type": "i2v"` or `-F task_type=i2v` | Video |
| OpenAI Python client | `extra_body={"task_type": "t2i"}` | Selected endpoint |
| Native Python `generate` sampling parameters | `"task_type": "t2i"` | Selected task |
| `sglang generate` | `--task-type t2i` | Selected task |
Choose only a task in the server's advertised set. An unsupported task, a task
with the wrong output type for the endpoint, or missing required conditioning
returns HTTP 400. For example, `i2v` needs image input, while `v2v` and `f2v`
need video input. Model-specific size, frame count, and reference constraints
still apply.
If you omit `task_type`, the server uses the endpoint and supplied inputs to
choose a compatible task, preferring its default when compatible. It asks for
an explicit task if several alternatives remain (for example `v2v` versus
`f2v`). Pipelines with one task keep that task as their default. Sending a
request never changes the server default or subsequent requests.
### Prompt enhancement
Image generation, image edits and video creation accept `enhance_prompt=true`
@@ -559,3 +559,69 @@ Common failure points:
- [ ] Verified image or video quality against a reference output.
- [ ] Completed the distributed and memory integration checks above.
- [ ] Tested CFG parallelism and distributed serving paths when they apply.
## Declare multiple task types
Keep `PipelineConfig.task_type` as the default task. To let one configured
pipeline serve several tasks, declare its capabilities as a class variable:
```python
from dataclasses import dataclass
from typing import ClassVar
from sglang.multimodal_gen.configs.pipeline_configs.base import PipelineConfig
from sglang.multimodal_gen.configs.task_type import ModelTaskType
@dataclass
class MultiTaskPipelineConfig(PipelineConfig):
task_type: ModelTaskType = ModelTaskType.T2V
supported_task_types: ClassVar[tuple[ModelTaskType, ...]] = (
ModelTaskType.T2I,
ModelTaskType.I2I,
ModelTaskType.T2V,
ModelTaskType.I2V,
)
```
The declaration must be nonempty, contain no duplicates, and include the
configured default. Existing pipelines that omit it retain the singleton
`(task_type,)`. Override `get_supported_task_types()` if capabilities depend
on the loaded checkpoint or configuration, preserving these invariants.
Flat CLI/Python `task_type` arguments select the request. To change the
configured default, use `pipeline_config={"task_type": ModelTaskType.I2V}`
(or its serialized configuration), choosing a supported task.
Capabilities describe implemented execution paths; declaring a task does not
load additional components or implement conditioning and decoding for it.
Requests carry `SamplingParams.task_type`, accepting an enum or its name
(case-insensitive). Admission resolves it to an enum and derives the output
`data_type`. Explicit tasks must belong to the supported set and match the
HTTP endpoint's output type. Without an explicit task, admission filters by
output type and conditioning inputs, prefers a compatible default, then
selects a unique compatible alternative. Python/CLI requests also prefer the
default's output type among alternatives. Ambiguous requests must specify
`task_type`. No request changes the shared configuration.
Use the admitted `batch.task_type` in model stages when behavior depends on
the request. Shared stages use `get_request_task_type(batch, pipeline_config)`
to preserve compatibility with direct callers that have no resolved task.
Audit model-specific frame adjustment, reference loading, latent construction,
and decoding for assumptions about the server default. Declare the union of
components used by all supported tasks in stage residency metadata. Synthetic
warmup uses the default task; provide the pipeline's warmup hook when that task
needs model-specific reference inputs. Dynamic batching compares the admitted
task along with the other sampling fields.
`I2I` and `I2V` require image input; `T2I` and `T2V` reject it. Existing `TI2I`
and `TI2V` retain optional image conditioning. `V2V` and `F2V` require video
input; `F2V` denotes video-prefix conditioning. Models own the details of
prefix length and temporal alignment. Pipelines with explicit capabilities
reject video input for tasks that do not accept it; legacy singleton pipelines
retain their model-owned video checks.
See [request task selection](/docs/sglang-diffusion/api/openai_api#select-a-request-task)
for the HTTP interface. Test task selection, invalid inputs, discovery,
serialization, batching, and alternating tasks on one running server.