[diffusion] feat: support multiple task types for pipelines (#38762)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
This commit is contained in:
@@ -75,6 +75,7 @@ curl -sS "http://localhost:30010/v1/models"
|
||||
"max_model_len": null,
|
||||
"num_gpus": 4,
|
||||
"task_type": "T2V",
|
||||
"supported_task_types": ["T2V"],
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16",
|
||||
"pipeline_name": "WanPipeline",
|
||||
@@ -96,6 +97,37 @@ curl -sS "http://localhost:30010/v1/models/wan-t2v"
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Select a request task
|
||||
|
||||
`GET /v1/models` reports `task_type` (the configured default) and
|
||||
`supported_task_types` (the tasks implemented by that configured pipeline).
|
||||
`GET /model_info` also reports the supported `output_types`. A pipeline may
|
||||
support image and video output on one server when its implementation and
|
||||
loaded components provide both paths.
|
||||
|
||||
Send the optional SGLang extension `task_type` as a case-insensitive enum name:
|
||||
|
||||
| Interface | Task field | Output constraint |
|
||||
| --- | --- | --- |
|
||||
| `POST /v1/images/generations` JSON | `"task_type": "t2i"` | Image |
|
||||
| `POST /v1/images/edits` multipart | `-F task_type=i2i` | Image |
|
||||
| `POST /v1/videos` JSON or multipart | `"task_type": "i2v"` or `-F task_type=i2v` | Video |
|
||||
| OpenAI Python client | `extra_body={"task_type": "t2i"}` | Selected endpoint |
|
||||
| Native Python `generate` sampling parameters | `"task_type": "t2i"` | Selected task |
|
||||
| `sglang generate` | `--task-type t2i` | Selected task |
|
||||
|
||||
Choose only a task in the server's advertised set. An unsupported task, a task
|
||||
with the wrong output type for the endpoint, or missing required conditioning
|
||||
returns HTTP 400. For example, `i2v` needs image input, while `v2v` and `f2v`
|
||||
need video input. Model-specific size, frame count, and reference constraints
|
||||
still apply.
|
||||
|
||||
If you omit `task_type`, the server uses the endpoint and supplied inputs to
|
||||
choose a compatible task, preferring its default when compatible. It asks for
|
||||
an explicit task if several alternatives remain (for example `v2v` versus
|
||||
`f2v`). Pipelines with one task keep that task as their default. Sending a
|
||||
request never changes the server default or subsequent requests.
|
||||
|
||||
### Prompt enhancement
|
||||
|
||||
Image generation, image edits and video creation accept `enhance_prompt=true`
|
||||
|
||||
@@ -559,3 +559,69 @@ Common failure points:
|
||||
- [ ] Verified image or video quality against a reference output.
|
||||
- [ ] Completed the distributed and memory integration checks above.
|
||||
- [ ] Tested CFG parallelism and distributed serving paths when they apply.
|
||||
|
||||
|
||||
## Declare multiple task types
|
||||
|
||||
Keep `PipelineConfig.task_type` as the default task. To let one configured
|
||||
pipeline serve several tasks, declare its capabilities as a class variable:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from typing import ClassVar
|
||||
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.base import PipelineConfig
|
||||
from sglang.multimodal_gen.configs.task_type import ModelTaskType
|
||||
|
||||
|
||||
@dataclass
|
||||
class MultiTaskPipelineConfig(PipelineConfig):
|
||||
task_type: ModelTaskType = ModelTaskType.T2V
|
||||
supported_task_types: ClassVar[tuple[ModelTaskType, ...]] = (
|
||||
ModelTaskType.T2I,
|
||||
ModelTaskType.I2I,
|
||||
ModelTaskType.T2V,
|
||||
ModelTaskType.I2V,
|
||||
)
|
||||
```
|
||||
|
||||
The declaration must be nonempty, contain no duplicates, and include the
|
||||
configured default. Existing pipelines that omit it retain the singleton
|
||||
`(task_type,)`. Override `get_supported_task_types()` if capabilities depend
|
||||
on the loaded checkpoint or configuration, preserving these invariants.
|
||||
Flat CLI/Python `task_type` arguments select the request. To change the
|
||||
configured default, use `pipeline_config={"task_type": ModelTaskType.I2V}`
|
||||
(or its serialized configuration), choosing a supported task.
|
||||
|
||||
Capabilities describe implemented execution paths; declaring a task does not
|
||||
load additional components or implement conditioning and decoding for it.
|
||||
|
||||
Requests carry `SamplingParams.task_type`, accepting an enum or its name
|
||||
(case-insensitive). Admission resolves it to an enum and derives the output
|
||||
`data_type`. Explicit tasks must belong to the supported set and match the
|
||||
HTTP endpoint's output type. Without an explicit task, admission filters by
|
||||
output type and conditioning inputs, prefers a compatible default, then
|
||||
selects a unique compatible alternative. Python/CLI requests also prefer the
|
||||
default's output type among alternatives. Ambiguous requests must specify
|
||||
`task_type`. No request changes the shared configuration.
|
||||
|
||||
Use the admitted `batch.task_type` in model stages when behavior depends on
|
||||
the request. Shared stages use `get_request_task_type(batch, pipeline_config)`
|
||||
to preserve compatibility with direct callers that have no resolved task.
|
||||
Audit model-specific frame adjustment, reference loading, latent construction,
|
||||
and decoding for assumptions about the server default. Declare the union of
|
||||
components used by all supported tasks in stage residency metadata. Synthetic
|
||||
warmup uses the default task; provide the pipeline's warmup hook when that task
|
||||
needs model-specific reference inputs. Dynamic batching compares the admitted
|
||||
task along with the other sampling fields.
|
||||
|
||||
`I2I` and `I2V` require image input; `T2I` and `T2V` reject it. Existing `TI2I`
|
||||
and `TI2V` retain optional image conditioning. `V2V` and `F2V` require video
|
||||
input; `F2V` denotes video-prefix conditioning. Models own the details of
|
||||
prefix length and temporal alignment. Pipelines with explicit capabilities
|
||||
reject video input for tasks that do not accept it; legacy singleton pipelines
|
||||
retain their model-owned video checks.
|
||||
|
||||
See [request task selection](/docs/sglang-diffusion/api/openai_api#select-a-request-task)
|
||||
for the HTTP interface. Test task selection, invalid inputs, discovery,
|
||||
serialization, batching, and alternating tasks on one running server.
|
||||
|
||||
Reference in New Issue
Block a user