[diffusion] feat: support --served-model-name in sglang serve (#34228)
Co-authored-by: TobyMint <tobymint@users.noreply.github.com> Co-authored-by: Mick <mickjagger19@icloud.com>
This commit is contained in:
co-authored by
TobyMint
Mick
parent
f5f0c3ee7a
commit
d07ac32d05
@@ -75,6 +75,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
### Model and runtime
|
||||
|
||||
- `--model-path {MODEL}`: model path or Hugging Face model ID
|
||||
- `--served-model-name {NAME}`: stable model name exposed by serving APIs. Defaults to `--model-id` when set, otherwise `--model-path`.
|
||||
- `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`. This is a server/load-time choice, unlike a request's `task`.
|
||||
- `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition.
|
||||
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
|
||||
|
||||
@@ -18,6 +18,7 @@ Launch the server using the `sglang serve` command.
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--served-model-name wan-t2v
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
@@ -30,34 +31,67 @@ sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- **--model-path**: Path to the model or model ID.
|
||||
- **--served-model-name**: Stable model name exposed by the serving APIs. It defaults to `--model-id` when set, otherwise `--model-path`.
|
||||
- **--port**: HTTP port to listen on (default: `30000`).
|
||||
|
||||
**Get Model Information**
|
||||
### Served model name
|
||||
|
||||
**Endpoint:** `GET /models`
|
||||
`--served-model-name` separates the public API identity from the checkpoint location. This is useful when replicas use different local mount paths or when a gateway needs one stable model name:
|
||||
|
||||
Returns information about the model served by this server, including model path, task type, pipeline configuration, and precision settings.
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path /models/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--served-model-name wan-t2v \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
`--model-id` is not a free-form deployment alias: it selects the registered model configuration for checkpoints whose local path cannot be identified. `--served-model-name` only controls the name exposed by serving APIs. When both are set, the served name takes precedence for API responses.
|
||||
|
||||
### Discover the served model
|
||||
|
||||
**Endpoint:** `GET /v1/models`
|
||||
|
||||
Returns the public model name together with diffusion-specific runtime information.
|
||||
|
||||
**Curl Example:**
|
||||
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/models"
|
||||
curl -sS "http://localhost:30010/v1/models"
|
||||
```
|
||||
|
||||
**Response Example:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
"task_type": "T2V",
|
||||
"pipeline_name": "wan_pipeline",
|
||||
"pipeline_class": "WanPipeline",
|
||||
"num_gpus": 4,
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16"
|
||||
"object": "list",
|
||||
"data": [
|
||||
{
|
||||
"id": "wan-t2v",
|
||||
"object": "model",
|
||||
"created": 1786348800,
|
||||
"owned_by": "sglang",
|
||||
"root": "wan-t2v",
|
||||
"parent": null,
|
||||
"max_model_len": null,
|
||||
"num_gpus": 4,
|
||||
"task_type": "T2V",
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16",
|
||||
"pipeline_name": "WanPipeline",
|
||||
"pipeline_class": "WanPipeline"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Retrieve the same model by its served name:
|
||||
|
||||
```bash curl
|
||||
curl -sS "http://localhost:30010/v1/models/wan-t2v"
|
||||
```
|
||||
|
||||
`GET /server_info` also reports `served_model_name` for gateway discovery. Video and action responses use this name when the request does not provide a model explicitly.
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
@@ -77,6 +77,22 @@ livenessProbe:
|
||||
See [Health endpoints](/docs/sglang-diffusion/api/cli#health-endpoints) for the
|
||||
status-code contract and warmup-mode behavior.
|
||||
|
||||
## Stable Model Identity
|
||||
|
||||
Use `--served-model-name` when the public model name must remain stable across replicas, hosts, or checkpoint mount paths:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path /mnt/checkpoints/Qwen-Image \
|
||||
--model-id Qwen-Image \
|
||||
--served-model-name image-production \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
The resolved public name follows `--served-model-name`, then `--model-id`, then `--model-path`. `--model-id` remains an internal model registry and configuration-resolution hint; it is not a replacement for a deployment alias. The resolved name is exposed through `/server_info` and `/v1/models` and is used by video and action responses when a request does not supply its own model.
|
||||
|
||||
See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served-model-name) for discovery and retrieval examples.
|
||||
|
||||
## Performance Modes
|
||||
|
||||
`--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.
|
||||
|
||||
Reference in New Issue
Block a user