[Diffusion] SGLang backend for GLM Image AR. Step 1 - Separate server (#25381)
Co-authored-by: yhyang201 <yhyang201@gmail.com> Co-authored-by: Xiaoyu Zhang <1182563586@qq.com> Co-authored-by: yuefeng Wu <33725817+ChefWu551@users.noreply.github.com> Co-authored-by: wuyuefeng <wuyuefeng@noreply.gitcode.com>
This commit is contained in:
co-authored by
yhyang201
Xiaoyu Zhang
yuefeng Wu
wuyuefeng
parent
6ab7a65d94
commit
7aab39a18b
@@ -74,27 +74,30 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
|
||||
### Model and runtime
|
||||
|
||||
- `--model-path {MODEL}`: model path or Hugging Face model ID
|
||||
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
|
||||
- `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
|
||||
- `--num-gpus {N}`: number of GPUs to use
|
||||
- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
|
||||
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
|
||||
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
|
||||
- `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines
|
||||
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
|
||||
- `--attention-backend-config {CONFIG}`: attention backend configuration
|
||||
- `--model-path {MODEL}`: model path or Hugging Face model ID
|
||||
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
|
||||
- `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
|
||||
- `--num-gpus {N}`: number of GPUs to use
|
||||
- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
|
||||
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
|
||||
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
|
||||
- `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines
|
||||
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
|
||||
- `--attention-backend-config {CONFIG}`: attention backend configuration
|
||||
- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models
|
||||
- `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server
|
||||
- `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server
|
||||
|
||||
### Sampling and output
|
||||
|
||||
- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
|
||||
- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
|
||||
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
|
||||
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
|
||||
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
|
||||
- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
|
||||
- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
|
||||
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
|
||||
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
|
||||
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
|
||||
|
||||
For frame interpolation and upscaling, see [Post-Processing](./post_processing).
|
||||
|
||||
@@ -296,7 +299,7 @@ Use `--backend diffusers` to force vanilla diffusers pipelines when no native SG
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--cache-dit-config</code></td>
|
||||
<td><code>{PATH}</code></td>
|
||||
<td><code>{PATH}</code></td>
|
||||
<td>Cache-DiT config for diffusers pipelines</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
|
||||
Reference in New Issue
Block a user