minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Diffusion] SGLang backend for GLM Image AR. Step 1 - Separate server (#25381)

Browse Source 
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Co-authored-by: yuefeng Wu <33725817+ChefWu551@users.noreply.github.com>
Co-authored-by: wuyuefeng <wuyuefeng@noreply.gitcode.com>
This commit is contained in:
Makcum888e
2026-07-09 15:54:50 +03:00
committed by GitHub
co-authored by yhyang201 Xiaoyu Zhang yuefeng Wu wuyuefeng
parent 6ab7a65d94
commit 7aab39a18b
15 changed files with 2144 additions and 170 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/sglang-diffusion/api/cli.mdx
+22 -19
View File
@@ -74,27 +74,30 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
### Model and runtime
- `--model-path &#123;MODEL&#125;`: model path or Hugging Face model ID
- `--lora-path &#123;PATH&#125;` and `--lora-nickname &#123;NAME&#125;`: load a LoRA adapter
- `--lora-merge-mode &#123;auto|merge|dynamic&#125;`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
- `--num-gpus &#123;N&#125;`: number of GPUs to use
- `--performance-mode &#123;manual|auto|speed|memory&#125;` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
- `--tp-size &#123;N&#125;`: tensor parallelism size, mainly for encoders
- `--sp-degree &#123;N&#125;`: sequence parallelism size
- `--ulysses-degree &#123;N&#125;` and `--ring-degree &#123;N&#125;`: USP parallelism controls
- `--enable-cfg-parallel &#123;true|false&#125;`: enable or explicitly disable CFG parallelism
- `--warmup-mode &#123;off|request|server&#125;`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
- `--attention-backend &#123;BACKEND&#125;`: attention backend for native SGLang and diffusers pipelines
- `--component-attention-backends &#123;MAP&#125;`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
- `--attention-backend-config &#123;CONFIG&#125;`: attention backend configuration
- `--model-path {MODEL}`: model path or Hugging Face model ID
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
- `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
- `--num-gpus {N}`: number of GPUs to use
- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
- `--sp-degree {N}`: sequence parallelism size
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
- `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
- `--attention-backend-config {CONFIG}`: attention backend configuration
- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models
- `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server
- `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server
### Sampling and output
- `--prompt &#123;PROMPT&#125;` and `--negative-prompt &#123;PROMPT&#125;`
- `--image-path &#123;PATH&#125; [&#123;PATH&#125; ...]`: input image(s) for image-to-video or image-to-image generation
- `--num-inference-steps &#123;STEPS&#125;` and `--seed &#123;SEED&#125;`
- `--height &#123;HEIGHT&#125;`, `--width &#123;WIDTH&#125;`, `--num-frames &#123;N&#125;`, `--fps &#123;FPS&#125;`
- `--output-path &#123;PATH&#125;`, `--output-file-name &#123;NAME&#125;`, `--save-output`, `--return-frames`
- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
For frame interpolation and upscaling, see [Post-Processing](./post_processing).
@@ -296,7 +299,7 @@ Use `--backend diffusers` to force vanilla diffusers pipelines when no native SG
</tr>
<tr>
<td><code>--cache-dit-config</code></td>
<td><code>&#123;PATH&#125;</code></td>
<td><code>{PATH}</code></td>
<td>Cache-DiT config for diffusers pipelines</td>
</tr>
</tbody>
docs_new/docs/sglang-diffusion/models_with_ar.mdx
+127
View File
@@ -0,0 +1,127 @@
---
title: "Diffusion models with AR stage like GLM-Image"
---
## Quick Start
Run model with transformers implementation for AR stage (default)
```bash
# Terminal 1 : launch server
sglang serve --model-path zai-org/GLM-Image --port ${PORT}
```
```bash
# Terminal 2 : launch client
curl http://${HOST}:${PORT}/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"prompt": "prompt",
"n": 1,
"size": "widthxheight"
}'
```
Run model with SGLang srt implementation for AR stage (high performance)
```bash
# Terminal 1 : launch server with AR model
sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \
--tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal --port ${AR_PORT}
```
```bash
# Terminal 2 : launch server with Diffusion model
sglang serve --model-path /path/to/zai-org/GLM-Image/ --srt-encoder-url "http://${HOST}:${AR_PORT}"
```
```bash
# Terminal 3 : launch client
curl http://${HOST}:${PORT}/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"prompt": "prompt",
"n": 1,
"size": "widthxheight"
}'
```
## Support matrix
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "10%"}} />
<col style={{width: "45%"}} />
<col style={{width: "45%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Transformers backend</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang backend</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GLM-Image</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>T2I, I2I, V2I</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>T2I</td>
</tr>
</tbody>
</table>
## Deployment Assumptions & Limitations
:::warning
**Network Latency & Timeouts:** In SGLang backend mode, the Diffusion server sends an HTTP request to `--srt-encoder-url` for **every auto-regressive (AR) step**.
- To prevent requests from breaking during long model generations, increase `--srt-encoder-timeout` (e.g., set to 100 seconds).
- To protect the system against temporary network delays or brief drops in connection, use `--srt-encoder-connection-timeout`.
:::
- **Recommended Setup:** Run both servers on the same machine or inside the same fast local network.
- **Cross-Region Warning:** Running the Diffusion server and the AR server in different geographic regions will slow down token generation and heavily reduce performance.
- **Startup Connection Check:** SGLang automatically checks the connection to `--srt-encoder-url` when starting up. The server will stop immediately if the remote AR host is offline.
## Ascend NPU ENV
To run 2 servers on same group of NPU you need to specify env variables
https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref_07_0144.html
Example:
```bash
# Terminal 1 : server with AR model
export HCCL_IF_BASE_PORT=23000
export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199"
export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399"
```
```bash
# Terminal 2 : server with diffusion model
export HCCL_IF_BASE_PORT=24000
export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199"
export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399"
```
## Best practices
GLM-Image example for Ascend A3 2 cards (4 devices)
```bash
# Terminal 1 : server with AR model
export HCCL_IF_BASE_PORT=23000
export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199"
export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399"
sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \
--tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \
--cuda-graph-bs 1 --device npu --attention-backend ascend --disable-fast-image-processor \
--tp-size 4 --port ${PORT} --mem-fraction-static 0.4
```
Second terminal with diffusion server:
```bash
# Terminal 2 : run SGL-Diffusion generate command
export HCCL_IF_BASE_PORT=24000
export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199"
export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399"
SGLANG_CACHE_DIT_FN=2 SGLANG_CACHE_DIT_BN=1 SGLANG_CACHE_DIT_WARMUP=4 SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 SGLANG_CACHE_DIT_TAYLORSEER=true SGLANG_CACHE_DIT_TS_ORDER=2 \
SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path /path/to/zai-org/GLM-Image/ \
--prompt "A curious raccoon" --height 1920 --width 1088 --num-inference-steps 50 --num-gpus 4 \
--sp-degree 4 --srt-encoder-url "http://${HOST}:${PORT}" --warmup
```
Result:
```bash
Warmed-up request processed in 33.82 seconds (with warmup excluded)
```