minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] Retune the Qwen3.8-27B RTX 5090 DFLASH2 cells against 1cf2b8c (#35786)

Browse Source
This commit is contained in:
Jimmy Shong
2026-08-20 21:54:51 -07:00
committed by GitHub
parent 78c964d9d7
commit 3efa057449
2 changed files with 71 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+32 -24
View File
@@ -22,7 +22,11 @@ pip install uv
uv pip install sglang
# For the DFLASH2 cells only — DFlash2 selector support is newer than the
# latest release, so install from source (Method 2) instead of the line above.
# latest release, so build from the commit those cells were validated on
# instead of the line above:
# git clone https://github.com/sgl-project/sglang.git && cd sglang
# git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e # PR #35496
# uv pip install -e "python[all]"
```
Then run the **Python** output of the command panel below in that environment.
@@ -34,9 +38,11 @@ Then run the **Python** output of the command panel below in that environment.
```bash Command
docker pull lmsysorg/sglang:qwen38-27b
# For the DFLASH2 cells only — that tag predates DFlash2 selector support,
# so pull a nightly built from main instead:
# docker pull lmsysorg/sglang:dev
# For the DFLASH2 cells only — that tag predates DFlash2 selector support.
# Build the image from the commit those cells were validated on instead:
# git clone https://github.com/sgl-project/sglang.git && cd sglang
# git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e # PR #35496
# docker build -t sglang:dflash2 -f docker/Dockerfile .
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
@@ -46,14 +52,16 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
</Tabs>
<Warning>
**DFLASH2 needs a build that tracks main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 + NVFP4
— the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer than
the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI release; a
build without #35496 fails on the NVFP4 cells at boot with `requires a dense
FP16/BF16/FP32 target lm_head`. Every other recipe on this page — no
speculation, MTP, DSpark — runs on the pinned tag as written.
**DFLASH2 needs a build from `1cf2b8c` (PR
[#35496](https://github.com/sgl-project/sglang/pull/35496)) or newer.** DFlash2
landed in [#35371](https://github.com/sgl-project/sglang/pull/35371) and its
quantized-`lm_head` path — what the NVFP4 cells need — in #35496, both newer
than the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI
release. A build without #35496 fails on the NVFP4 cells at boot with `requires
a dense FP16/BF16/FP32 target lm_head`. The DFLASH2 pins on this page were
measured on `1cf2b8c` exactly, which is why both install paths above check that
commit out rather than tracking a moving branch. Every other recipe on this
page — no speculation, MTP, DSpark — runs on the pinned tag as written.
</Warning>
</Accordion>
@@ -269,18 +277,18 @@ checkpoint's calibration scales automatically.
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
those cells carry the **Final Verification In Progress**
badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept
3.30 vs 3.09) — the opposite of the EAGLE trade, so measure before assuming.
float32 still fits, but only at `--mem-fraction-static 0.945` with
`--mamba-full-memory-ratio 10` pinned in place of the balanced value: the
L = 9216 ratio leaves the fp32 state pool one slot short at every
serviceable mem-fraction (0.94 allocates four of Low-Latency's five slots;
0.95 OOMs at runtime), and the re-weighted split leaves the Low-Latency KV
pool a single-request envelope (~9.4k tokens) — no headroom for longer
requests or radix reuse. The panel's DFLASH2 option applies these re-pins
automatically.
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 the pins differ per
state dtype, and the panel applies them automatically. bfloat16 serves at
`--mem-fraction-static 0.88` on the balanced ratio (0.90, DSpark's pin, OOMs
on the first request). float32 reaches only the High-Throughput tier, at
`--mem-fraction-static 0.895` with `--mamba-full-memory-ratio 10` overriding
the balanced value — these cells pin `--max-running-requests 1`, so the
balanced ratio provisions KV for concurrency the recipe never uses and
starves the state pool of the slots fp32 needs. The Low-Latency tier is
greyed out for float32: it needs five fp32 slots plus a full request's KV,
and no mem-fraction holds both — buying the fifth slot cuts KV to 7,752
tokens against the 9,216 one 8192/1024 request needs, while every
mem-fraction with a large enough pool dies in prefill graph capture.
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per