minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] Tell Qwen3.8-27B DFLASH2 users to build from main (#35753)

Browse Source
This commit is contained in:
Jimmy Shong
2026-08-20 23:34:44 +00:00
committed by GitHub
parent 779e593bd1
commit 1a138e13b9
6 changed files with 122 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+45 -12
View File
@@ -20,6 +20,9 @@ For all methods and hardware platforms, see the [official SGLang installation gu
pip install --upgrade pip
pip install uv
uv pip install sglang
# For the DFLASH2 cells only — DFlash2 selector support is newer than the
# latest release, so install from source (Method 2) instead of the line above.
```
Then run the **Python** output of the command panel below in that environment.
@@ -30,6 +33,10 @@ Then run the **Python** output of the command panel below in that environment.
```bash Command
docker pull lmsysorg/sglang:qwen38-27b
# For the DFLASH2 cells only — that tag predates DFlash2 selector support,
# so pull a nightly built from main instead:
# docker pull lmsysorg/sglang:dev-nightly-0820
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
@@ -38,6 +45,17 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
</Tabs>
<Warning>
**DFLASH2 needs a build that tracks main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 + NVFP4
— the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer than
the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI release; a
build without #35496 fails on the NVFP4 cells at boot with `requires a dense
FP16/BF16/FP32 target lm_head`. Every other recipe on this page — no
speculation, MTP, DSpark — runs on the pinned tag as written.
</Warning>
</Accordion>
Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point.
@@ -56,19 +74,34 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Deployment config={config} />
<Warning>
**Picking DFLASH2 requires SGLang built from main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 +
NVFP4 — the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer
than the `lmsysorg/sglang:qwen38-27b` tag pinned in
[Install SGLang](#install) and than the latest PyPI release; on NVFP4 a build
without #35496 fails at boot with `requires a dense FP16/BF16/FP32 target
lm_head`. Every other Speculative Decoding pick — None, EAGLE, DSPARK — runs
on the pinned tag as written.
</Warning>
<Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on
NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the
remaining platforms the DFLASH2 pick is offered but not yet exercised, and
the composed command carries a `# DFLASH2 on this platform: final
verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus
DFLASH2, but to a weaker standard: each was confirmed to **boot and
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or
acceptance-length numbers taken. The remaining platforms' recipes carry their
original validation, which covers the default overlay picks (plus MTP on
GB300); non-default overlay picks there are valid but unmeasured.
ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on
NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark
cells cover the same combination set minus DFLASH2, but to a weaker standard:
each was confirmed to **boot and serve** at ISL 8192 / OSL 1024,
concurrency 1, with no throughput or acceptance-length numbers taken. The
H200 and GB300 recipes carry their original validation, which covers the
default overlay picks (plus MTP on GB300); non-default overlay picks there
are valid but unmeasured.
DFLASH2 has not been exercised on H200, DGX Spark or GB300: selecting it
there switches the cell's badge to **Final Verification In Progress** until
that round lands. The other picks on those platforms keep the validation
described above.
</Note>
### Mamba ratio calculator
@@ -234,8 +267,8 @@ checkpoint's calibration scales automatically.
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
the pick is offered with a final-verification-in-progress hint line
above the command. The
those cells carry the **Final Verification In Progress**
badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept