minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] Re-measure the Qwen3.8-27B RTX 5090, RTX PRO 6000 and DGX Spark grids on 1cf2b8c (#35825)

Browse Source 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jimmy Shong
2026-08-22 13:03:18 +08:00
committed by GitHub
co-authored by Claude Fable 5
parent 96bfd2476c
commit 4cb5aebfe0
2 changed files with 118 additions and 149 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+45 -76
View File
@@ -19,14 +19,10 @@ For all methods and hardware platforms, see the [official SGLang installation gu
```bash Command
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang
# For the DFLASH2 cells only — DFlash2 selector support is newer than the
# latest release, so build from the commit those cells were validated on
# instead of the line above:
# git clone https://github.com/sgl-project/sglang.git && cd sglang
# git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e # PR #35496
# uv pip install --prerelease=allow -e "python[all]"
git clone https://github.com/sgl-project/sglang.git && cd sglang
git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e
uv pip install --prerelease=allow -e "python[all]"
```
Then run the **Python** output of the command panel below in that environment.
@@ -36,13 +32,7 @@ Then run the **Python** output of the command panel below in that environment.
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:qwen38-27b
# For the DFLASH2 cells only — that tag predates DFlash2 selector support.
# Build the image from the commit those cells were validated on instead:
# git clone https://github.com/sgl-project/sglang.git && cd sglang
# git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e # PR #35496
# docker build -t sglang:dflash2 -f docker/Dockerfile .
docker pull lmsysorg/sglang:dev-qwen38-27b-dflash2
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
@@ -51,19 +41,6 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
</Tabs>
<Warning>
**DFLASH2 needs a build from `1cf2b8c` (PR
[#35496](https://github.com/sgl-project/sglang/pull/35496)) or newer.** DFlash2
landed in [#35371](https://github.com/sgl-project/sglang/pull/35371) and its
quantized-`lm_head` path — what the NVFP4 cells need — in #35496, both newer
than the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI
release. A build without #35496 fails on the NVFP4 cells at boot with `requires
a dense FP16/BF16/FP32 target lm_head`. The DFLASH2 pins on this page were
measured on `1cf2b8c` exactly, which is why both install paths above check that
commit out rather than tracking a moving branch. Every other recipe on this
page — no speculation, MTP, DSpark — runs on the pinned tag as written.
</Warning>
</Accordion>
Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point.
@@ -82,34 +59,15 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Deployment config={config} />
<Warning>
**Picking DFLASH2 requires SGLang built from main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 +
NVFP4 — the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer
than the `lmsysorg/sglang:qwen38-27b` tag pinned in
[Install SGLang](#install) and than the latest PyPI release; on NVFP4 a build
without #35496 fails at boot with `requires a dense FP16/BF16/FP32 target
lm_head`. Every other Speculative Decoding pick — None, EAGLE, DSPARK — runs
on the pinned tag as written.
</Warning>
<Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on
NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark
cells cover the same combination set minus DFLASH2, but to a weaker standard:
each was confirmed to **boot and serve** at ISL 8192 / OSL 1024,
concurrency 1, with no throughput or acceptance-length numbers taken. The
H200 and GB300 recipes carry their original validation, which covers the
default overlay picks (plus MTP on GB300); non-default overlay picks there
are valid but unmeasured.
DFLASH2 has not been exercised on H200, DGX Spark or GB300: selecting it
there switches the cell's badge to **Final Verification In Progress** until
that round lands. The other picks on those platforms keep the validation
described above.
cells cover the full combination set — DFLASH2 included — re-measured end to
end on `1cf2b8c`, but to a weaker standard: each of the 48 was confirmed to
**boot and serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput
or acceptance-length numbers taken.
</Note>
### Mamba ratio calculator
@@ -228,17 +186,26 @@ checkpoint's calibration scales automatically.
requires a FlashInfer build whose prefill `plan` accepts `uniform_q_len`
(newer than 0.6.15.post1); otherwise run spec with `--attention-backend triton`.
On DGX Spark the 128GB is unified memory shared with the host CPU, so all
three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe verbatim
rather than a separate operating point. **Validated on SM121 / aarch64**: all
36 configurations (3 checkpoints x Speculative Decoding x Serving Strategy x
Mamba SSM Dtype) booted and served on GB10 under `lmsysorg/sglang:qwen38-27b`
at ISL 8192 / OSL 1024, concurrency 1. That is boot-and-serve coverage only —
no throughput or acceptance-length numbers — and it includes the FlashInfer `plan` /
`uniform_q_len` path above, which raised no arity error on that image. Two
host quirks when reproducing on GB10: docker GPU access is CDI-only
(`--device nvidia.com/gpu=all`, as no `nvidia` runtime is registered), and
`nvidia-smi` reports `Not Supported` for memory because it is unified with the
CPU — gate a relaunch on `MemAvailable` in `/proc/meminfo` instead.
three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe at
`--mem-fraction-static 0.80` rather than a separate operating point. The one
lower pin is the unified pool pricing the host's memory too: 0.85 of 128GB
leaves ~8GB for the OS — exactly DGX OS earlyoom's SIGTERM threshold — and
the first long prefill or boot-time graph capture dips under it and gets the
scheduler killed with `exit code -15` and no traceback (`journalctl -u
earlyoom` shows the kill). At 0.85, 15 of the 48 cells were killed that way,
and which cells is margin noise; at 0.80 every cell served on every attempt.
**Validated on SM121 / aarch64**: all 48 configurations (3 checkpoints x
Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included)
booted and served on GB10 at `1cf2b8c` at ISL 8192 / OSL 1024, concurrency 1.
That is boot-and-serve coverage only — no throughput or acceptance-length
numbers — and it includes the FlashInfer `plan` / `uniform_q_len` path above,
which raised no arity error on that build. Three host quirks when reproducing
on GB10: docker GPU access is CDI-only (`--device nvidia.com/gpu=all`, as no
`nvidia` runtime is registered); `nvidia-smi` reports `Not Supported` for
memory because it is unified with the CPU — gate a relaunch on `MemAvailable`
in `/proc/meminfo` instead; and the BF16 checkpoint takes ~6.5 minutes just
to load its 18 shards from NVMe, so budget ~10 minutes to READY before
calling a boot hung.
- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so the
NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path
and its cell is greyed out. The H200 recipes use 32768-token prefill chunks
@@ -274,21 +241,23 @@ checkpoint's calibration scales automatically.
including quantized heads, so it runs on the NVFP4 checkpoint (whose head
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
those cells carry the **Final Verification In Progress**
badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 the pins differ per
state dtype, and the panel applies them automatically. bfloat16 serves at
`--mem-fraction-static 0.88` on the balanced ratio (0.90, DSpark's pin, OOMs
on the first request). float32 reaches only the High-Throughput tier, at
`--mem-fraction-static 0.895` with `--mamba-full-memory-ratio 10` overriding
the balanced value — these cells pin `--max-running-requests 1`, so the
balanced ratio provisions KV for concurrency the recipe never uses and
starves the state pool of the slots fp32 needs. The Low-Latency tier is
greyed out for float32: it needs five fp32 slots plus a full request's KV,
and no mem-fraction holds both — buying the fifth slot cuts KV to 7,752
tokens against the 9,216 one 8192/1024 request needs, while every
mem-fraction with a large enough pool dies in prefill graph capture.
RTX PRO 6000 BF16/FP8 cells boot and serve; all 12 DGX Spark DFLASH2 cells
boot and serve on `1cf2b8c` with the selector folded into the draft CUDA
graph; on H200 and GB300 those cells carry the
**Final Verification In Progress** badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 every pin is
re-measured against that commit, and the panel applies them automatically:
DFlash2 at `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` —
the only cell on this page needing a smaller prefill chunk, because at 0.91
the pools fit but a 2048-token chunk's activations do not — DSpark at 0.88,
EAGLE at 0.93 (bfloat16) and 0.94 (float32), and no-speculation at 0.90.
float32 is greyed out for both draft-model picks on that card: an fp32 state
slot costs 154 MB against bfloat16's 78 MB, so below ~0.92 the state pool
never reaches the tier's slot count, while at or above it prefill graph
capture or the first request runs out of memory. That was verified across
0.86–0.96 at both chunk sizes and with balanced-ratio overrides up to 20.
bfloat16 is also the faster choice: DFlash2 posts 4.92 ms median TPOT at an
accept length of 4.29, the best result on this card.
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per