minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[docs] Add the NVIDIA NVFP4 export to the Qwen3.8-27B cookbook (#38611)

Browse Source 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jiminator <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
This commit is contained in:
zijiexia
2026-09-11 11:58:30 -07:00
committed by GitHub
co-authored by Claude Opus 5 Jiminator Jimmy Shong
parent d7c284b894
commit df6424967a
3 changed files with 283 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+73 -44
View File
@@ -19,9 +19,7 @@ For all methods and hardware platforms, see the [official SGLang installation gu
pip install --upgrade pip
pip install uv
git clone https://github.com/sgl-project/sglang.git && cd sglang
git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e
uv pip install --prerelease=allow -e "python[all]"
uv pip install --prerelease=allow sglang
```
Then run the **Python** output of the command panel below in that environment.
@@ -31,7 +29,7 @@ Then run the **Python** output of the command panel below in that environment.
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:dev-qwen38-27b-dflash2
docker pull lmsysorg/sglang:latest
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
@@ -59,14 +57,12 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Deployment config={config} />
<Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on
NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark
cells cover the full combination set — DFLASH2 included — re-measured end to
end on `1cf2b8c`, but to a weaker standard: each of the 48 was confirmed to
**boot and serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput
or acceptance-length numbers taken.
Every cell above — RTX 5090, RTX PRO 6000 and DGX Spark, across all five
checkpoints and every Speculative Decoding / Serving Strategy / SSM dtype
combination — is measured on **v0.5.19**. That is 202 cells, each one served
and scored on the full 1319-question GSM8K (93.18-95.15%). The serving
envelope behind the pins is ISL 8192 / OSL 1024 at concurrency 1; throughput
and acceptance-length numbers were not re-taken in that sweep.
</Note>
### Mamba ratio calculator
@@ -176,18 +172,49 @@ context from earlier messages.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Same body, `lm_head` left dense in BF16</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead">RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.8-27B-NVFP4 (NVIDIA)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA's ModelOpt export of the same W4A4 body, `lm_head` packed to FP4</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4">nvidia/Qwen3.8-27B-NVFP4</a></td>
</tr>
</tbody>
</table>
The two NVFP4 exports differ only in the `lm_head`: one packs it to FP4, the
other leaves it dense in BF16. The dense head is ~1.7 GB larger on disk and
~3.2 GB larger at runtime, so it is the harder of the two to fit — every
The two RadixArk NVFP4 exports differ only in the `lm_head`: one packs it to
FP4, the other leaves it dense in BF16. The dense head is ~1.7 GB larger on disk
and ~3.2 GB larger at runtime, so it is the harder of the two to fit — every
recipe on this page was measured against it, and the FP4-head cells reuse
those pins unchanged.
Both NVFP4 checkpoints declare `kv_cache_quant_algo: FP8`; SGLang's default
`--kv-cache-dtype auto` honors it, so the KV pool runs in `fp8_e4m3` with the
checkpoint's calibration scales automatically.
NVIDIA's own export is that same W4A4 body with that same FP4 head: identical
quantized-layer map (FP8 attention and GDN projections, NVFP4 MLPs), identical
tensor set, identical 21.9 GB on disk. On GB300, RTX PRO 6000 and DGX Spark its
cells reuse the FP4-head pins unchanged, and both SM12x grids have been
re-measured against this export on v0.5.19: all 16 overlay combinations per
card serve and score 94.01-95.00% (RTX PRO 6000) and 94.16-95.07% (DGX Spark)
on the full 1319-question GSM8K.
The RTX 5090 is measured too — all 15 overlay combinations it offers serve and
score 93.93-94.92% — and every winning launch command there is identical to the
FP4-head export's, which is the strongest form of the claim above. What the
32GB card does need is the draft-model rows pinning their own pools: those
recipes pin `--max-running-requests 1`, but nothing caps the pools to match, so
the KV pool sizes itself for 127,332 tokens against the 9,216 one
8192-in/1024-out request needs, and the engine's default split then leaves the
GDN state pool far short of the slots it needs once the draft model's weights
are counted against `--mem-fraction-static`. The DSPARK row therefore pins
`--max-total-tokens` and a measured `--mamba-full-memory-ratio`, as do DFLASH2
and MTP on the dense-lm_head export at float32 state. Those pins override the
calculator's live value for the selections that carry them. The no-speculation
row needs none of it and runs at the pins shown.
The two RadixArk checkpoints declare `kv_cache_quant_algo: FP8`, so SGLang's
default `--kv-cache-dtype auto` already puts their KV pool in `fp8_e4m3`. The
NVIDIA export ships no `kv_cache_scheme`, so `auto` would leave its pool in
BF16 instead. Every recipe on this page pins `--kv-cache-dtype fp8_e4m3`
explicitly, so all three run the same `fp8_e4m3` pool regardless; the
difference only shows up if you switch the Playground's **KV Cache Precision**
row back to Auto.
## 2. Configuration Tips
@@ -204,25 +231,26 @@ checkpoint's calibration scales automatically.
scheduler killed with `exit code -15` and no traceback (`journalctl -u
earlyoom` shows the kill). At 0.85, 15 of the 48 cells were killed that way,
and which cells is margin noise; at 0.80 every cell served on every attempt.
**Validated on SM121 / aarch64**: all 48 configurations (3 checkpoints x
**Validated on SM121 / aarch64**: all 80 configurations (5 checkpoints x
Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included)
booted and served on GB10 at `1cf2b8c` at ISL 8192 / OSL 1024, concurrency 1.
That is boot-and-serve coverage only — no throughput or acceptance-length
numbers — and it includes the FlashInfer `plan` / `uniform_q_len` path above,
which raised no arity error on that build. Three host quirks when reproducing
served on GB10 on `v0.5.19` at ISL 8192 / OSL 1024, concurrency 1, and each
scored the full 1319-question GSM8K (93.18-95.15%); the float32 and bfloat16
halves ran on two separate GB10 boxes. No throughput or acceptance-length
numbers were re-taken. The sweep exercises the FlashInfer `plan` /
`uniform_q_len` path above, which raised no arity error on that build. Three host quirks when reproducing
on GB10: docker GPU access is CDI-only (`--device nvidia.com/gpu=all`, as no
`nvidia` runtime is registered); `nvidia-smi` reports `Not Supported` for
memory because it is unified with the CPU — gate a relaunch on `MemAvailable`
in `/proc/meminfo` instead; and the BF16 checkpoint takes ~6.5 minutes just
to load its 18 shards from NVMe, so budget ~10 minutes to READY before
calling a boot hung.
- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so the
NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path
and its cell is greyed out. The H200 recipes use 32768-token prefill chunks
(SM90 prefill is fast enough that a big chunk barely stalls decode, unlike
the SM120 guidance below), and the FlashInfer GDN prefill backend engages by
default under them. `--attention-backend fa3` is a valid alternative,
measured slightly faster at bs=1.
- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so an
NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path,
and all three NVFP4 cells are greyed out. The H200 recipes use 32768-token
prefill chunks (SM90 prefill is fast enough that a big chunk barely stalls
decode, unlike the SM120 guidance below), and the FlashInfer GDN prefill
backend engages by default under them. `--attention-backend fa3` is a valid
alternative, measured slightly faster at bs=1.
- **MTP**: `--speculative-algorithm EAGLE --speculative-num-steps 3
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4` uses the
in-checkpoint MTP head. (This recipe was originally documented with `NEXTN`,
@@ -263,23 +291,24 @@ checkpoint's calibration scales automatically.
--mamba-radix-cache-strategy extra_buffer`, and disabled RadixCache for both
baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2
run added the three flags shown above.
Accuracy used zero-shot GSM8K with greedy sampling, `max_new_tokens=2048`,
128 examples, and concurrency levels 1, 2, 4, 8, and 16.
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; all 12 DGX Spark DFLASH2 cells
boot and serve on `1cf2b8c` with the selector folded into the draft CUDA
graph; on H200 and GB300 those cells carry the
**Final Verification In Progress** badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 every pin is
re-measured against that commit, and the panel applies them automatically:
DFlash2 at `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` —
the only cell on this page needing a smaller prefill chunk, because at 0.91
the pools fit but a 2048-token chunk's activations do not — DSpark at 0.88,
EAGLE at 0.93 (bfloat16) and 0.94 (float32), and no-speculation at 0.90.
That comparison's accuracy used zero-shot GSM8K with greedy sampling,
`max_new_tokens=2048`, 128 examples, and concurrency levels 1, 2, 4, 8
and 16 — a different protocol from this page's own sweep below.
Validation: every SM12x cell on this page is measured end to end on
v0.5.19 — 202 cells over the five checkpoints, four speculative options, two
serving tiers and two GDN state dtypes, full 1319-question GSM8K on each,
93.18-95.15%. The RTX PRO 6000 and DGX Spark recipes need no changes. On the
32GB RTX 5090 the panel applies the measured pins automatically: DFlash2 at
`--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` — at 0.91 the
pools fit but a 2048-token chunk's activations do not — DSpark at 0.88
(bfloat16), 0.91 (float32) and 0.92 on the dense-lm_head export, all three
with their pools pinned and the last two also cutting the prefill chunk to
1024 and 512, EAGLE at 0.93 (bfloat16) and 0.94 (float32), and
no-speculation at 0.90.
Whether float32 is available with a draft model depends on the `lm_head`: on
the BF16-head export it is greyed out for both DSpark and DFlash2, since the
dense head's ~3.2 GB leave no fp32 state pool that also clears prefill graph
capture. The FP4-head export frees that headroom back — DSpark serves at 0.89
capture. The FP4-head export frees that headroom back — DSpark serves at 0.91
and DFlash2 High-Throughput at 0.895 with `--mamba-full-memory-ratio 10`
overriding the balanced value — and only DFlash2 Low-Latency stays out of
reach, where five fp32 slots and a full request's KV never coexist. bfloat16