docs(cookbook): add Qwen3.8-27B DGX Spark configs (#35121)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
816ea65058
commit
b956e916ae
@@ -59,7 +59,10 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
|
|||||||
<Note>
|
<Note>
|
||||||
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
|
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
|
||||||
Decoding / Serving Strategy / SSM dtype combination — were validated at
|
Decoding / Serving Strategy / SSM dtype combination — were validated at
|
||||||
ISL 8192 / OSL 1024, concurrency 1. The other platforms' recipes carry their
|
ISL 8192 / OSL 1024, concurrency 1. The DGX Spark cells cover that same full
|
||||||
|
combination set, but to a weaker standard: each was confirmed to **boot and
|
||||||
|
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or
|
||||||
|
acceptance-length numbers taken. The remaining platforms' recipes carry their
|
||||||
original validation, which covers the default overlay picks (plus MTP on
|
original validation, which covers the default overlay picks (plus MTP on
|
||||||
GB300); non-default overlay picks there are valid but unmeasured.
|
GB300); non-default overlay picks there are valid but unmeasured.
|
||||||
</Note>
|
</Note>
|
||||||
@@ -179,9 +182,17 @@ checkpoint's calibration scales automatically.
|
|||||||
requires a FlashInfer build whose prefill `plan` accepts `uniform_q_len`
|
requires a FlashInfer build whose prefill `plan` accepts `uniform_q_len`
|
||||||
(newer than 0.6.15.post1); otherwise run spec with `--attention-backend triton`.
|
(newer than 0.6.15.post1); otherwise run spec with `--attention-backend triton`.
|
||||||
On DGX Spark the 128GB is unified memory shared with the host CPU, so all
|
On DGX Spark the 128GB is unified memory shared with the host CPU, so all
|
||||||
three checkpoints fit; its cells use 8192-token prefill chunks,
|
three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe verbatim
|
||||||
`--mem-fraction-static 0.95`, and `--disable-prefill-cuda-graph`. The SM121
|
rather than a separate operating point. **Validated on SM121 / aarch64**: all
|
||||||
recipe is not yet validated on that platform.
|
36 configurations (3 checkpoints x Speculative Decoding x Serving Strategy x
|
||||||
|
Mamba SSM Dtype) booted and served on GB10 under `lmsysorg/sglang:qwen38-27b`
|
||||||
|
at ISL 8192 / OSL 1024, concurrency 1. That is boot-and-serve coverage only —
|
||||||
|
no throughput or acceptance-length numbers — and it includes the FlashInfer `plan` /
|
||||||
|
`uniform_q_len` path above, which raised no arity error on that image. Two
|
||||||
|
host quirks when reproducing on GB10: docker GPU access is CDI-only
|
||||||
|
(`--device nvidia.com/gpu=all`, as no `nvidia` runtime is registered), and
|
||||||
|
`nvidia-smi` reports `Not Supported` for memory because it is unified with the
|
||||||
|
CPU — gate a relaunch on `MemAvailable` in `/proc/meminfo` instead.
|
||||||
- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so the
|
- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so the
|
||||||
NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path
|
NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path
|
||||||
and its cell is greyed out. The H200 recipes use 32768-token prefill chunks
|
and its cell is greyed out. The H200 recipes use 32768-token prefill chunks
|
||||||
@@ -238,8 +249,7 @@ checkpoint's calibration scales automatically.
|
|||||||
- `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk
|
- `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk
|
||||||
on hybrid GDN models, and 8192-token chunks stall them ~600ms at a time.
|
on hybrid GDN models, and 8192-token chunks stall them ~600ms at a time.
|
||||||
2048 keeps decode inter-token latency smooth under mixed load and also
|
2048 keeps decode inter-token latency smooth under mixed load and also
|
||||||
improves single-wave TTFT. (DGX Spark is the exception: its cells run
|
improves single-wave TTFT.
|
||||||
8192-token chunks.)
|
|
||||||
|
|
||||||
## 3. Agent Harnesses
|
## 3. Agent Harnesses
|
||||||
|
|
||||||
|
|||||||
@@ -338,8 +338,9 @@ export const config = {
|
|||||||
// Verification: RTX 5090 / RTX PRO 6000 cells were measured across their
|
// Verification: RTX 5090 / RTX PRO 6000 cells were measured across their
|
||||||
// whole overlay envelope; the h200/gb300 badges carry the source page's
|
// whole overlay envelope; the h200/gb300 badges carry the source page's
|
||||||
// validation, which covers the overlay defaults (plus plain MTP on gb300) —
|
// validation, which covers the overlay defaults (plus plain MTP on gb300) —
|
||||||
// non-default overlay picks there are valid but unmeasured. DGX Spark stays
|
// non-default overlay picks there are valid but unmeasured. DGX Spark was
|
||||||
// unverified (SM121 / aarch64 unvalidated).
|
// measured across its whole overlay envelope too, but to a weaker standard
|
||||||
|
// (boot-and-serve only — see the cell block comment below).
|
||||||
//
|
//
|
||||||
// Cells carry NO --mamba-full-memory-ratio: the ratio depends on workload,
|
// Cells carry NO --mamba-full-memory-ratio: the ratio depends on workload,
|
||||||
// S, D and kv_bytes_per_token, so the page's calculator computes it live
|
// S, D and kv_bytes_per_token, so the page's calculator computes it live
|
||||||
@@ -480,21 +481,30 @@ export const config = {
|
|||||||
},
|
},
|
||||||
// DGX Spark (GB10, SM121): single node, 128GB coherent unified memory
|
// DGX Spark (GB10, SM121): single node, 128GB coherent unified memory
|
||||||
// shared with the CPU — every checkpoint fits, so all three quants get a
|
// shared with the CPU — every checkpoint fits, so all three quants get a
|
||||||
// cell. FlashInfer attention comes from the SM120 pair; the platform gets
|
// cell. These cells reuse the RTX PRO 6000 recipe verbatim rather than a
|
||||||
// its own operating point at 8192-token prefill chunks, 0.95 static
|
// separate SM121 operating point: both cards are SM12x Blackwell, and
|
||||||
// fraction, and prefill CUDA graphs disabled. Unvalidated on SM121 /
|
// GB10's 128GB unified pool is larger than the 6000's 96GB, so a recipe
|
||||||
// aarch64.
|
// that fits the smaller card has headroom here.
|
||||||
|
//
|
||||||
|
// Validated on GB10 (SM121 / aarch64): all 36 configurations booted and
|
||||||
|
// served at ISL 8192 / OSL 1024, concurrency 1. Boot-and-serve only -- no
|
||||||
|
// throughput or acceptance-length numbers were taken, so this is a weaker
|
||||||
|
// standard than the SM120 pair's validation, and the Deploy-panel Note says
|
||||||
|
// so.
|
||||||
{
|
{
|
||||||
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", nodes: "single" },
|
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", nodes: "single" },
|
||||||
|
// All 12 overlay combinations served on GB10. DSPARK here also
|
||||||
|
// exercises the 4-bit `lm_head` this checkpoint quantizes, with no shape
|
||||||
|
// error.
|
||||||
|
verified: true,
|
||||||
env: [],
|
env: [],
|
||||||
flags: [
|
flags: [
|
||||||
"--trust-remote-code",
|
"--trust-remote-code",
|
||||||
"--model-path {{MODEL_NAME}}",
|
"--model-path {{MODEL_NAME}}",
|
||||||
"--kv-cache-dtype fp8_e4m3",
|
"--kv-cache-dtype fp8_e4m3",
|
||||||
"--mem-fraction-static 0.95",
|
"--mem-fraction-static 0.85",
|
||||||
"--attention-backend flashinfer",
|
"--attention-backend flashinfer",
|
||||||
"--chunked-prefill-size 8192",
|
"--chunked-prefill-size 2048",
|
||||||
"--disable-prefill-cuda-graph",
|
|
||||||
"--reasoning-parser qwen3",
|
"--reasoning-parser qwen3",
|
||||||
"--tool-call-parser qwen3_coder",
|
"--tool-call-parser qwen3_coder",
|
||||||
"--host {{HOST_IP}}",
|
"--host {{HOST_IP}}",
|
||||||
@@ -503,15 +513,16 @@ export const config = {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
match: { hw: "dgx-spark", variant: "default", quant: "fp8", nodes: "single" },
|
match: { hw: "dgx-spark", variant: "default", quant: "fp8", nodes: "single" },
|
||||||
|
// All 12 overlay combinations served on GB10.
|
||||||
|
verified: true,
|
||||||
env: [],
|
env: [],
|
||||||
flags: [
|
flags: [
|
||||||
"--trust-remote-code",
|
"--trust-remote-code",
|
||||||
"--model-path {{MODEL_NAME}}",
|
"--model-path {{MODEL_NAME}}",
|
||||||
"--kv-cache-dtype fp8_e4m3",
|
"--kv-cache-dtype fp8_e4m3",
|
||||||
"--mem-fraction-static 0.95",
|
"--mem-fraction-static 0.85",
|
||||||
"--attention-backend flashinfer",
|
"--attention-backend flashinfer",
|
||||||
"--chunked-prefill-size 8192",
|
"--chunked-prefill-size 2048",
|
||||||
"--disable-prefill-cuda-graph",
|
|
||||||
"--reasoning-parser qwen3",
|
"--reasoning-parser qwen3",
|
||||||
"--tool-call-parser qwen3_coder",
|
"--tool-call-parser qwen3_coder",
|
||||||
"--host {{HOST_IP}}",
|
"--host {{HOST_IP}}",
|
||||||
@@ -520,15 +531,17 @@ export const config = {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
match: { hw: "dgx-spark", variant: "default", quant: "bf16", nodes: "single" },
|
match: { hw: "dgx-spark", variant: "default", quant: "bf16", nodes: "single" },
|
||||||
|
// All 12 overlay combinations served on GB10. Heaviest checkpoint, so
|
||||||
|
// it holds the sweep's tightest cell: DSPARK + float32 + extra_buffer.
|
||||||
|
verified: true,
|
||||||
env: [],
|
env: [],
|
||||||
flags: [
|
flags: [
|
||||||
"--trust-remote-code",
|
"--trust-remote-code",
|
||||||
"--model-path {{MODEL_NAME}}",
|
"--model-path {{MODEL_NAME}}",
|
||||||
"--kv-cache-dtype fp8_e4m3",
|
"--kv-cache-dtype fp8_e4m3",
|
||||||
"--mem-fraction-static 0.95",
|
"--mem-fraction-static 0.85",
|
||||||
"--attention-backend flashinfer",
|
"--attention-backend flashinfer",
|
||||||
"--chunked-prefill-size 8192",
|
"--chunked-prefill-size 2048",
|
||||||
"--disable-prefill-cuda-graph",
|
|
||||||
"--reasoning-parser qwen3",
|
"--reasoning-parser qwen3",
|
||||||
"--tool-call-parser qwen3_coder",
|
"--tool-call-parser qwen3_coder",
|
||||||
"--host {{HOST_IP}}",
|
"--host {{HOST_IP}}",
|
||||||
|
|||||||
Reference in New Issue
Block a user