docs(cookbook): add Qwen3.8 GB300 long-context recipe (#43159)
Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
This commit is contained in:
@@ -218,3 +218,45 @@ Pending update — a sample transcript will be added here once the weights are p
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 2.3 GB300 long-context serving with HiCache and MTP
|
||||
|
||||
Use this single-GPU, aggregated TP1/EP1 configuration for long-context, multi-turn workloads on GB300. It combines the NVIDIA NVFP4 checkpoint's embedded MTP head, CPU DRAM KV offload through HiCache, and pinned-memory PLE embedding offload.
|
||||
|
||||
This serving configuration comes from [InferenceX #3816](https://github.com/SemiAnalysisAI/InferenceX/pull/3816). [Run 37714835620, attempt 1](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37714835620) passed concurrency 1, 2, 4, 8, 16, 32, 48 and 64 on commit `e67ca6fc9ecf4cd57f27654bda8a3e9059923820`. Its [matching GSM8K eval](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37714835620/job/113109083376) used real verification and scored 97.35% on 1,319 questions.
|
||||
|
||||
The throughput sweep used InferenceX's simulated acceptance; the generated command uses normal MTP verification and does not claim those throughput numbers for production traffic.
|
||||
|
||||
#### Prerequisites and memory budget
|
||||
|
||||
- Use a Linux ARM64 GB300 host with Docker, NVIDIA Container Toolkit, and a driver compatible with the image's CUDA 13.0 runtime. See [Docker installation](/docs/get-started/install#method-3-using-docker).
|
||||
- Reserve one GB300 GPU. The recipe keeps TP, PP, DP and EP at 1.
|
||||
- Budget 224 GiB of CPU memory for the combined target/draft KV and Mamba HiCache host pool, plus approximately 48 GiB for the offloaded FP8 PLE table and additional headroom for checkpoint loading and host processes. `--hicache-size` sets the combined cache budget; it is separate from PLE weight offload. Keep sufficient free host memory for both.
|
||||
- The Docker command mounts your Hugging Face cache to retain downloaded model files across restarts. The image digest and both target/draft checkpoint revisions are pinned to the validated configuration.
|
||||
|
||||
#### Generate the launch command
|
||||
|
||||
In the [Deployment panel](#deployment), select **GB300 → NVFP4 (NVDA) → Low Latency → Single Node**, and leave **PLE Offload** at **Auto** or **On**. The matched cell includes all five serving environment variables and the full argument set, including ReplaySSM, HiCache and pinned PLE offload. Choose **Docker** to use the pinned image, or **Python** for a matching local installation. Changing PLE Offload to **Off** produces an unverified configuration.
|
||||
|
||||
The Docker wrapper comes from the shared command builder and is adapted from the validated Slurm/Enroot launch; it has not been run separately on a GB300 host. Use the command generated by the panel rather than a separate static copy.
|
||||
|
||||
<Accordion title="Reference startup output">
|
||||
|
||||
The validated run loaded the target and embedded draft through the stock image's mixed-precision loader. This excerpt is from the real-verification eval; timings and memory availability vary by host.
|
||||
|
||||
```text Output
|
||||
[2026-10-08 02:22:36] Load weight end. elapsed=1011.56 s, type=Qwen4ExpForConditionalGeneration, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=202.03 GB, mem usage=73.64 GB.
|
||||
[2026-10-08 02:23:03] Load weight end. elapsed=25.97 s, type=Qwen4ExpForCausalLMMTP, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=197.09 GB, mem usage=4.89 GB.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
#### Serving choices
|
||||
|
||||
- **Draft precision:** revision `fc694b54` stores the MTP routed experts in block-scaled FP8 and the rest of the head in BF16. The pinned image, built from SGLang `e00930c5`, preserves this per-layer `modelopt_mixed` configuration. The draft uses the same checkpoint revision as the target. `SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0` disables the optional draft precision conversion path; target and draft KV caches stay in BF16.
|
||||
- **Hybrid attention:** the recipe selects FlashInfer for linear-attention prefill/decode, Triton for linear verification, and `trtllm_mha` as the attention backend. ReplaySSM speculation, `extra_buffer_lazy`, a tracking interval of 256 and 480 Mamba cache slots retain the qualified state-cache configuration.
|
||||
- **Admission and graphs:** the server admits up to 96 requests, uses 16,384-token prefill chunks, and captures full decode graphs at the listed batch sizes up to 96. Prefill graphs remain disabled. The qualification varied client concurrency while retaining these server settings.
|
||||
- **CPU offload:** HiCache uses the `kernel` I/O backend, `page_first` layout and `write_back` policy. PLE uses a separate pinned-memory embedding table. See [HiCache design](/docs/advanced_features/hicache_design) for the cache tiers and policies.
|
||||
- **API and telemetry:** send requests through the chat-completions API at `http://localhost:30000/v1`, using model name `nvidia/Qwen3.8-Flash-Next-NVFP4`. `--reasoning-parser auto` separates reasoning from answer content. Engine metrics and cache reporting remain enabled.
|
||||
|
||||
The generated command omits arguments that match the pinned image's defaults, including TP/PP/DP/EP=1, the inherited served-model name and draft MoE backend, full decode graphs, and default Mamba tracking, HiCache layout, PLE backend, tokenizer and scheduler settings. It retains the qualified lane's non-default serving settings. InferenceX-only trace loading, simulated-acceptance controls, client concurrency overrides, result collection and Slurm allocation settings are excluded.
|
||||
|
||||
@@ -103,16 +103,22 @@ export const config = {
|
||||
disableReason: (sel) => sel.hw === "rtx6000"
|
||||
? "RTX PRO 6000 (96 GB) only fits this checkpoint with the 47.7 GiB FP8 N-gram table in pinned host RAM; the verified cells pass --ple-offload-embedding explicitly, so On is the only pick."
|
||||
: "DGX Spark is unified memory: PLE offload to RAM frees nothing (the pinned table shares the 128 GB pool with the weights). The verified settings are Off for the two-node cells and On (NVMe file) for a single Spark.",
|
||||
hints: ["PLE Offload: auto-enabled for BF16 on CUDA, off otherwise"] },
|
||||
hints: (sel) => sel.hw === "gb300" && sel.quant === "nvfp4-nvda" && sel.strategy === "low-latency"
|
||||
? ["PLE Offload: enabled in the GB300 base recipe; reserve approximately 48 GiB of host RAM in addition to the 224 GiB HiCache pool."]
|
||||
: ["PLE Offload: auto-enabled for BF16 on CUDA, off otherwise"] },
|
||||
{ id: "on", label: "On",
|
||||
disabled: (sel) => sel.hw === "dgx-spark",
|
||||
disableReason: "DGX Spark is unified memory: PLE offload to RAM frees nothing (the pinned table shares the 128 GB pool with the weights). The verified settings are Off for the two-node cells and On (NVMe file) for a single Spark.",
|
||||
stripPrefixes: (sel) => sel.hw === "gb300" && sel.quant === "nvfp4-nvda" && sel.strategy === "low-latency"
|
||||
? ["--ple-offload-embedding"] : [],
|
||||
flags: ["--ple-offload-embedding"] },
|
||||
{ id: "off", label: "Off",
|
||||
disabled: (sel) => sel.hw === "rtx6000" || (sel.hw === "dgx-spark" && sel.nodes === "single"),
|
||||
disableReason: (sel) => sel.hw === "dgx-spark"
|
||||
? "A single DGX Spark cannot hold the 126 GiB checkpoint in its 128 GB of unified memory; the verified single-Spark cells keep the 47.7 GiB FP8 N-gram table in a file on the local NVMe (On (NVMe file))."
|
||||
: "RTX PRO 6000 (96 GB) cannot hold the 47.7 GiB FP8 N-gram table alongside the other 78 GiB of the checkpoint; the table must be offloaded to pinned host RAM (On).",
|
||||
stripPrefixes: (sel) => sel.hw === "gb300" && sel.quant === "nvfp4-nvda" && sel.strategy === "low-latency"
|
||||
? ["--ple-offload-embedding", "--ple-offload-backend"] : [],
|
||||
flags: ["--no-ple-offload-embedding"] },
|
||||
// File-backed table (sgl-project/sglang#37068, merged into qwen4-main-squashed):
|
||||
// a sparse 47.7 GiB file under $SGLANG_CACHE_DIR/ple/<model> (override
|
||||
@@ -206,6 +212,7 @@ export const config = {
|
||||
"b300|nvfp4-nvda": "lmsysorg/sglang:latest",
|
||||
gb300: "lmsysorg/sglang:qwen38flashnext",
|
||||
"gb300|nvfp4-nvda": "lmsysorg/sglang:latest",
|
||||
"gb300|nvfp4-nvda|low-latency": "lmsysorg/sglang:v0.5.21-cu130@sha256:1f176b3a1dd1205d2a462ab80ea9fc17769a32e53cb6f3c690efa2dc2c384977",
|
||||
mi350x: "lmsysorg/sglang-rocm:qwen38flashnext",
|
||||
mi355x: "lmsysorg/sglang-rocm:qwen38flashnext",
|
||||
},
|
||||
@@ -803,20 +810,60 @@ export const config = {
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "default", quant: "nvfp4-nvda", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
warn: "Requires SGLang v0.5.20 or later for this NVIDIA ModelOpt MIXED_PRECISION export. Quantization and MoE backends are selected automatically from the checkpoint.",
|
||||
env: [],
|
||||
verified: true,
|
||||
verificationStatus: (sel) => sel.pleOffload === "off" ? "unverified" : "verified",
|
||||
warn: "Long-context recipe: reserve 224 GiB of host RAM for HiCache, approximately 48 GiB for pinned PLE offload, and loading headroom. See [GB300 configuration notes](#23-gb300-long-context-serving-with-hicache-and-mtp). The Docker wrapper is adapted from the qualified Slurm/Enroot launch.",
|
||||
env: [
|
||||
"PYTHONNOUSERSITE=1",
|
||||
"PYTHONUNBUFFERED=1",
|
||||
"NO_COLOR=1",
|
||||
"SGLANG_TIMEOUT_KEEP_ALIVE=1800",
|
||||
"SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0",
|
||||
],
|
||||
// Omit defaults from the pinned v0.5.21 build: TP/PP/DP/EP=1,
|
||||
// model-name and draft-MoE inheritance, full decode graphs, and
|
||||
// the standard Mamba, HiCache, PLE, tokenizer and scheduler settings.
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 1",
|
||||
"--moe-runner-backend flashinfer_trtllm",
|
||||
"--disable-shared-experts-fusion",
|
||||
"--revision fc694b54fb0174e0913e6adf86691ef85a4ead47",
|
||||
"--speculative-draft-model-revision fc694b54fb0174e0913e6adf86691ef85a4ead47",
|
||||
"--trust-remote-code",
|
||||
"--quantization modelopt_mixed",
|
||||
"--context-length 262144",
|
||||
"--attention-backend trtllm_mha",
|
||||
"--linear-attn-prefill-backend flashinfer",
|
||||
"--linear-attn-decode-backend flashinfer",
|
||||
"--linear-attn-verify-backend triton",
|
||||
"--mamba-ssm-dtype bfloat16",
|
||||
"--mamba-radix-cache-strategy extra_buffer_lazy",
|
||||
"--max-mamba-cache-size 480",
|
||||
"--mem-fraction-static 0.88",
|
||||
"--max-running-requests 96",
|
||||
"--page-size 64",
|
||||
"--chunked-prefill-size 16384",
|
||||
"--speculative-algorithm NEXTN",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--enable-linear-replayssm-spec",
|
||||
"--cuda-graph-backend-prefill disabled",
|
||||
"--cuda-graph-max-bs-decode 96",
|
||||
"--cuda-graph-bs-decode 1 2 4 8 16 24 32 40 48 56 64 72 80 88 96",
|
||||
"--kv-cache-dtype bfloat16",
|
||||
"--speculative-draft-kv-cache-dtype bfloat16",
|
||||
"--enable-hierarchical-cache",
|
||||
"--hicache-size 224",
|
||||
"--hicache-write-policy write_back",
|
||||
"--ple-offload-embedding",
|
||||
"--enable-metrics",
|
||||
"--enable-cache-report",
|
||||
"--reasoning-parser auto",
|
||||
"--decode-log-interval 10",
|
||||
"--stream-interval 20",
|
||||
"--watchdog-timeout 1800",
|
||||
"--random-seed 35",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user