diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx index 28df6bee3e..9fd95259df 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx @@ -218,3 +218,45 @@ Pending update — a sample transcript will be added here once the weights are p ``` + +### 2.3 GB300 long-context serving with HiCache and MTP + +Use this single-GPU, aggregated TP1/EP1 configuration for long-context, multi-turn workloads on GB300. It combines the NVIDIA NVFP4 checkpoint's embedded MTP head, CPU DRAM KV offload through HiCache, and pinned-memory PLE embedding offload. + +This serving configuration comes from [InferenceX #3816](https://github.com/SemiAnalysisAI/InferenceX/pull/3816). [Run 37714835620, attempt 1](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37714835620) passed concurrency 1, 2, 4, 8, 16, 32, 48 and 64 on commit `e67ca6fc9ecf4cd57f27654bda8a3e9059923820`. Its [matching GSM8K eval](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37714835620/job/113109083376) used real verification and scored 97.35% on 1,319 questions. + +The throughput sweep used InferenceX's simulated acceptance; the generated command uses normal MTP verification and does not claim those throughput numbers for production traffic. + +#### Prerequisites and memory budget + +- Use a Linux ARM64 GB300 host with Docker, NVIDIA Container Toolkit, and a driver compatible with the image's CUDA 13.0 runtime. See [Docker installation](/docs/get-started/install#method-3-using-docker). +- Reserve one GB300 GPU. The recipe keeps TP, PP, DP and EP at 1. +- Budget 224 GiB of CPU memory for the combined target/draft KV and Mamba HiCache host pool, plus approximately 48 GiB for the offloaded FP8 PLE table and additional headroom for checkpoint loading and host processes. `--hicache-size` sets the combined cache budget; it is separate from PLE weight offload. Keep sufficient free host memory for both. +- The Docker command mounts your Hugging Face cache to retain downloaded model files across restarts. The image digest and both target/draft checkpoint revisions are pinned to the validated configuration. + +#### Generate the launch command + +In the [Deployment panel](#deployment), select **GB300 → NVFP4 (NVDA) → Low Latency → Single Node**, and leave **PLE Offload** at **Auto** or **On**. The matched cell includes all five serving environment variables and the full argument set, including ReplaySSM, HiCache and pinned PLE offload. Choose **Docker** to use the pinned image, or **Python** for a matching local installation. Changing PLE Offload to **Off** produces an unverified configuration. + +The Docker wrapper comes from the shared command builder and is adapted from the validated Slurm/Enroot launch; it has not been run separately on a GB300 host. Use the command generated by the panel rather than a separate static copy. + + + +The validated run loaded the target and embedded draft through the stock image's mixed-precision loader. This excerpt is from the real-verification eval; timings and memory availability vary by host. + +```text Output +[2026-10-08 02:22:36] Load weight end. elapsed=1011.56 s, type=Qwen4ExpForConditionalGeneration, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=202.03 GB, mem usage=73.64 GB. +[2026-10-08 02:23:03] Load weight end. elapsed=25.97 s, type=Qwen4ExpForCausalLMMTP, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=197.09 GB, mem usage=4.89 GB. +``` + + + +#### Serving choices + +- **Draft precision:** revision `fc694b54` stores the MTP routed experts in block-scaled FP8 and the rest of the head in BF16. The pinned image, built from SGLang `e00930c5`, preserves this per-layer `modelopt_mixed` configuration. The draft uses the same checkpoint revision as the target. `SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0` disables the optional draft precision conversion path; target and draft KV caches stay in BF16. +- **Hybrid attention:** the recipe selects FlashInfer for linear-attention prefill/decode, Triton for linear verification, and `trtllm_mha` as the attention backend. ReplaySSM speculation, `extra_buffer_lazy`, a tracking interval of 256 and 480 Mamba cache slots retain the qualified state-cache configuration. +- **Admission and graphs:** the server admits up to 96 requests, uses 16,384-token prefill chunks, and captures full decode graphs at the listed batch sizes up to 96. Prefill graphs remain disabled. The qualification varied client concurrency while retaining these server settings. +- **CPU offload:** HiCache uses the `kernel` I/O backend, `page_first` layout and `write_back` policy. PLE uses a separate pinned-memory embedding table. See [HiCache design](/docs/advanced_features/hicache_design) for the cache tiers and policies. +- **API and telemetry:** send requests through the chat-completions API at `http://localhost:30000/v1`, using model name `nvidia/Qwen3.8-Flash-Next-NVFP4`. `--reasoning-parser auto` separates reasoning from answer content. Engine metrics and cache reporting remain enabled. + +The generated command omits arguments that match the pinned image's defaults, including TP/PP/DP/EP=1, the inherited served-model name and draft MoE backend, full decode graphs, and default Mamba tracking, HiCache layout, PLE backend, tokenizer and scheduler settings. It retains the qualified lane's non-default serving settings. InferenceX-only trace loading, simulated-acceptance controls, client concurrency overrides, result collection and Slurm allocation settings are excluded. diff --git a/docs/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx b/docs/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx index 326de3ba68..ea06946634 100644 --- a/docs/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx +++ b/docs/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx @@ -103,16 +103,22 @@ export const config = { disableReason: (sel) => sel.hw === "rtx6000" ? "RTX PRO 6000 (96 GB) only fits this checkpoint with the 47.7 GiB FP8 N-gram table in pinned host RAM; the verified cells pass --ple-offload-embedding explicitly, so On is the only pick." : "DGX Spark is unified memory: PLE offload to RAM frees nothing (the pinned table shares the 128 GB pool with the weights). The verified settings are Off for the two-node cells and On (NVMe file) for a single Spark.", - hints: ["PLE Offload: auto-enabled for BF16 on CUDA, off otherwise"] }, + hints: (sel) => sel.hw === "gb300" && sel.quant === "nvfp4-nvda" && sel.strategy === "low-latency" + ? ["PLE Offload: enabled in the GB300 base recipe; reserve approximately 48 GiB of host RAM in addition to the 224 GiB HiCache pool."] + : ["PLE Offload: auto-enabled for BF16 on CUDA, off otherwise"] }, { id: "on", label: "On", disabled: (sel) => sel.hw === "dgx-spark", disableReason: "DGX Spark is unified memory: PLE offload to RAM frees nothing (the pinned table shares the 128 GB pool with the weights). The verified settings are Off for the two-node cells and On (NVMe file) for a single Spark.", + stripPrefixes: (sel) => sel.hw === "gb300" && sel.quant === "nvfp4-nvda" && sel.strategy === "low-latency" + ? ["--ple-offload-embedding"] : [], flags: ["--ple-offload-embedding"] }, { id: "off", label: "Off", disabled: (sel) => sel.hw === "rtx6000" || (sel.hw === "dgx-spark" && sel.nodes === "single"), disableReason: (sel) => sel.hw === "dgx-spark" ? "A single DGX Spark cannot hold the 126 GiB checkpoint in its 128 GB of unified memory; the verified single-Spark cells keep the 47.7 GiB FP8 N-gram table in a file on the local NVMe (On (NVMe file))." : "RTX PRO 6000 (96 GB) cannot hold the 47.7 GiB FP8 N-gram table alongside the other 78 GiB of the checkpoint; the table must be offloaded to pinned host RAM (On).", + stripPrefixes: (sel) => sel.hw === "gb300" && sel.quant === "nvfp4-nvda" && sel.strategy === "low-latency" + ? ["--ple-offload-embedding", "--ple-offload-backend"] : [], flags: ["--no-ple-offload-embedding"] }, // File-backed table (sgl-project/sglang#37068, merged into qwen4-main-squashed): // a sparse 47.7 GiB file under $SGLANG_CACHE_DIR/ple/ (override @@ -206,6 +212,7 @@ export const config = { "b300|nvfp4-nvda": "lmsysorg/sglang:latest", gb300: "lmsysorg/sglang:qwen38flashnext", "gb300|nvfp4-nvda": "lmsysorg/sglang:latest", + "gb300|nvfp4-nvda|low-latency": "lmsysorg/sglang:v0.5.21-cu130@sha256:1f176b3a1dd1205d2a462ab80ea9fc17769a32e53cb6f3c690efa2dc2c384977", mi350x: "lmsysorg/sglang-rocm:qwen38flashnext", mi355x: "lmsysorg/sglang-rocm:qwen38flashnext", }, @@ -803,20 +810,60 @@ export const config = { }, { match: { hw: "gb300", variant: "default", quant: "nvfp4-nvda", strategy: "low-latency", nodes: "single" }, - verified: false, - warn: "Requires SGLang v0.5.20 or later for this NVIDIA ModelOpt MIXED_PRECISION export. Quantization and MoE backends are selected automatically from the checkpoint.", - env: [], + verified: true, + verificationStatus: (sel) => sel.pleOffload === "off" ? "unverified" : "verified", + warn: "Long-context recipe: reserve 224 GiB of host RAM for HiCache, approximately 48 GiB for pinned PLE offload, and loading headroom. See [GB300 configuration notes](#23-gb300-long-context-serving-with-hicache-and-mtp). The Docker wrapper is adapted from the qualified Slurm/Enroot launch.", + env: [ + "PYTHONNOUSERSITE=1", + "PYTHONUNBUFFERED=1", + "NO_COLOR=1", + "SGLANG_TIMEOUT_KEEP_ALIVE=1800", + "SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0", + ], + // Omit defaults from the pinned v0.5.21 build: TP/PP/DP/EP=1, + // model-name and draft-MoE inheritance, full decode graphs, and + // the standard Mamba, HiCache, PLE, tokenizer and scheduler settings. flags: [ "--model-path {{MODEL_NAME}}", - "--tp 1", + "--moe-runner-backend flashinfer_trtllm", + "--disable-shared-experts-fusion", + "--revision fc694b54fb0174e0913e6adf86691ef85a4ead47", + "--speculative-draft-model-revision fc694b54fb0174e0913e6adf86691ef85a4ead47", + "--trust-remote-code", + "--quantization modelopt_mixed", + "--context-length 262144", + "--attention-backend trtllm_mha", "--linear-attn-prefill-backend flashinfer", "--linear-attn-decode-backend flashinfer", + "--linear-attn-verify-backend triton", "--mamba-ssm-dtype bfloat16", + "--mamba-radix-cache-strategy extra_buffer_lazy", + "--max-mamba-cache-size 480", + "--mem-fraction-static 0.88", + "--max-running-requests 96", + "--page-size 64", + "--chunked-prefill-size 16384", "--speculative-algorithm NEXTN", "--speculative-num-steps 3", "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4", + "--enable-linear-replayssm-spec", + "--cuda-graph-backend-prefill disabled", + "--cuda-graph-max-bs-decode 96", + "--cuda-graph-bs-decode 1 2 4 8 16 24 32 40 48 56 64 72 80 88 96", + "--kv-cache-dtype bfloat16", + "--speculative-draft-kv-cache-dtype bfloat16", + "--enable-hierarchical-cache", + "--hicache-size 224", + "--hicache-write-policy write_back", + "--ple-offload-embedding", + "--enable-metrics", + "--enable-cache-report", "--reasoning-parser auto", + "--decode-log-interval 10", + "--stream-interval 20", + "--watchdog-timeout 1800", + "--random-seed 35", "--host {{HOST_IP}}", "--port {{PORT}}", ],