diff --git a/docs/cookbook/autoregressive/Tencent/Hy3.mdx b/docs/cookbook/autoregressive/Tencent/Hy3.mdx index 6143d9434..9375aa29c 100644 --- a/docs/cookbook/autoregressive/Tencent/Hy3.mdx +++ b/docs/cookbook/autoregressive/Tencent/Hy3.mdx @@ -1,7 +1,6 @@ --- title: Hy3 description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding." -tag: NEW --- ## Deployment diff --git a/docs/cookbook/autoregressive/Tencent/Hy4-Preview.mdx b/docs/cookbook/autoregressive/Tencent/Hy4-Preview.mdx new file mode 100644 index 000000000..4254cfb1a --- /dev/null +++ b/docs/cookbook/autoregressive/Tencent/Hy4-Preview.mdx @@ -0,0 +1,279 @@ +--- +title: Hy4-Preview +description: "Deploy Tencent Hy4-Preview with SGLang — launch recipes for the 760B-parameter Mixture-of-Experts model with MLA, DeepSeek Sparse Attention (DSA), and MTP speculative decoding, in BF16 on H200/B200/B300/GB300 and MXFP8 on Blackwell GPUs." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](/docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +```bash Command +docker pull lmsysorg/sglang:hy4-preview +``` + + +The `hy4-preview` image bundles the HYV4 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the NEXTN MTP runtime. Switch to `:latest` once a tagged release picks them up. + + +For how to launch the image, see [Install → Method 3: Using Docker](/docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick your hardware + recipe to generate the launch command. The two serving strategies map to whether the MTP (NextN) draft layer is on: + +- **Low-Latency** — MTP speculative decoding on (steps=3, draft-tokens=4). Fastest reply for a single user; pick for chat. +- **High-Throughput** — MTP off. At saturation the draft+verify overhead outweighs the speedup; best for batch jobs. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/tencent/hy4-preview.jsx"; +import { benchmarks } from "/src/snippets/configs/tencent/hy4-preview-benchmarks.jsx"; + + + +
+

Panel controls (top of the command box):

+ +
+ +## Playground + +The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. + +The knobs come in two flavors: + +- **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. +- **Hy4 specific** — `--reasoning-parser auto` / `--tool-call-parser auto` resolve to the suffix-aware `hunyuan` parsers and read the real structural-token strings from the tokenizer vocab at runtime. + +Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the base recipe but stripped by an override. Parallelism combinations beyond the listed recipes (DP-Attention, DeepEP/EP, other TP degrees) are experimentation territory — any override flips the badge to **Not Verified** until the configuration is run end-to-end and submitted back. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +
+

Panel controls reuse Python / Docker · ⧉ Copy · $ cURL · ⚙ Env from the Deploy panel, plus one extra:

+ +
+ +## 1. Model Introduction + +**Hy4-Preview** is Tencent's next-generation flagship Mixture-of-Experts language model: ~760B total parameters with ~40B active per token, pairing a DeepSeek-style MLA + sparse-attention stack with Hunyuan's own residual control, MoE routing, and attention gating. It is a **text-only** model (vocab 120,832; the endpoint rejects image input by design). + +**Key architecture:** + +- **MoE**: 78 layers — layer 0 is a dense MLP, the remaining 77 are sparse MoE with 256 routed experts + 1 shared expert, top-8 sigmoid-scored routing (routed scaling 2.827), expert intermediate size 2048, and bounded SwiGLU (clamp 10.0). +- **MLA + DSA on every layer**: Multi-head Latent Attention (`q_lora_rank` 2048, `kv_lora_rank` 512, 192 nope + 64 rope head dims, `v_head_dim` 256) under DeepSeek Sparse Attention — indexer top-k 2048 with 32 index heads, indexers alternating full/shared (one full indexer every 4 layers), and an FP8 index cache. +- **iHC residual control**: Hunyuan's own inter-layer residual gating (`enable_ihc`, `hc_mult` 4) with pre- and post-residual gate groups. Semantically distinct from DeepSeek-V4's mHC (no combination step, no Sinkhorn) — the implementations are not interchangeable. +- **Gated MLA + attention sink**: element-wise attention output gating evaluated in fp32, plus a learnable per-head attention sink propagated through all attention backends. +- **MTP (NextN)**: one built-in multi-token-prediction draft layer (`model.mtp_layers.0`, present in both checkpoints) enabling speculative decoding out of the box. +- **Long context**: 1M max positions (RoPE theta 1e7). Size `--context-length` to your KV budget — the sizing table in §2 suggests 262,144 (131,072 on H200). + +**Available models:** + +- [tencent/Hy4-preview](https://huggingface.co/tencent/Hy4-preview) — BF16 (~1.5TB weights) +- [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8) — MXFP8 (ModelOpt recipe, UE8M0 group-32 weight scales, dynamic activations; ~760GB weights) + +**Recommended generation:** SGLang applies the checkpoint's `generation_config.json` defaults — don't hardcode sampling parameters in client code. Thinking depth is controlled per request via OpenAI-standard `reasoning_effort` (`none` / `low` / `high`; see §3.1). + +**Special tokens.** The Hy4-Preview tokenizer's structural tokens are suffix-bearing (``, ``, ``, ``, ``). SGLang's `hunyuan` reasoning/tool-call parsers resolve the real token strings from the tokenizer vocab at runtime, so `--reasoning-parser auto --tool-call-parser auto` work out of the box — auto-detection resolves both to the `hunyuan` parsers. + +## 2. Configuration Tips + +**Hardware sizing.** BF16 weights are ~1.5TB and MXFP8 ~760GB; the MLA KV cache (compressed `kv_lora` 512 + rope 64, plus the DSA FP8 indexer cache, ≈95KB/token) is replicated per TP rank, so the per-rank pool left after weights sets the context ceiling: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GPUVRAMMXFP8 (~760GB)BF16 (~1.5TB)
H200141GBNot supported — the MXFP8 kernel path requires SM100 (Blackwell)TP16, 2×8 nodes · 131K context
B200192GBTP8, single node · 262K contextTP16, 2×8 nodes · 262K context (8×192GB ≈ the weights alone)
B300288GBTP4, single node · 262K context (~190GB/rank)TP8, single 8-GPU node · 262K context
GB300288GBTP4, single node · 262K contextTP8, 2×4 nodes · 262K context (GB300 hosts carry 4 GPUs)
+ +On H200 (BF16, TP16 → ~95GB of weights per rank) the per-rank pool holds roughly 260K tokens of KV — set `--context-length 131072` there (the model default is 1M positions, far beyond the pool); the 192GB+ parts comfortably support 262144. + +**DSA attention backend.** Every layer runs DeepSeek Sparse Attention; SGLang auto-selects the DSA backend for HYV4 (`--attention-backend dsa` with `flashmla_sparse` prefill/decode and an FP8 indexer cache over a bf16 KV pool), so the recipes don't pass attention flags. Override only with a kernel-specific reason. + +**MXFP8 kernel stack.** The MXFP8 checkpoint self-describes via its ModelOpt `hf_quant_config` (dynamic activations, UE8M0 group-32 weight scales) — no `--quantization` flag needed; the recipes pin `--fp8-gemm-backend deep_gemm` with the `deep_gemm` MoE runner on B300/GB300 and the `triton` MoE runner on B200. The MXFP8 kernel path requires SM100+ (Blackwell); H200 (SM90) cannot serve the MXFP8 checkpoint — use BF16 there. + +**CUDA graph decode vs eager.** Decode CUDA-graph capture is on by default; long-duration soak validation of the graph path on Hy4 is still in progress. If you hit instability under long mixed agentic workloads, pass `--disable-cuda-graph` to fall back to eager decode (a restart recovers cleanly either way). + +**MTP (NextN) speculative decoding.** Both checkpoints ship one draft layer; the preset is `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-num-draft-tokens 4` (top-k 1). Speculative decoding reserves 4 draft-token slots per request, so the effective request budget is `prompt_tokens + max_tokens + 4 ≤ context length` — requests at the exact context boundary are rejected with the reservation accounted for. + +**Fail-fast guardrails.** The model rejects pipeline parallelism and `--enable-prefill-cp` before allocation. The recipes run pure TP; EP, DP-Attention, and other TP degrees are Playground experimentation territory. + +**Multi-node BF16.** The BF16 weights don't fit a single H200/B200/GB300 host, so those cells are 2-node TP recipes — run the generated command on every node (the panel injects `--nnodes 2 --node-rank --dist-init-addr`) and keep the weights on storage shared across ranks. + +**Text-only.** Image input is rejected with HTTP 400 by design — don't route vision traffic to this endpoint. + +**Large prefills under concurrency.** First-prefill latency on very large prompts can exceed 30 s under high concurrency; use a client timeout of 300 s (and moderate concurrency) for long-context agentic workloads instead of the common 30 s default. + +## 3. Advanced Usage + +### 3.1 Reasoning (`reasoning_effort`) + +Hy4-Preview is a hybrid-thinking model driven by the OpenAI-standard `reasoning_effort` field (`none` / `low` / `high` — note `none`, not `no_think`). Invalid values (including float efforts) are rejected with HTTP 400. The Deploy recipes enable the reasoning parser (`--reasoning-parser auto`) so thinking is separated into `reasoning_content` and the final answer into `content`: + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="tencent/Hy4-preview-FP8", + messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], + reasoning_effort="high", + max_tokens=2048, +) + +msg = response.choices[0].message +print("=============== Thinking =================") +print(msg.reasoning_content) +print("=============== Content =================") +print(msg.content) +``` + + + + + +```text Output +Pending update — will be captured verbatim from a live Hy4-Preview server. +``` + + + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="tencent/Hy4-preview-FP8", + messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], + reasoning_effort="none", + max_tokens=256, +) + +print("Content:", response.choices[0].message.content) +``` + + + + + +```text Output +Pending update — will be captured verbatim from a live Hy4-Preview server. +``` + + + +### 3.2 Tool Calling + +Hy4-Preview emits tool calls through suffixed structural tokens with an `arg_key` / `arg_value` argument format; SGLang's `hunyuan` tool-call parser reassembles them into OpenAI-compatible `message.tool_calls` with schema-aware type coercion, for both streaming and non-streaming requests. The Deploy recipes enable both parsers together (`--reasoning-parser auto --tool-call-parser auto`) — the reasoning parser strips thinking tokens before the tool-call parser runs. + + +Tool-call output is parsed, not grammar-constrained: `tool_choice: "required"` / named-function forcing is not enforced with structural-tag guided decoding on the current implementation. + + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city.", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, + }, + "required": ["city"], + }, + }, + } +] + +response = client.chat.completions.create( + model="tencent/Hy4-preview-FP8", + messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], + tools=tools, +) + +msg = response.choices[0].message +print("Reasoning:", msg.reasoning_content) +print("Content: ", msg.content) +for tc in msg.tool_calls or []: + print(f"Tool Call: {tc.function.name}") + print(f" Arguments: {tc.function.arguments}") +``` + + + + + +```text Output +Pending update — will be captured verbatim from a live Hy4-Preview server. +``` + + diff --git a/docs/cookbook/autoregressive/intro.mdx b/docs/cookbook/autoregressive/intro.mdx index 176293793..69b565570 100644 --- a/docs/cookbook/autoregressive/intro.mdx +++ b/docs/cookbook/autoregressive/intro.mdx @@ -160,7 +160,7 @@ metatags: }` (harness settings in +// the config's benchmarkCommands.accuracy). + +export const benchmarks = [ + // H200 (BF16 only — tested: SM90 cannot serve the MXFP8 checkpoint) + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } }, + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } }, + // B200 + { match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } }, + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } }, + // B300 + { match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } }, + // GB300 + { match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } }, + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } }, +]; diff --git a/docs/src/snippets/configs/tencent/hy4-preview.jsx b/docs/src/snippets/configs/tencent/hy4-preview.jsx new file mode 100644 index 000000000..df877f7f9 --- /dev/null +++ b/docs/src/snippets/configs/tencent/hy4-preview.jsx @@ -0,0 +1,574 @@ +// Hy4-Preview cookbook config. Consumed by _deployment.jsx + _playground.jsx; +// see _deployment.jsx header for the field contract. +// +// Sizing (drives the TP/nodes choices below): +// ~760B total / ~40B active MoE. BF16 weights ≈ 1.5TB → TP16 on H200/B200 +// (2x8 multi-node) or TP8 on B300 (single 8-GPU node) / GB300 (2x4 multi-node +// — GB300 hosts carry 4 GPUs). MXFP8 ≈ 760GB → TP4 on B300/GB300 (288GB), +// TP8 on B200; the MXFP8 kernel path requires SM100+, and H200 (SM90) was +// tested and cannot serve it — H200 gets BF16 cells only. MLA KV (kv_lora +// 512 + rope 64, bf16, replicated per TP rank) plus the DSA FP8 indexer +// cache ≈ 95KB/token/rank; at ~95GB weights/rank (H200 BF16 TP16) the pool +// left is ~25GB/rank ≈ 260K tokens — size `--context-length` to the pool +// (the page's sizing table suggests 131072 there). +// +// Every cell carries `verificationStatus: "in-progress"`. When a recipe's +// end-to-end verification lands, REPLACE that line with `verified: true` — +// `verificationStatus` takes precedence over `verified` in the engine, so +// merely adding `verified: true` would leave the badge amber. + +export const config = { + modelName: "Hy4-Preview", + + supportedHardware: ["h200", "b200", "b300", "gb300"], + + variants: [ + { id: "default", label: "Default" }, + ], + quantizations: [ + { id: "bf16", label: "BF16" }, + { id: "mxfp8", label: "MXFP8" }, + ], + // Two operating points: NEXTN MTP on → low-latency, MTP off → high-throughput. + strategies: [ + { id: "low-latency", label: "Low-Latency" }, + { id: "high-throughput", label: "High-Throughput" }, + ], + nodesOptions: [ + { id: "single", label: "Single Node" }, + { id: "multi-2", label: "Multi-Nodes" }, + ], + + modelNames: { + "default|bf16": "tencent/Hy4-preview", + "default|mxfp8": "tencent/Hy4-preview-FP8", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + NODE0_IP: { target: "command", label: "Head node IP", default: "" }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --flush-cache`, + // GSM8K harness — keep these exact settings for comparability across runs. + accuracy: { + gsm8k_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gsm8k \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --model {{MODEL_NAME}} \\ + --num-examples 1319 --num-threads 64 \\ + --max-tokens 4096 --temperature 0 --top-p 0.95 --seed 0`, + }, + numPromptsByConc: { 1: 32, 16: 32, 64: 128, 256: 512, 1024: 2048 }, + }, + + accuracyLabels: [ + ["gsm8k_pct", "GSM8K", "%"], + ], + + multiNodeHints: { + // GB300 NVL/MNNVL fabric may need manual NIC configuration; NVSHMEM lines + // only matter if you switch the MoE a2a backend to DeepEP in the Playground. + gb300: [ + "The following env vars may be needed depending on your cluster:", + " GLOO_SOCKET_IFNAME=", + " NVSHMEM_ENABLE_NIC_PE_MAPPING=1", + " NVSHMEM_HCA_LIST=", + ], + }, + + dockerImages: { + // The hy4-preview image bundles the HYV4 model code, the suffix-aware + // `hunyuan` parsers, and the NEXTN MTP runtime. Switch to `:latest` once + // a tagged release picks them up. + h200: "lmsysorg/sglang:hy4-preview", + b200: "lmsysorg/sglang:hy4-preview", + b300: "lmsysorg/sglang:hy4-preview", + gb300: "lmsysorg/sglang:hy4-preview", + }, + + github: { + cookbookModel: "tencent/Hy4-preview", + }, + + playgroundFeatures: { + + // ----- Card 1: "Attention Parallelism" ----- + // No CP knob: HYV4ForCausalLM rejects --enable-prefill-cp before + // allocation (and pipeline parallelism likewise raises). TP starts at 4 — + // no listed GPU holds the weights below TP4 (MXFP8 ≈ 760GB, BF16 ≈ + // 1.5TB) — and each degree is gated per hardware/quant so the panel + // never emits a command whose weights don't fit or whose rank count + // exceeds the selected topology (GB300 hosts carry 4 GPUs). + attention: { + knobs: [ + { id: "tp", label: "TP", values: [ + null, + { value: 4, + disable: [ + { when: { quant: ["bf16"] }, + reason: "TP=4 cannot hold the ~1.5TB BF16 weights (~380GB/rank)." }, + { when: { hw: ["h200", "b200"] }, + reason: "TP=4 MXFP8 needs ~190GB/rank — requires 288GB GPUs (B300/GB300)." }, + ] }, + { value: 8, + disable: [ + { when: { hw: ["h200", "b200"], quant: ["bf16"] }, + reason: "TP=8 BF16 needs ~190GB/rank — exceeds H200/B200 VRAM; use TP=16 across 2 nodes." }, + { when: { hw: ["gb300"], nodes: ["single"] }, + reason: "GB300 hosts carry 4 GPUs — TP=8 needs Multi-Nodes (2×4)." }, + ] }, + { value: 16, + disable: [ + { when: { nodes: ["single"] }, + reason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + { when: { hw: ["gb300"] }, + reason: "GB300 hosts carry 4 GPUs — 2 nodes provide only 8 ranks." }, + ] }, + ]}, + { id: "dpAttn", label: "DP-Attention", + values: [ + null, + false, + 4, + { value: 8, + disable: [ + { when: { effTp: [4] }, + reason: "DP-Attention=8 needs TP ≥ 8 (TP must be divisible by the DP degree) — raise TP in this card first." }, + { when: { hw: ["gb300"], nodes: ["single"] }, + reason: "GB300 hosts carry 4 GPUs — 8 attention ranks need Multi-Nodes (2×4)." }, + ] }, + { value: 16, + disable: [ + { when: { effTp: [4, 8] }, + reason: "DP-Attention=16 needs TP=16 — raise TP in this card first." }, + { when: { nodes: ["single"] }, + reason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + { when: { hw: ["gb300"] }, + reason: "GB300 hosts carry 4 GPUs — 2 nodes provide only 8 ranks." }, + ] }, + ], + labels: { "auto": "Auto", "false": "Off" } }, + ], + }, + + // ----- Card 2: "MoE Parallelism" ----- + // 256 routed + 1 shared experts, top-8 sigmoid routing. The recipes run + // the MoE under pure TP (deep_gemm runner on B300/GB300 MXFP8, triton on + // B200); DeepEP/EP are experimentation overrides. No MegaMoE option — + // its fused path is not wired for Hy4's sigmoid-scored, bounded-SwiGLU + // experts. + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + { id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] }, + ], + }, + ep: { label: "EP", values: [ + null, + 4, + { value: 8, + disable: [ + { when: { effTp: [4] }, + reason: "EP=8 needs TP ≥ 8 (TP must be divisible by the EP degree) — raise TP in the Attention card first." }, + { when: { hw: ["gb300"], nodes: ["single"] }, + reason: "GB300 hosts carry 4 GPUs — EP=8 needs Multi-Nodes (2×4)." }, + ] }, + { value: 16, + disable: [ + { when: { effTp: [4, 8] }, + reason: "EP=16 needs TP=16 — raise TP in the Attention card first." }, + { when: { nodes: ["single"] }, + reason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + { when: { hw: ["gb300"] }, + reason: "GB300 hosts carry 4 GPUs — 2 nodes provide only 8 ranks." }, + ] }, + ]}, + }, + + // ----- Card 3: "Parsers" ----- + // Auto-detection resolves to the `hunyuan` reasoning/tool-call parsers; + // the parser reads Hy4's suffix-bearing structural tokens (, + // , /) from the tokenizer + // vocab at runtime. + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser auto" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser auto" }, + ], + }, + + // ----- Card 4: "Speculative Decoding" ----- + // One MTP (NextN) draft layer ships in both checkpoints; the preset is + // steps=3 / top-k 1 / draft-tokens=4. No NGRAM option — untested against + // the DSA sparse-attention backend. + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off (greedy)" }, + { id: "nextn-34", label: "MTP / NextN 3-1-4", + flags: ["--speculative-algorithm NEXTN", "--speculative-num-steps 3", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] }, + ], + }, + + // ----- Card 5: "PD Disaggregation" ----- + // Generic SGLang prefill/decode disaggregation; not yet exercised on + // Hy4 — treat as experimentation. + pdDisagg: { + modes: [ + { id: "off", label: "Off" }, + { id: "prefill", label: "Prefill role" }, + { id: "decode", label: "Decode role" }, + ], + transferBackends: [ + { id: "mooncake", label: "Mooncake", + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True", + "MC_FORCE_MNNVL=1", + ], + envWhen: { hw: ["gb300"] } }, + { id: "nixl", label: "NiXL" }, + ], + // `auto` is a sentinel (emits no --disaggregation-ib-device flag). + ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"], + // Router fronting the prefill + decode roles; substitute /. + router: { + port: 8000, + command: +`python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://:{{PREFILL_PORT}} \\ + --decode http://:{{DECODE_PORT}} \\ + --host 0.0.0.0 --port {{ROUTER_PORT}} \\ + --disable-circuit-breaker \\ + --health-check-interval-secs 999999`, + }, + }, + + // ----- Card 6: "Hierarchical KV Cache" ----- + // Generic SGLang tiers; not yet exercised against Hy4's DSA FP8 indexer + // cache — treat as experimentation. + hicache: { + backends: [ + { id: null, label: "Auto" }, + { id: "file", label: "File" }, + { id: "mooncake", label: "Mooncake" }, + { id: "hf3fs", label: "HF3FS" }, + { id: "nixl", label: "NiXL" }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + { id: "write_through_selective", label: "Write-through (selective)" }, + ], + }, + + // ----- Card 7: "HiSparse" ----- + // DSA-style decode-side hierarchical sparse attention (Hy4's DSA indexer + // top-k is 2048). Shown/emitted only when the live PD-Disagg mode is + // `decode`; not yet exercised on Hy4 — treat as experimentation. + hisparse: { + requiredFlags: ["--disable-radix-cache"], + config: { top_k: 2048, device_buffer_size: 6144 }, + hostRatios: [ + { id: 5, label: "5 (~1TB host)" }, + { id: 10, label: "10 (~2TB host)" }, + ], + defaultHostRatio: 10, + }, + }, + + cells: [ + // NOTE: the engine defaults a hash-less visit to cells[0], so the B300 + // MXFP8 anchor (the recipe that has actually been served) stays first. + // + // The runtime derives the rest from the checkpoint and model defaults: + // quantization comes from the ModelOpt hf_quant_config, the DSA + // sparse-attention backend is auto-selected for HYV4, and decode + // CUDA-graph capture is on by default. + + // ==================================================================== + // B300 (288GB) × MXFP8 — TP4 single node (~190GB weights/rank). + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend deep_gemm", + "--fp8-gemm-backend deep_gemm", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend deep_gemm", + "--fp8-gemm-backend deep_gemm", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B300 (288GB) × BF16 — TP8 single node (~190GB weights/rank on an + // 8-GPU node; same per-rank footprint as MXFP8 TP4). + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // H200 (141GB) × BF16 — TP16 across 2x8 (~95GB weights/rank; the + // ~25GB/rank KV pool ≈ 260K tokens — size --context-length to it). + // No H200 MXFP8 cells: tested — SM90 cannot serve the MXFP8 checkpoint + // (the kernel path requires SM100+). + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B200 (192GB) × MXFP8 — TP8 single node (~95GB weights/rank). B200 + // uses the triton MoE runner (the deep_gemm fused-expert path targets + // the 288GB SM103 parts). + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend triton", + "--fp8-gemm-backend deep_gemm", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend triton", + "--fp8-gemm-backend deep_gemm", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B200 (192GB) × BF16 — TP16 across 2x8 (~95GB weights/rank; 8x192GB + // cannot hold the ~1.5TB weights single-node). + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB300 (288GB, 4-GPU hosts, sm_103 + aarch64) × MXFP8 — TP4 single + // node, same per-rank footprint as B300. + // ==================================================================== + { + match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend deep_gemm", + "--fp8-gemm-backend deep_gemm", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" }, + verificationStatus: "in-progress", + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend deep_gemm", + "--fp8-gemm-backend deep_gemm", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB300 (288GB, 4-GPU hosts) × BF16 — TP8 across 2x4 nodes (~190GB + // weights/rank; a single 4-GPU host cannot hold the ~1.5TB weights). + // NCCL MNNVL env follows the GB-platform multi-node convention. + // ==================================================================== + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verificationStatus: "in-progress", + env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--speculative-algorithm NEXTN", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verificationStatus: "in-progress", + env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};