From 6f22790943a8a5aefe9a86912d6d2043be8ae444 Mon Sep 17 00:00:00 2001 From: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com> Date: Mon, 6 Jul 2026 13:30:47 +0800 Subject: [PATCH] cookbook: add Hunyuan 3 (Hy3) Day-0 page (#30201) --- .../Tencent/Hunyuan3-Preview.mdx | 1 - .../cookbook/autoregressive/Tencent/Hy3.mdx | 370 ++++++++++++ docs_new/cookbook/autoregressive/intro.mdx | 2 +- docs_new/docs.json | 1 + .../configs/tencent/hy3-benchmarks.jsx | 26 + docs_new/src/snippets/configs/tencent/hy3.jsx | 546 ++++++++++++++++++ 6 files changed, 944 insertions(+), 2 deletions(-) create mode 100644 docs_new/cookbook/autoregressive/Tencent/Hy3.mdx create mode 100644 docs_new/src/snippets/configs/tencent/hy3-benchmarks.jsx create mode 100644 docs_new/src/snippets/configs/tencent/hy3.jsx diff --git a/docs_new/cookbook/autoregressive/Tencent/Hunyuan3-Preview.mdx b/docs_new/cookbook/autoregressive/Tencent/Hunyuan3-Preview.mdx index 0e730954d..e22d2b12e 100644 --- a/docs_new/cookbook/autoregressive/Tencent/Hunyuan3-Preview.mdx +++ b/docs_new/cookbook/autoregressive/Tencent/Hunyuan3-Preview.mdx @@ -2,7 +2,6 @@ title: Hunyuan 3 Preview metatags: description: "Deploy Tencent Hunyuan 3 Preview BF16 (~276B / ~20B active MoE) on NVIDIA GPUs with SGLang — hybrid thinking, native tool calling, 256K context, and built-in MTP speculative decoding." -tag: NEW --- ## 1. Model Introduction diff --git a/docs_new/cookbook/autoregressive/Tencent/Hy3.mdx b/docs_new/cookbook/autoregressive/Tencent/Hy3.mdx new file mode 100644 index 000000000..945fe8bf7 --- /dev/null +++ b/docs_new/cookbook/autoregressive/Tencent/Hy3.mdx @@ -0,0 +1,370 @@ +--- +title: Hy3 +description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +```bash +pip install -U uv +uv venv --python 3.12 && source .venv/bin/activate + +# Install from source (main carries the suffix-aware `hunyuan` parser + the +# HYV3 model code). Once a tagged release picks it up, `uv pip install sglang` +# is enough. +git clone https://github.com/sgl-project/sglang.git +cd sglang +uv pip install -e python +``` + +Run the **Python** output of the command panel below in that environment. + + + + + +```bash Command +# The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser. +docker pull lmsysorg/sglang:dev +``` + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces. + + +The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding. + + + + + + + + +Pick your hardware + recipe to generate the launch command. + +- **Low-Latency** — fastest reply for a single user. Pick for chat. +- **Balanced** — good speed with several users at once. Use for typical multi-user serving. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/tencent/hy3.jsx"; +import { benchmarks } from "/src/snippets/configs/tencent/hy3-benchmarks.jsx"; + + + +
+

Panel controls (top of the command box):

+ +
+ +## Playground + +The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. + +The knobs come in two flavors: + +- **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. +- **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime). + +Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +
+

Panel controls reuse Python / Docker · ⧉ Copy · $ cURL · ⚙ Env from the Deploy panel, plus one extra:

+ +
+ +## 1. Model Introduction + +**Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. + +**Key Features:** + +- **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost. +- **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning. +- **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output. +- **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning. +- **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box. + +**Available Model:** + +- [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct +- [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8 + +**Recommended Generation Parameters:** + + + + + + + + + + + + + + + + + + + + + + +
ParameterValue
temperature0.9
top_p1.0
reasoning_efforthigh / low (thinking) or no_think (instant)
+ +**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `` instead of the bare ``). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model. + +## 2. Configuration Tips + +**Hardware requirements (BF16, ~590GB weights):** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GPUVRAMTPNotes
H200141GB8minimum single-node for BF16
B200192GB4BF16 590GB → 148GB/GPU
B300 / GB300288GB4BF16 590GB → 148GB/GPU; ample KV headroom
GB200192GB4single-node 4×192GB = 768GB fits BF16 590GB
+ +**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason. + +**MTP (Multi-Token Prediction, EAGLE).** + +- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. +- `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient. + +**`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is. + +## 3. Advanced Usage + +### 3.1 Reasoning (Hybrid Thinking) + +Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`: + +- `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content` +- `no_think` — skip thinking entirely (instant responses, content-only) + +Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`: + + + +```bash Command +sglang serve \ + --model-path tencent/Hy3 \ + --tp 8 \ + --reasoning-parser auto \ + --tool-call-parser auto +``` + + + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="tencent/Hy3", + messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], + reasoning_effort="high", + max_tokens=2048, +) + +msg = response.choices[0].message +print("=============== Thinking =================") +print(msg.reasoning_content) +print("=============== Content =================") +print(msg.content) +``` + +```text Output +=============== Thinking ================= +We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240. +10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36. +=============== Content ================= +To find 15% of 240, follow these steps: + +1. 15% = 15/100 or 0.15. +2. Multiply 240 by 0.15: 0.15 × 240 = 36. +3. Check: 10% of 240 = 24, 5% = 12, 15% = 36. + +Thus, 15% of 240 is 36. +``` + + + + + +```python Example +response = client.chat.completions.create( + model="tencent/Hy3", + messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], + reasoning_effort="no_think", + max_tokens=256, +) + +print("Content:", response.choices[0].message.content) +``` + +```text Output +Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe. +``` + + + +### 3.2 Tool Calling + +Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs: + + + +```bash Command +sglang serve \ + --model-path tencent/Hy3 \ + --tp 8 \ + --reasoning-parser auto \ + --tool-call-parser auto +``` + + + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city.", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, + }, + "required": ["city"], + }, + }, + } +] + +response = client.chat.completions.create( + model="tencent/Hy3", + messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], + tools=tools, +) + +msg = response.choices[0].message +print("Reasoning:", msg.reasoning_content) +print("Content: ", msg.content) +for tc in msg.tool_calls or []: + print(f"Tool Call: {tc.function.name}") + print(f" Arguments: {tc.function.arguments}") +``` + +```text Output +Reasoning: None +Content: I'll get the current weather for Beijing in Fahrenheit for you. +Tool Call: get_weather + Arguments: {"city": "Beijing", "unit": "fahrenheit"} +``` + + + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +stream = client.chat.completions.create( + model="tencent/Hy3", + messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], + tools=tools, + stream=True, +) + +tool_buffer = {} +for chunk in stream: + delta = chunk.choices[0].delta + if delta.content: + print(delta.content, end="", flush=True) + for tc in delta.tool_calls or []: + buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""}) + if tc.function and tc.function.name: + buf["name"] += tc.function.name + if tc.function and tc.function.arguments: + buf["args"] += tc.function.arguments + +for idx, buf in tool_buffer.items(): + print(f"\nTool[{idx}] {buf['name']}({buf['args']})") +``` + +```text Output +I'll check the current weather in Beijing for you using Fahrenheit. +Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"}) +``` + + diff --git a/docs_new/cookbook/autoregressive/intro.mdx b/docs_new/cookbook/autoregressive/intro.mdx index e6b6a97c0..03737967d 100644 --- a/docs_new/cookbook/autoregressive/intro.mdx +++ b/docs_new/cookbook/autoregressive/intro.mdx @@ -136,7 +136,7 @@ metatags: ); SGLang's `hunyuan` reasoning/tool-call parsers +// resolve the real token strings from the vocab at runtime (PR #29920), so the +// same recipe serves both the preview (suffix-less) and the shipping (suffixed) +// tokenizer — no per-model hard-coding. +// +// BF16 weights are ~590GB. Single-node TP fits: H200 (141GB, TP8 = 74GB/GPU), +// B200 (180GB, TP4 = 148GB/GPU), B300/GB300 (272GB, TP4), GB200 (192GB, TP4). + +export const config = { + modelName: "Hy3", + + supportedHardware: ["h200", "b200", "b300", "gb200", "gb300"], + + variants: [ + { id: "default", label: "Default" }, + ], + quantizations: [ + { id: "bf16", label: "BF16" }, + { id: "fp8", label: "FP8" }, + ], + strategies: [ + { id: "low-latency", label: "Low-Latency" }, + { id: "balanced", label: "Balanced" }, + ], + nodesOptions: [ + { id: "single", label: "Single Node" }, + { id: "multi-2", label: "Multi-Nodes" }, + ], + + modelNames: { + "default|bf16": "tencent/Hy3", + "default|fp8": "tencent/Hy3-FP8", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + NODE0_IP: { target: "command", label: "Head node IP", default: "" }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --warmup-requests 64`, + accuracy: { + gsm8k_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gsm8k \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --num-threads 32`, + aime26_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run aime26 \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --model {{MODEL_NAME}} --api-key \\ + --n-repeats 1 --max-tokens 28672 \\ + --temperature 0.6 --top-p 0.95 --thinking \\ + --out-dir /sgl-workspace/logs`, + }, + numPromptsByConc: { 1: 32, 16: 32, 64: 128, 256: 512, 1024: 2048 }, + }, + + accuracyLabels: [ + ["gsm8k_pct", "GSM8K (1-shot)", "%"], + ["aime26_pct", "AIME26", "%"], + ], + + multiNodeHints: { + gb200: [ + "The following env vars may be needed depending on your cluster:", + " GLOO_SOCKET_IFNAME=", + " NVSHMEM_ENABLE_NIC_PE_MAPPING=1", + " NVSHMEM_HCA_LIST=", + ], + }, + + dockerImages: { + // The dev image bundles the HYV3 model code + the suffix-aware `hunyuan` + // parser. Switch to `:latest` once a tagged release picks it up. + h200: "lmsysorg/sglang:dev", + b200: "lmsysorg/sglang:dev", + b300: "lmsysorg/sglang:dev", + gb200: "lmsysorg/sglang:dev", + gb300: "lmsysorg/sglang:dev", + }, + + github: { + cookbookModel: "tencent/Hy3", + }, + + playgroundFeatures: { + + // ----- Card 1: "Attention Parallelism" ----- + attention: { + knobs: [ + { id: "tp", label: "TP", values: [ + null, + 1, + 2, + 4, + 8, + { value: 16, disable: { nodes: ["single"] }, + disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + ]}, + { id: "cp", label: "CP", values: [null, 1, 2, 4] }, + { id: "dpAttn", label: "DP-Attention", + values: [ + null, + false, + 1, + 2, + 4, + 8, + { value: 16, disable: { nodes: ["single"] }, + disableReason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + ], + labels: { "auto": "Auto", "false": "Off" } }, + ], + }, + + // ----- Card 2: "MoE Parallelism" ----- + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + { id: "deepep", label: "DeepEP", + flags: ["--moe-a2a-backend deepep"] }, + { id: "megamoe", label: "MegaMoE", + flags: ["--moe-a2a-backend megamoe"], + requiresHw: ["b200", "b300", "gb200", "gb300"] }, + ], + }, + ep: { label: "EP", values: [ + null, + 1, + 2, + 4, + 8, + { value: 16, disable: { nodes: ["single"] }, + disableReason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + ]}, + }, + + // ----- Card 3: "Parsers" ----- + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser auto" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser auto" }, + ], + }, + + // ----- Card 4: "Speculative Decoding" ----- + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off (greedy)" }, + { id: "mtp-314", label: "EAGLE / MTP 3-1-4", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] }, + { id: "mtp-112", label: "EAGLE / MTP 1-1-2", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"] }, + { id: "ngram", label: "NGRAM", + flags: ["--speculative-algorithm NGRAM", + "--speculative-num-draft-tokens 16", + "--speculative-ngram-max-bfs-breadth 10"], + disable: { dpAttnOn: [true] }, + disableReason: "NGRAM is incompatible with DP-Attention. Turn DP-Attention off in the Attention card above to use NGRAM." }, + ], + }, + + // ----- Card 5: "PD Disaggregation" ----- + pdDisagg: { + modes: [ + { id: "off", label: "Off" }, + { id: "prefill", label: "Prefill role" }, + { id: "decode", label: "Decode role" }, + ], + transferBackends: [ + { id: "mooncake", label: "Mooncake", + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True", + "MC_FORCE_MNNVL=1", + ], + envWhen: { hw: ["gb200", "gb300"] } }, + { id: "nixl", label: "NiXL" }, + ], + ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"], + router: { + port: 8000, + command: +`python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://:{{PREFILL_PORT}} \\ + --decode http://:{{DECODE_PORT}} \\ + --policy round_robin \\ + --host 0.0.0.0 --port {{ROUTER_PORT}}`, + }, + }, + + // ----- Card 6: "Hierarchical KV Cache" ----- + hicache: { + backends: [ + { id: "null_placeholder", label: "Auto" }, + { id: "file", label: "File" }, + { id: "mooncake", label: "Mooncake" }, + { id: "hf3fs", label: "HF3FS" }, + { id: "nixl", label: "NiXL" }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + { id: "write_through_selective", label: "Write-through (selective)" }, + ], + }, + }, + + cells: [ + // ==================================================================== + // H200 (141GB) — TP=8 for BF16 (~590GB) + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B200 (180GB) — TP=4 (BF16 590GB → 148GB/GPU, fits with KV headroom) + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B300 (272GB) — TP=4 + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB300 — TP=4 (inferred from B300, same sm_103 + aarch64) + // ==================================================================== + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB200 (sm_100 + aarch64) — TP=4 (single-node 4×192GB = 768GB fits BF16 590GB) + // ==================================================================== + { + match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // FP8 (~300GB) — TP=4 on H200/B200, TP=2 on B300/GB300/GB200 + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 2", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 2", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 2", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 2", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 2", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser auto", + "--tool-call-parser auto", + "--tp 2", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};