minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Cookbook] Add the Hy4-Preview model page (Tencent) (#36804)

Browse Source 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-08-27 23:26:13 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent 9fc60a8afd
commit 1948b61ad4
6 changed files with 888 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Tencent/Hy3.mdx
-1
View File
@@ -1,7 +1,6 @@
---
title: Hy3
description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding."
tag: NEW
---
## Deployment
docs/cookbook/autoregressive/Tencent/Hy4-Preview.mdx
+279
View File
@@ -0,0 +1,279 @@
---
title: Hy4-Preview
description: "Deploy Tencent Hy4-Preview with SGLang — launch recipes for the 760B-parameter Mixture-of-Experts model with MLA, DeepSeek Sparse Attention (DSA), and MTP speculative decoding, in BF16 on H200/B200/B300/GB300 and MXFP8 on Blackwell GPUs."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](/docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
<Tabs>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:hy4-preview
```
<Note>
The `hy4-preview` image bundles the HYV4 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the NEXTN MTP runtime. Switch to `:latest` once a tagged release picks them up.
</Note>
For how to launch the image, see [Install → Method 3: Using Docker](/docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick your hardware + recipe to generate the launch command. The two serving strategies map to whether the MTP (NextN) draft layer is on:
- **Low-Latency** — MTP speculative decoding on (steps=3, draft-tokens=4). Fastest reply for a single user; pick for chat.
- **High-Throughput** — MTP off. At saturation the draft+verify overhead outweighs the speedup; best for batch jobs.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/tencent/hy4-preview.jsx";
import { benchmarks } from "/src/snippets/configs/tencent/hy4-preview-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
<li><strong>Badge</strong> — every Hy4-Preview recipe currently shows <strong>In Progress</strong>: the B300 MXFP8 and H200 BF16 anchors have been served end-to-end, the remaining cells are sized from them by VRAM and architecture, and each cell flips to green <strong>Verified</strong> as its end-to-end verification lands.</li>
</ul>
</div>
## Playground
The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.
The knobs come in two flavors:
- **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
- **Hy4 specific** — `--reasoning-parser auto` / `--tool-call-parser auto` resolve to the suffix-aware `hunyuan` parsers and read the real structural-token strings from the tokenizer vocab at runtime.
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the base recipe but stripped by an override. Parallelism combinations beyond the listed recipes (DP-Attention, DeepEP/EP, other TP degrees) are experimentation territory — any override flips the badge to **Not Verified** until the configuration is run end-to-end and submitted back.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
</ul>
</div>
## 1. Model Introduction
**Hy4-Preview** is Tencent's next-generation flagship Mixture-of-Experts language model: ~760B total parameters with ~40B active per token, pairing a DeepSeek-style MLA + sparse-attention stack with Hunyuan's own residual control, MoE routing, and attention gating. It is a **text-only** model (vocab 120,832; the endpoint rejects image input by design).
**Key architecture:**
- **MoE**: 78 layers — layer 0 is a dense MLP, the remaining 77 are sparse MoE with 256 routed experts + 1 shared expert, top-8 sigmoid-scored routing (routed scaling 2.827), expert intermediate size 2048, and bounded SwiGLU (clamp 10.0).
- **MLA + DSA on every layer**: Multi-head Latent Attention (`q_lora_rank` 2048, `kv_lora_rank` 512, 192 nope + 64 rope head dims, `v_head_dim` 256) under DeepSeek Sparse Attention — indexer top-k 2048 with 32 index heads, indexers alternating full/shared (one full indexer every 4 layers), and an FP8 index cache.
- **iHC residual control**: Hunyuan's own inter-layer residual gating (`enable_ihc`, `hc_mult` 4) with pre- and post-residual gate groups. Semantically distinct from DeepSeek-V4's mHC (no combination step, no Sinkhorn) — the implementations are not interchangeable.
- **Gated MLA + attention sink**: element-wise attention output gating evaluated in fp32, plus a learnable per-head attention sink propagated through all attention backends.
- **MTP (NextN)**: one built-in multi-token-prediction draft layer (`model.mtp_layers.0`, present in both checkpoints) enabling speculative decoding out of the box.
- **Long context**: 1M max positions (RoPE theta 1e7). Size `--context-length` to your KV budget — the sizing table in §2 suggests 262,144 (131,072 on H200).
**Available models:**
- [tencent/Hy4-preview](https://huggingface.co/tencent/Hy4-preview) — BF16 (~1.5TB weights)
- [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8) — MXFP8 (ModelOpt recipe, UE8M0 group-32 weight scales, dynamic activations; ~760GB weights)
**Recommended generation:** SGLang applies the checkpoint's `generation_config.json` defaults — don't hardcode sampling parameters in client code. Thinking depth is controlled per request via OpenAI-standard `reasoning_effort` (`none` / `low` / `high`; see §3.1).
**Special tokens.** The Hy4-Preview tokenizer's structural tokens are suffix-bearing (`<think:opensource>`, `<tool_calls:opensource>`, `<tool_call:opensource>`, `<arg_key:opensource>`, `<arg_value:opensource>`). SGLang's `hunyuan` reasoning/tool-call parsers resolve the real token strings from the tokenizer vocab at runtime, so `--reasoning-parser auto --tool-call-parser auto` work out of the box — auto-detection resolves both to the `hunyuan` parsers.
## 2. Configuration Tips
**Hardware sizing.** BF16 weights are ~1.5TB and MXFP8 ~760GB; the MLA KV cache (compressed `kv_lora` 512 + rope 64, plus the DSA FP8 indexer cache, ≈95KB/token) is replicated per TP rank, so the per-rank pool left after weights sets the context ceiling:
<table style={{width: "100%", borderCollapse: "collapse"}}>
<thead>
<tr style={{borderBottom: "2px solid #0052d9"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPU</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>VRAM</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>MXFP8 (~760GB)</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 (~1.5TB)</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Not supported — the MXFP8 kernel path requires SM100 (Blackwell)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>TP16, 2×8 nodes · 131K context</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TP8, single node · 262K context</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>TP16, 2×8 nodes · 262K context (8×192GB ≈ the weights alone)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B300</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>288GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TP4, single node · 262K context (~190GB/rank)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>TP8, single 8-GPU node · 262K context</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>GB300</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>288GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TP4, single node · 262K context</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>TP8, 2×4 nodes · 262K context (GB300 hosts carry 4 GPUs)</td>
</tr>
</tbody>
</table>
On H200 (BF16, TP16 → ~95GB of weights per rank) the per-rank pool holds roughly 260K tokens of KV — set `--context-length 131072` there (the model default is 1M positions, far beyond the pool); the 192GB+ parts comfortably support 262144.
**DSA attention backend.** Every layer runs DeepSeek Sparse Attention; SGLang auto-selects the DSA backend for HYV4 (`--attention-backend dsa` with `flashmla_sparse` prefill/decode and an FP8 indexer cache over a bf16 KV pool), so the recipes don't pass attention flags. Override only with a kernel-specific reason.
**MXFP8 kernel stack.** The MXFP8 checkpoint self-describes via its ModelOpt `hf_quant_config` (dynamic activations, UE8M0 group-32 weight scales) — no `--quantization` flag needed; the recipes pin `--fp8-gemm-backend deep_gemm` with the `deep_gemm` MoE runner on B300/GB300 and the `triton` MoE runner on B200. The MXFP8 kernel path requires SM100+ (Blackwell); H200 (SM90) cannot serve the MXFP8 checkpoint — use BF16 there.
**CUDA graph decode vs eager.** Decode CUDA-graph capture is on by default; long-duration soak validation of the graph path on Hy4 is still in progress. If you hit instability under long mixed agentic workloads, pass `--disable-cuda-graph` to fall back to eager decode (a restart recovers cleanly either way).
**MTP (NextN) speculative decoding.** Both checkpoints ship one draft layer; the preset is `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-num-draft-tokens 4` (top-k 1). Speculative decoding reserves 4 draft-token slots per request, so the effective request budget is `prompt_tokens + max_tokens + 4 ≤ context length` — requests at the exact context boundary are rejected with the reservation accounted for.
**Fail-fast guardrails.** The model rejects pipeline parallelism and `--enable-prefill-cp` before allocation. The recipes run pure TP; EP, DP-Attention, and other TP degrees are Playground experimentation territory.
**Multi-node BF16.** The BF16 weights don't fit a single H200/B200/GB300 host, so those cells are 2-node TP recipes — run the generated command on every node (the panel injects `--nnodes 2 --node-rank --dist-init-addr`) and keep the weights on storage shared across ranks.
**Text-only.** Image input is rejected with HTTP 400 by design — don't route vision traffic to this endpoint.
**Large prefills under concurrency.** First-prefill latency on very large prompts can exceed 30 s under high concurrency; use a client timeout of 300 s (and moderate concurrency) for long-context agentic workloads instead of the common 30 s default.
## 3. Advanced Usage
### 3.1 Reasoning (`reasoning_effort`)
Hy4-Preview is a hybrid-thinking model driven by the OpenAI-standard `reasoning_effort` field (`none` / `low` / `high` — note `none`, not `no_think`). Invalid values (including float efforts) are rejected with HTTP 400. The Deploy recipes enable the reasoning parser (`--reasoning-parser auto`) so thinking is separated into `reasoning_content` and the final answer into `content`:
<Accordion title="Example: thinking (reasoning_effort=high) (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="tencent/Hy4-preview-FP8",
messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
reasoning_effort="high",
max_tokens=2048,
)
msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update — will be captured verbatim from a live Hy4-Preview server.
```
</Accordion>
<Accordion title="Example: instant mode (reasoning_effort=none) (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="tencent/Hy4-preview-FP8",
messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
reasoning_effort="none",
max_tokens=256,
)
print("Content:", response.choices[0].message.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update — will be captured verbatim from a live Hy4-Preview server.
```
</Accordion>
### 3.2 Tool Calling
Hy4-Preview emits tool calls through suffixed structural tokens with an `arg_key` / `arg_value` argument format; SGLang's `hunyuan` tool-call parser reassembles them into OpenAI-compatible `message.tool_calls` with schema-aware type coercion, for both streaming and non-streaming requests. The Deploy recipes enable both parsers together (`--reasoning-parser auto --tool-call-parser auto`) — the reasoning parser strips thinking tokens before the tool-call parser runs.
<Note>
Tool-call output is parsed, not grammar-constrained: `tool_choice: "required"` / named-function forcing is not enforced with structural-tag guided decoding on the current implementation.
</Note>
<Accordion title="Example: non-streaming tool call (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["city"],
},
},
}
]
response = client.chat.completions.create(
model="tencent/Hy4-preview-FP8",
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
tools=tools,
)
msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content: ", msg.content)
for tc in msg.tool_calls or []:
print(f"Tool Call: {tc.function.name}")
print(f" Arguments: {tc.function.arguments}")
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update — will be captured verbatim from a live Hy4-Preview server.
```
</Accordion>
docs/cookbook/autoregressive/intro.mdx
+1 -1
View File
@@ -160,7 +160,7 @@ metatags:
<Card
title="Tencent"
mode="card"
href="/cookbook/autoregressive/Tencent/Hy3"
href="/cookbook/autoregressive/Tencent/Hy4-Preview"
img="/cards/logos/tencent.png"
/>
<Card
docs/docs.json
+1
View File
@@ -1426,6 +1426,7 @@
{
"group": "Tencent",
"pages": [
"cookbook/autoregressive/Tencent/Hy4-Preview",
"cookbook/autoregressive/Tencent/Hy3",
"cookbook/autoregressive/Tencent/Hunyuan3-Preview"
]
docs/src/snippets/configs/tencent/hy4-preview-benchmarks.jsx
+33
View File
@@ -0,0 +1,33 @@
// Hy4-Preview benchmark data — one entry per config cell `match` tuple.
//
// All entries are bare-match stubs (the card shows "pending"). When a cell's
// numbers land, fill the entry with the measured data and set
// `sglang_version` to the exact commit/tag they were measured on (a
// reproducible anchor — never a moving ref like "main").
//
// Speed shape when filling:
// speed: [{ workload: {dataset, isl, osl, max_concurrency}, ttft_ms,
// tpot_ms, tokens_per_sec_per_gpu }, ...] // P50 latencies
// Accuracy: per-cell `accuracy: { gsm8k_pct: <pct> }` (harness settings in
// the config's benchmarkCommands.accuracy).
export const benchmarks = [
// H200 (BF16 only — tested: SM90 cannot serve the MXFP8 checkpoint)
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
// B200
{ match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" } },
{ match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" } },
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
// B300
{ match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" } },
{ match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" } },
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } },
// GB300
{ match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" } },
{ match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" } },
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
];
docs/src/snippets/configs/tencent/hy4-preview.jsx
+574
View File
@@ -0,0 +1,574 @@
// Hy4-Preview cookbook config. Consumed by _deployment.jsx + _playground.jsx;
// see _deployment.jsx header for the field contract.
//
// Sizing (drives the TP/nodes choices below):
// ~760B total / ~40B active MoE. BF16 weights ≈ 1.5TB → TP16 on H200/B200
// (2x8 multi-node) or TP8 on B300 (single 8-GPU node) / GB300 (2x4 multi-node
// — GB300 hosts carry 4 GPUs). MXFP8 ≈ 760GB → TP4 on B300/GB300 (288GB),
// TP8 on B200; the MXFP8 kernel path requires SM100+, and H200 (SM90) was
// tested and cannot serve it — H200 gets BF16 cells only. MLA KV (kv_lora
// 512 + rope 64, bf16, replicated per TP rank) plus the DSA FP8 indexer
// cache ≈ 95KB/token/rank; at ~95GB weights/rank (H200 BF16 TP16) the pool
// left is ~25GB/rank ≈ 260K tokens — size `--context-length` to the pool
// (the page's sizing table suggests 131072 there).
//
// Every cell carries `verificationStatus: "in-progress"`. When a recipe's
// end-to-end verification lands, REPLACE that line with `verified: true` —
// `verificationStatus` takes precedence over `verified` in the engine, so
// merely adding `verified: true` would leave the badge amber.
export const config = {
modelName: "Hy4-Preview",
supportedHardware: ["h200", "b200", "b300", "gb300"],
variants: [
{ id: "default", label: "Default" },
],
quantizations: [
{ id: "bf16", label: "BF16" },
{ id: "mxfp8", label: "MXFP8" },
],
// Two operating points: NEXTN MTP on → low-latency, MTP off → high-throughput.
strategies: [
{ id: "low-latency", label: "Low-Latency" },
{ id: "high-throughput", label: "High-Throughput" },
],
nodesOptions: [
{ id: "single", label: "Single Node" },
{ id: "multi-2", label: "Multi-Nodes" },
],
modelNames: {
"default|bf16": "tencent/Hy4-preview",
"default|mxfp8": "tencent/Hy4-preview-FP8",
},
placeholders: {
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
PORT: { target: "command", label: "Bind port", default: "30000" },
NODE0_IP: { target: "command", label: "Head node IP", default: "<node0-ip>" },
NODE_RANK: { target: "command", label: "This node rank", default: "<node-rank>" },
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
},
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
-H 'Content-Type: application/json' \\
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
benchmarkCommands: {
speed:
`python3 -m sglang.bench_serving \\
--backend sglang \\
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
--model {{MODEL_NAME}} \\
--dataset-name {{DATASET}} \\
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\
--flush-cache`,
// GSM8K harness — keep these exact settings for comparability across runs.
accuracy: {
gsm8k_pct:
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
sgl-eval run gsm8k \\
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
--model {{MODEL_NAME}} \\
--num-examples 1319 --num-threads 64 \\
--max-tokens 4096 --temperature 0 --top-p 0.95 --seed 0`,
},
numPromptsByConc: { 1: 32, 16: 32, 64: 128, 256: 512, 1024: 2048 },
},
accuracyLabels: [
["gsm8k_pct", "GSM8K", "%"],
],
multiNodeHints: {
// GB300 NVL/MNNVL fabric may need manual NIC configuration; NVSHMEM lines
// only matter if you switch the MoE a2a backend to DeepEP in the Playground.
gb300: [
"The following env vars may be needed depending on your cluster:",
" GLOO_SOCKET_IFNAME=<your-nic>",
" NVSHMEM_ENABLE_NIC_PE_MAPPING=1",
" NVSHMEM_HCA_LIST=<your-hca-list>",
],
},
dockerImages: {
// The hy4-preview image bundles the HYV4 model code, the suffix-aware
// `hunyuan` parsers, and the NEXTN MTP runtime. Switch to `:latest` once
// a tagged release picks them up.
h200: "lmsysorg/sglang:hy4-preview",
b200: "lmsysorg/sglang:hy4-preview",
b300: "lmsysorg/sglang:hy4-preview",
gb300: "lmsysorg/sglang:hy4-preview",
},
github: {
cookbookModel: "tencent/Hy4-preview",
},
playgroundFeatures: {
// ----- Card 1: "Attention Parallelism" -----
// No CP knob: HYV4ForCausalLM rejects --enable-prefill-cp before
// allocation (and pipeline parallelism likewise raises). TP starts at 4 —
// no listed GPU holds the weights below TP4 (MXFP8 ≈ 760GB, BF16 ≈
// 1.5TB) — and each degree is gated per hardware/quant so the panel
// never emits a command whose weights don't fit or whose rank count
// exceeds the selected topology (GB300 hosts carry 4 GPUs).
attention: {
knobs: [
{ id: "tp", label: "TP", values: [
null,
{ value: 4,
disable: [
{ when: { quant: ["bf16"] },
reason: "TP=4 cannot hold the ~1.5TB BF16 weights (~380GB/rank)." },
{ when: { hw: ["h200", "b200"] },
reason: "TP=4 MXFP8 needs ~190GB/rank — requires 288GB GPUs (B300/GB300)." },
] },
{ value: 8,
disable: [
{ when: { hw: ["h200", "b200"], quant: ["bf16"] },
reason: "TP=8 BF16 needs ~190GB/rank — exceeds H200/B200 VRAM; use TP=16 across 2 nodes." },
{ when: { hw: ["gb300"], nodes: ["single"] },
reason: "GB300 hosts carry 4 GPUs — TP=8 needs Multi-Nodes (2×4)." },
] },
{ value: 16,
disable: [
{ when: { nodes: ["single"] },
reason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
{ when: { hw: ["gb300"] },
reason: "GB300 hosts carry 4 GPUs — 2 nodes provide only 8 ranks." },
] },
]},
{ id: "dpAttn", label: "DP-Attention",
values: [
null,
false,
4,
{ value: 8,
disable: [
{ when: { effTp: [4] },
reason: "DP-Attention=8 needs TP ≥ 8 (TP must be divisible by the DP degree) — raise TP in this card first." },
{ when: { hw: ["gb300"], nodes: ["single"] },
reason: "GB300 hosts carry 4 GPUs — 8 attention ranks need Multi-Nodes (2×4)." },
] },
{ value: 16,
disable: [
{ when: { effTp: [4, 8] },
reason: "DP-Attention=16 needs TP=16 — raise TP in this card first." },
{ when: { nodes: ["single"] },
reason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
{ when: { hw: ["gb300"] },
reason: "GB300 hosts carry 4 GPUs — 2 nodes provide only 8 ranks." },
] },
],
labels: { "auto": "Auto", "false": "Off" } },
],
},
// ----- Card 2: "MoE Parallelism" -----
// 256 routed + 1 shared experts, top-8 sigmoid routing. The recipes run
// the MoE under pure TP (deep_gemm runner on B300/GB300 MXFP8, triton on
// B200); DeepEP/EP are experimentation overrides. No MegaMoE option —
// its fused path is not wired for Hy4's sigmoid-scored, bounded-SwiGLU
// experts.
moe: {
backend: {
options: [
{ id: null, label: "Inherited" },
{ id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] },
],
},
ep: { label: "EP", values: [
null,
4,
{ value: 8,
disable: [
{ when: { effTp: [4] },
reason: "EP=8 needs TP ≥ 8 (TP must be divisible by the EP degree) — raise TP in the Attention card first." },
{ when: { hw: ["gb300"], nodes: ["single"] },
reason: "GB300 hosts carry 4 GPUs — EP=8 needs Multi-Nodes (2×4)." },
] },
{ value: 16,
disable: [
{ when: { effTp: [4, 8] },
reason: "EP=16 needs TP=16 — raise TP in the Attention card first." },
{ when: { nodes: ["single"] },
reason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
{ when: { hw: ["gb300"] },
reason: "GB300 hosts carry 4 GPUs — 2 nodes provide only 8 ranks." },
] },
]},
},
// ----- Card 3: "Parsers" -----
// Auto-detection resolves to the `hunyuan` reasoning/tool-call parsers;
// the parser reads Hy4's suffix-bearing structural tokens (<think:...>,
// <tool_calls:...>, <arg_key:...>/<arg_value:...>) from the tokenizer
// vocab at runtime.
parsers: {
items: [
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser auto" },
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser auto" },
],
},
// ----- Card 4: "Speculative Decoding" -----
// One MTP (NextN) draft layer ships in both checkpoints; the preset is
// steps=3 / top-k 1 / draft-tokens=4. No NGRAM option — untested against
// the DSA sparse-attention backend.
speculative: {
options: [
{ id: "current", label: "Inherited from base" },
{ id: "off", label: "Off (greedy)" },
{ id: "nextn-34", label: "MTP / NextN 3-1-4",
flags: ["--speculative-algorithm NEXTN", "--speculative-num-steps 3",
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] },
],
},
// ----- Card 5: "PD Disaggregation" -----
// Generic SGLang prefill/decode disaggregation; not yet exercised on
// Hy4 — treat as experimentation.
pdDisagg: {
modes: [
{ id: "off", label: "Off" },
{ id: "prefill", label: "Prefill role" },
{ id: "decode", label: "Decode role" },
],
transferBackends: [
{ id: "mooncake", label: "Mooncake",
env: [
"NCCL_MNNVL_ENABLE=1",
"NCCL_CUMEM_ENABLE=1",
"SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True",
"MC_FORCE_MNNVL=1",
],
envWhen: { hw: ["gb300"] } },
{ id: "nixl", label: "NiXL" },
],
// `auto` is a sentinel (emits no --disaggregation-ib-device flag).
ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
// Router fronting the prefill + decode roles; substitute <prefill-host>/<decode-host>.
router: {
port: 8000,
command:
`python3 -m sglang_router.launch_router \\
--pd-disaggregation \\
--prefill http://<prefill-host>:{{PREFILL_PORT}} \\
--decode http://<decode-host>:{{DECODE_PORT}} \\
--host 0.0.0.0 --port {{ROUTER_PORT}} \\
--disable-circuit-breaker \\
--health-check-interval-secs 999999`,
},
},
// ----- Card 6: "Hierarchical KV Cache" -----
// Generic SGLang tiers; not yet exercised against Hy4's DSA FP8 indexer
// cache — treat as experimentation.
hicache: {
backends: [
{ id: null, label: "Auto" },
{ id: "file", label: "File" },
{ id: "mooncake", label: "Mooncake" },
{ id: "hf3fs", label: "HF3FS" },
{ id: "nixl", label: "NiXL" },
],
writePolicies: [
{ id: "auto", label: "Auto" },
{ id: "write_through", label: "Write-through" },
{ id: "write_back", label: "Write-back" },
{ id: "write_through_selective", label: "Write-through (selective)" },
],
},
// ----- Card 7: "HiSparse" -----
// DSA-style decode-side hierarchical sparse attention (Hy4's DSA indexer
// top-k is 2048). Shown/emitted only when the live PD-Disagg mode is
// `decode`; not yet exercised on Hy4 — treat as experimentation.
hisparse: {
requiredFlags: ["--disable-radix-cache"],
config: { top_k: 2048, device_buffer_size: 6144 },
hostRatios: [
{ id: 5, label: "5 (~1TB host)" },
{ id: 10, label: "10 (~2TB host)" },
],
defaultHostRatio: 10,
},
},
cells: [
// NOTE: the engine defaults a hash-less visit to cells[0], so the B300
// MXFP8 anchor (the recipe that has actually been served) stays first.
//
// The runtime derives the rest from the checkpoint and model defaults:
// quantization comes from the ModelOpt hf_quant_config, the DSA
// sparse-attention backend is auto-selected for HYV4, and decode
// CUDA-graph capture is on by default.
// ====================================================================
// B300 (288GB) × MXFP8 — TP4 single node (~190GB weights/rank).
// ====================================================================
{
match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend deep_gemm",
"--fp8-gemm-backend deep_gemm",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "b300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend deep_gemm",
"--fp8-gemm-backend deep_gemm",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ====================================================================
// B300 (288GB) × BF16 — TP8 single node (~190GB weights/rank on an
// 8-GPU node; same per-rank footprint as MXFP8 TP4).
// ====================================================================
{
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ====================================================================
// H200 (141GB) × BF16 — TP16 across 2x8 (~95GB weights/rank; the
// ~25GB/rank KV pool ≈ 260K tokens — size --context-length to it).
// No H200 MXFP8 cells: tested — SM90 cannot serve the MXFP8 checkpoint
// (the kernel path requires SM100+).
// ====================================================================
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ====================================================================
// B200 (192GB) × MXFP8 — TP8 single node (~95GB weights/rank). B200
// uses the triton MoE runner (the deep_gemm fused-expert path targets
// the 288GB SM103 parts).
// ====================================================================
{
match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--moe-runner-backend triton",
"--fp8-gemm-backend deep_gemm",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "b200", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--moe-runner-backend triton",
"--fp8-gemm-backend deep_gemm",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ====================================================================
// B200 (192GB) × BF16 — TP16 across 2x8 (~95GB weights/rank; 8x192GB
// cannot hold the ~1.5TB weights single-node).
// ====================================================================
{
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ====================================================================
// GB300 (288GB, 4-GPU hosts, sm_103 + aarch64) × MXFP8 — TP4 single
// node, same per-rank footprint as B300.
// ====================================================================
{
match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "low-latency", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend deep_gemm",
"--fp8-gemm-backend deep_gemm",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "gb300", variant: "default", quant: "mxfp8", strategy: "high-throughput", nodes: "single" },
verificationStatus: "in-progress",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--moe-runner-backend deep_gemm",
"--fp8-gemm-backend deep_gemm",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ====================================================================
// GB300 (288GB, 4-GPU hosts) × BF16 — TP8 across 2x4 nodes (~190GB
// weights/rank; a single 4-GPU host cannot hold the ~1.5TB weights).
// NCCL MNNVL env follows the GB-platform multi-node convention.
// ====================================================================
{
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" },
verificationStatus: "in-progress",
env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" },
verificationStatus: "in-progress",
env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--reasoning-parser auto",
"--tool-call-parser auto",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
};