cookbook: add Hunyuan 3 (Hy3) Day-0 page (#30201)
This commit is contained in:
Xinyuan Tong
@@ -2,7 +2,6 @@
|
||||
title: Hunyuan 3 Preview
|
||||
metatags:
|
||||
description: "Deploy Tencent Hunyuan 3 Preview BF16 (~276B / ~20B active MoE) on NVIDIA GPUs with SGLang — hybrid thinking, native tool calling, 256K context, and built-in MTP speculative decoding."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
@@ -0,0 +1,370 @@
|
||||
---
|
||||
title: Hy3
|
||||
description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
# Install from source (main carries the suffix-aware `hunyuan` parser + the
|
||||
# HYV3 model code). Once a tagged release picks it up, `uv pip install sglang`
|
||||
# is enough.
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser.
|
||||
docker pull lmsysorg/sglang:dev
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
<Note>
|
||||
The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding.
|
||||
</Note>
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command.
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/tencent/hy3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/tencent/hy3-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
The knobs come in two flavors:
|
||||
|
||||
- **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
|
||||
- **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime).
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost.
|
||||
- **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning.
|
||||
- **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output.
|
||||
- **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
|
||||
- **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box.
|
||||
|
||||
**Available Model:**
|
||||
|
||||
- [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct
|
||||
- [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8
|
||||
|
||||
**Recommended Generation Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #0052d9"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>temperature</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>top_p</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>reasoning_effort</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>high</code> / <code>low</code> (thinking) or <code>no_think</code> (instant)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `<tool_calls:TAG>` instead of the bare `<tool_calls>`). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model.
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Hardware requirements (BF16, ~590GB weights):**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #0052d9"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPU</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>VRAM</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>TP</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>minimum single-node for BF16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 590GB → 148GB/GPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B300 / GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>288GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 590GB → 148GB/GPU; ample KV headroom</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>GB200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node 4×192GB = 768GB fits BF16 590GB</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason.
|
||||
|
||||
**MTP (Multi-Token Prediction, EAGLE).**
|
||||
|
||||
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
|
||||
- `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient.
|
||||
|
||||
**`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning (Hybrid Thinking)
|
||||
|
||||
Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`:
|
||||
|
||||
- `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content`
|
||||
- `no_think` — skip thinking entirely (instant responses, content-only)
|
||||
|
||||
Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`:
|
||||
|
||||
<Accordion title="Deploy with reasoning parser">
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3 \
|
||||
--tp 8 \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: thinking (reasoning_effort=high)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
|
||||
reasoning_effort="high",
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("=============== Thinking =================")
|
||||
print(msg.reasoning_content)
|
||||
print("=============== Content =================")
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240.
|
||||
10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36.
|
||||
=============== Content =================
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. 15% = 15/100 or 0.15.
|
||||
2. Multiply 240 by 0.15: 0.15 × 240 = 36.
|
||||
3. Check: 10% of 240 = 24, 5% = 12, 15% = 36.
|
||||
|
||||
Thus, 15% of 240 is 36.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: instant mode (reasoning_effort=no_think)">
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
|
||||
reasoning_effort="no_think",
|
||||
max_tokens=256,
|
||||
)
|
||||
|
||||
print("Content:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
```text Output
|
||||
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs:
|
||||
|
||||
<Accordion title="Deploy with tool-call parser">
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3 \
|
||||
--tp 8 \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: non-streaming tool call">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {"type": "string"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("Reasoning:", msg.reasoning_content)
|
||||
print("Content: ", msg.content)
|
||||
for tc in msg.tool_calls or []:
|
||||
print(f"Tool Call: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
```text Output
|
||||
Reasoning: None
|
||||
Content: I'll get the current weather for Beijing in Fahrenheit for you.
|
||||
Tool Call: get_weather
|
||||
Arguments: {"city": "Beijing", "unit": "fahrenheit"}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: streaming tool call (incremental argument deltas)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
|
||||
tools=tools,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
tool_buffer = {}
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
for tc in delta.tool_calls or []:
|
||||
buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
|
||||
if tc.function and tc.function.name:
|
||||
buf["name"] += tc.function.name
|
||||
if tc.function and tc.function.arguments:
|
||||
buf["args"] += tc.function.arguments
|
||||
|
||||
for idx, buf in tool_buffer.items():
|
||||
print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
|
||||
```
|
||||
|
||||
```text Output
|
||||
I'll check the current weather in Beijing for you using Fahrenheit.
|
||||
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -136,7 +136,7 @@ metatags:
|
||||
<Card
|
||||
title="Tencent"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Tencent/Hunyuan3-Preview"
|
||||
href="/cookbook/autoregressive/Tencent/Hy3"
|
||||
img="/cards/logos/tencent.png"
|
||||
/>
|
||||
<Card
|
||||
|
||||
@@ -1146,6 +1146,7 @@
|
||||
{
|
||||
"group": "Tencent",
|
||||
"pages": [
|
||||
"cookbook/autoregressive/Tencent/Hy3",
|
||||
"cookbook/autoregressive/Tencent/Hunyuan3-Preview"
|
||||
]
|
||||
},
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
// Hy3 per-cell benchmark numbers, keyed by the same `match` tuple as hy3.jsx cells.
|
||||
// See _deployment.jsx for the speed/accuracy schema.
|
||||
// H200 BF16 low-latency + balanced verified on 8×H200 (sgl-eval, single-shot, temp=0).
|
||||
// FP8 cells not yet verified.
|
||||
export const benchmarks = [
|
||||
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, gsm8k_pct: 95.75 },
|
||||
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, gsm8k_pct: 95.83 },
|
||||
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "gb200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
];
|
||||
@@ -0,0 +1,546 @@
|
||||
// Hy3 cookbook config. Consumed by _deployment.jsx + _playground.jsx;
|
||||
// see _deployment.jsx header for the field contract.
|
||||
//
|
||||
// The shipping Hy3 tokenizer appends a shared suffix to every special token
|
||||
// (e.g. <tool_calls:TAG>); SGLang's `hunyuan` reasoning/tool-call parsers
|
||||
// resolve the real token strings from the vocab at runtime (PR #29920), so the
|
||||
// same recipe serves both the preview (suffix-less) and the shipping (suffixed)
|
||||
// tokenizer — no per-model hard-coding.
|
||||
//
|
||||
// BF16 weights are ~590GB. Single-node TP fits: H200 (141GB, TP8 = 74GB/GPU),
|
||||
// B200 (180GB, TP4 = 148GB/GPU), B300/GB300 (272GB, TP4), GB200 (192GB, TP4).
|
||||
|
||||
export const config = {
|
||||
modelName: "Hy3",
|
||||
|
||||
supportedHardware: ["h200", "b200", "b300", "gb200", "gb300"],
|
||||
|
||||
variants: [
|
||||
{ id: "default", label: "Default" },
|
||||
],
|
||||
quantizations: [
|
||||
{ id: "bf16", label: "BF16" },
|
||||
{ id: "fp8", label: "FP8" },
|
||||
],
|
||||
strategies: [
|
||||
{ id: "low-latency", label: "Low-Latency" },
|
||||
{ id: "balanced", label: "Balanced" },
|
||||
],
|
||||
nodesOptions: [
|
||||
{ id: "single", label: "Single Node" },
|
||||
{ id: "multi-2", label: "Multi-Nodes" },
|
||||
],
|
||||
|
||||
modelNames: {
|
||||
"default|bf16": "tencent/Hy3",
|
||||
"default|fp8": "tencent/Hy3-FP8",
|
||||
},
|
||||
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
NODE0_IP: { target: "command", label: "Head node IP", default: "<node0-ip>" },
|
||||
NODE_RANK: { target: "command", label: "This node rank", default: "<node-rank>" },
|
||||
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
|
||||
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||
},
|
||||
|
||||
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
|
||||
|
||||
benchmarkCommands: {
|
||||
speed:
|
||||
`python3 -m sglang.bench_serving \\
|
||||
--backend sglang \\
|
||||
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
|
||||
--model {{MODEL_NAME}} \\
|
||||
--dataset-name {{DATASET}} \\
|
||||
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
|
||||
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\
|
||||
--warmup-requests 64`,
|
||||
accuracy: {
|
||||
gsm8k_pct:
|
||||
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
|
||||
sgl-eval run gsm8k \\
|
||||
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||
--num-threads 32`,
|
||||
aime26_pct:
|
||||
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
|
||||
sgl-eval run aime26 \\
|
||||
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||
--model {{MODEL_NAME}} --api-key <api-key> \\
|
||||
--n-repeats 1 --max-tokens 28672 \\
|
||||
--temperature 0.6 --top-p 0.95 --thinking \\
|
||||
--out-dir /sgl-workspace/logs`,
|
||||
},
|
||||
numPromptsByConc: { 1: 32, 16: 32, 64: 128, 256: 512, 1024: 2048 },
|
||||
},
|
||||
|
||||
accuracyLabels: [
|
||||
["gsm8k_pct", "GSM8K (1-shot)", "%"],
|
||||
["aime26_pct", "AIME26", "%"],
|
||||
],
|
||||
|
||||
multiNodeHints: {
|
||||
gb200: [
|
||||
"The following env vars may be needed depending on your cluster:",
|
||||
" GLOO_SOCKET_IFNAME=<your-nic>",
|
||||
" NVSHMEM_ENABLE_NIC_PE_MAPPING=1",
|
||||
" NVSHMEM_HCA_LIST=<your-hca-list>",
|
||||
],
|
||||
},
|
||||
|
||||
dockerImages: {
|
||||
// The dev image bundles the HYV3 model code + the suffix-aware `hunyuan`
|
||||
// parser. Switch to `:latest` once a tagged release picks it up.
|
||||
h200: "lmsysorg/sglang:dev",
|
||||
b200: "lmsysorg/sglang:dev",
|
||||
b300: "lmsysorg/sglang:dev",
|
||||
gb200: "lmsysorg/sglang:dev",
|
||||
gb300: "lmsysorg/sglang:dev",
|
||||
},
|
||||
|
||||
github: {
|
||||
cookbookModel: "tencent/Hy3",
|
||||
},
|
||||
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card 1: "Attention Parallelism" -----
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
null,
|
||||
1,
|
||||
2,
|
||||
4,
|
||||
8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
{ id: "cp", label: "CP", values: [null, 1, 2, 4] },
|
||||
{ id: "dpAttn", label: "DP-Attention",
|
||||
values: [
|
||||
null,
|
||||
false,
|
||||
1,
|
||||
2,
|
||||
4,
|
||||
8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
],
|
||||
labels: { "auto": "Auto", "false": "Off" } },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card 2: "MoE Parallelism" -----
|
||||
moe: {
|
||||
backend: {
|
||||
options: [
|
||||
{ id: null, label: "Inherited" },
|
||||
{ id: "deepep", label: "DeepEP",
|
||||
flags: ["--moe-a2a-backend deepep"] },
|
||||
{ id: "megamoe", label: "MegaMoE",
|
||||
flags: ["--moe-a2a-backend megamoe"],
|
||||
requiresHw: ["b200", "b300", "gb200", "gb300"] },
|
||||
],
|
||||
},
|
||||
ep: { label: "EP", values: [
|
||||
null,
|
||||
1,
|
||||
2,
|
||||
4,
|
||||
8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
},
|
||||
|
||||
// ----- Card 3: "Parsers" -----
|
||||
parsers: {
|
||||
items: [
|
||||
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser auto" },
|
||||
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser auto" },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card 4: "Speculative Decoding" -----
|
||||
speculative: {
|
||||
options: [
|
||||
{ id: "current", label: "Inherited from base" },
|
||||
{ id: "off", label: "Off (greedy)" },
|
||||
{ id: "mtp-314", label: "EAGLE / MTP 3-1-4",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] },
|
||||
{ id: "mtp-112", label: "EAGLE / MTP 1-1-2",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"] },
|
||||
{ id: "ngram", label: "NGRAM",
|
||||
flags: ["--speculative-algorithm NGRAM",
|
||||
"--speculative-num-draft-tokens 16",
|
||||
"--speculative-ngram-max-bfs-breadth 10"],
|
||||
disable: { dpAttnOn: [true] },
|
||||
disableReason: "NGRAM is incompatible with DP-Attention. Turn DP-Attention off in the Attention card above to use NGRAM." },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card 5: "PD Disaggregation" -----
|
||||
pdDisagg: {
|
||||
modes: [
|
||||
{ id: "off", label: "Off" },
|
||||
{ id: "prefill", label: "Prefill role" },
|
||||
{ id: "decode", label: "Decode role" },
|
||||
],
|
||||
transferBackends: [
|
||||
{ id: "mooncake", label: "Mooncake",
|
||||
env: [
|
||||
"NCCL_MNNVL_ENABLE=1",
|
||||
"NCCL_CUMEM_ENABLE=1",
|
||||
"SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True",
|
||||
"MC_FORCE_MNNVL=1",
|
||||
],
|
||||
envWhen: { hw: ["gb200", "gb300"] } },
|
||||
{ id: "nixl", label: "NiXL" },
|
||||
],
|
||||
ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
|
||||
router: {
|
||||
port: 8000,
|
||||
command:
|
||||
`python3 -m sglang_router.launch_router \\
|
||||
--pd-disaggregation \\
|
||||
--prefill http://<prefill-host>:{{PREFILL_PORT}} \\
|
||||
--decode http://<decode-host>:{{DECODE_PORT}} \\
|
||||
--policy round_robin \\
|
||||
--host 0.0.0.0 --port {{ROUTER_PORT}}`,
|
||||
},
|
||||
},
|
||||
|
||||
// ----- Card 6: "Hierarchical KV Cache" -----
|
||||
hicache: {
|
||||
backends: [
|
||||
{ id: "null_placeholder", label: "Auto" },
|
||||
{ id: "file", label: "File" },
|
||||
{ id: "mooncake", label: "Mooncake" },
|
||||
{ id: "hf3fs", label: "HF3FS" },
|
||||
{ id: "nixl", label: "NiXL" },
|
||||
],
|
||||
writePolicies: [
|
||||
{ id: "auto", label: "Auto" },
|
||||
{ id: "write_through", label: "Write-through" },
|
||||
{ id: "write_back", label: "Write-back" },
|
||||
{ id: "write_through_selective", label: "Write-through (selective)" },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
cells: [
|
||||
// ====================================================================
|
||||
// H200 (141GB) — TP=8 for BF16 (~590GB)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 8",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 8",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// B200 (180GB) — TP=4 (BF16 590GB → 148GB/GPU, fits with KV headroom)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// B300 (272GB) — TP=4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// GB300 — TP=4 (inferred from B300, same sm_103 + aarch64)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// GB200 (sm_100 + aarch64) — TP=4 (single-node 4×192GB = 768GB fits BF16 590GB)
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// FP8 (~300GB) — TP=4 on H200/B200, TP=2 on B300/GB300/GB200
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 2",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 2",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 2",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 2",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 2",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--reasoning-parser auto",
|
||||
"--tool-call-parser auto",
|
||||
"--tp 2",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
Reference in New Issue
Block a user