sglang/docs/cookbook/autoregressive/Tencent/Hy3.mdx

---
title: Hy3
description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding."
---

## Deployment

<a id="install" />

<Accordion title="Install SGLang">

For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.

<Tabs>

<Tab title="Python (pip / uv)">

```bash
pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate

# Install from source (main carries the suffix-aware `hunyuan` parser + the
# HYV3 model code). Once a tagged release picks it up,
# `uv pip install --prerelease=allow sglang` is enough.
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv pip install --prerelease=allow -e python
```

Run the **Python** output of the command panel below in that environment.

</Tab>

<Tab title="Docker">

```bash Command
# The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser.
docker pull lmsysorg/sglang:dev
```

For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces.

<Note>
The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding.
</Note>

</Tab>

</Tabs>

</Accordion>

Pick your hardware + recipe to generate the launch command.

- **Low-Latency** — fastest reply for a single user. Pick for chat.
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.

import { Deployment } from "/src/snippets/_deployment.jsx";
import { config }     from "/src/snippets/configs/tencent/hy3.jsx";
import { benchmarks } from "/src/snippets/configs/tencent/hy3-benchmarks.jsx";

<Deployment config={config} benchmarks={benchmarks} />

<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
  <p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
  <ul style={{margin: 0, paddingLeft: "1.25rem"}}>
    <li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
    <li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
    <li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
    <li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
    <li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
  </ul>
</div>

## Playground

The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.

The knobs come in two flavors:

- **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
- **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime).

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

import { Playground } from "/src/snippets/_playground.jsx";

<Playground config={config} />

<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
  <p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
  <ul style={{margin: 0, paddingLeft: "1.25rem"}}>
    <li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
  </ul>
</div>

## 1. Model Introduction

**Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.

**Key Features:**

- **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost.
- **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning.
- **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output.
- **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
- **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box.

**Available Model:**

- [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct
- [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8

**Recommended Generation Parameters:**

<table style={{width: "100%", borderCollapse: "collapse"}}>
  <thead>
    <tr style={{borderBottom: "2px solid #0052d9"}}>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>temperature</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>top_p</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>reasoning_effort</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>high</code> / <code>low</code> (thinking) or <code>no_think</code> (instant)</td>
    </tr>
  </tbody>
</table>

**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `<tool_calls:TAG>` instead of the bare `<tool_calls>`). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model.

## 2. Configuration Tips

**Hardware requirements (BF16, ~590GB weights):**

<table style={{width: "100%", borderCollapse: "collapse"}}>
  <thead>
    <tr style={{borderBottom: "2px solid #0052d9"}}>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPU</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>VRAM</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>TP</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>minimum single-node for BF16</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 590GB → 148GB/GPU</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B300 / GB300</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>288GB</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 590GB → 148GB/GPU; ample KV headroom</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>GB200</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node 4×192GB = 768GB fits BF16 590GB</td>
    </tr>
  </tbody>
</table>

**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason.

**MTP (Multi-Token Prediction, EAGLE).**

- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
- `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient.

**`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is.

## 3. Advanced Usage

### 3.1 Reasoning (Hybrid Thinking)

Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`:

- `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content`
- `no_think` — skip thinking entirely (instant responses, content-only)

Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`:

<Accordion title="Deploy with reasoning parser">

```bash Command
sglang serve \
  --model-path tencent/Hy3 \
  --tp 8 \
  --reasoning-parser auto \
  --tool-call-parser auto
```

</Accordion>

<Accordion title="Example: thinking (reasoning_effort=high)">

```python Example
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
    reasoning_effort="high",
    max_tokens=2048,
)

msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)
```

```text Output
=============== Thinking =================
We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240.
10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36.
=============== Content =================
To find 15% of 240, follow these steps:

1. 15% = 15/100 or 0.15.
2. Multiply 240 by 0.15: 0.15 × 240 = 36.
3. Check: 10% of 240 = 24, 5% = 12, 15% = 36.

Thus, 15% of 240 is 36.
```

</Accordion>

<Accordion title="Example: instant mode (reasoning_effort=no_think)">

```python Example
response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    reasoning_effort="no_think",
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)
```

```text Output
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.
```

</Accordion>

### 3.2 Tool Calling

Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs:

<Accordion title="Deploy with tool-call parser">

```bash Command
sglang serve \
  --model-path tencent/Hy3 \
  --tp 8 \
  --reasoning-parser auto \
  --tool-call-parser auto
```

</Accordion>

<Accordion title="Example: non-streaming tool call">

```python Example
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
)

msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content:  ", msg.content)
for tc in msg.tool_calls or []:
    print(f"Tool Call: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")
```

```text Output
Reasoning: None
Content:   I'll get the current weather for Beijing in Fahrenheit for you.
Tool Call: get_weather
  Arguments: {"city": "Beijing", "unit": "fahrenheit"}
```

</Accordion>

<Accordion title="Example: streaming tool call (incremental argument deltas)">

```python Example
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
    stream=True,
)

tool_buffer = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
    for tc in delta.tool_calls or []:
        buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
        if tc.function and tc.function.name:
            buf["name"] += tc.function.name
        if tc.function and tc.function.arguments:
            buf["args"] += tc.function.arguments

for idx, buf in tool_buffer.items():
    print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
```

```text Output
I'll check the current weather in Beijing for you using Fahrenheit.
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})
```

</Accordion>