--- title: Hy3 description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding." --- ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # Install from source (main carries the suffix-aware `hunyuan` parser + the # HYV3 model code). Once a tagged release picks it up, # `uv pip install --prerelease=allow sglang` is enough. git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Run the **Python** output of the command panel below in that environment. ```bash Command # The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser. docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces. The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding. Pick your hardware + recipe to generate the launch command. - **Low-Latency** — fastest reply for a single user. Pick for chat. - **Balanced** — good speed with several users at once. Use for typical multi-user serving. import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/tencent/hy3.jsx"; import { benchmarks } from "/src/snippets/configs/tencent/hy3-benchmarks.jsx";

Panel controls (top of the command box):
## Playground The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors: - **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. - **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime). Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back. import { Playground } from "/src/snippets/_playground.jsx";

Panel controls reuse Python / Docker · ⧉ Copy · $ cURL · ⚙ Env from the Deploy panel, plus one extra:

## 1. Model Introduction **Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. **Key Features:** - **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost. - **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning. - **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output. - **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning. - **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box. **Available Model:** - [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct - [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8 **Recommended Generation Parameters:**
Parameter Value
temperature 0.9
top_p 1.0
reasoning_effort high / low (thinking) or no_think (instant)
**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `` instead of the bare ``). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model. ## 2. Configuration Tips **Hardware requirements (BF16, ~590GB weights):**
GPU VRAM TP Notes
H200 141GB 8 minimum single-node for BF16
B200 192GB 4 BF16 590GB → 148GB/GPU
B300 / GB300 288GB 4 BF16 590GB → 148GB/GPU; ample KV headroom
GB200 192GB 4 single-node 4×192GB = 768GB fits BF16 590GB
**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason. **MTP (Multi-Token Prediction, EAGLE).** - `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. - `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient. **`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is. ## 3. Advanced Usage ### 3.1 Reasoning (Hybrid Thinking) Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`: - `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content` - `no_think` — skip thinking entirely (instant responses, content-only) Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`: ```bash Command sglang serve \ --model-path tencent/Hy3 \ --tp 8 \ --reasoning-parser auto \ --tool-call-parser auto ``` ```python Example from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], reasoning_effort="high", max_tokens=2048, ) msg = response.choices[0].message print("=============== Thinking =================") print(msg.reasoning_content) print("=============== Content =================") print(msg.content) ``` ```text Output =============== Thinking ================= We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240. 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36. =============== Content ================= To find 15% of 240, follow these steps: 1. 15% = 15/100 or 0.15. 2. Multiply 240 by 0.15: 0.15 × 240 = 36. 3. Check: 10% of 240 = 24, 5% = 12, 15% = 36. Thus, 15% of 240 is 36. ``` ```python Example response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], reasoning_effort="no_think", max_tokens=256, ) print("Content:", response.choices[0].message.content) ``` ```text Output Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe. ``` ### 3.2 Tool Calling Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs: ```bash Command sglang serve \ --model-path tencent/Hy3 \ --tp 8 \ --reasoning-parser auto \ --tool-call-parser auto ``` ```python Example from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, ) msg = response.choices[0].message print("Reasoning:", msg.reasoning_content) print("Content: ", msg.content) for tc in msg.tool_calls or []: print(f"Tool Call: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` ```text Output Reasoning: None Content: I'll get the current weather for Beijing in Fahrenheit for you. Tool Call: get_weather Arguments: {"city": "Beijing", "unit": "fahrenheit"} ``` ```python Example from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, stream=True, ) tool_buffer = {} for chunk in stream: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) for tc in delta.tool_calls or []: buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""}) if tc.function and tc.function.name: buf["name"] += tc.function.name if tc.function and tc.function.arguments: buf["args"] += tc.function.arguments for idx, buf in tool_buffer.items(): print(f"\nTool[{idx}] {buf['name']}({buf['args']})") ``` ```text Output I'll check the current weather in Beijing for you using Fahrenheit. Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"}) ```