[Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c949e91f18
commit
b819d2fb5b
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: Unlimited-OCR
|
||||
description: "Deploy Baidu Unlimited-OCR with SGLang for long document OCR using prefill-aware sliding-window attention."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Unlimited-OCR support is in [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186). Until that PR is included in a tagged SGLang release, install from a build that contains the PR.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git fetch origin pull/29186/head && git checkout FETCH_HEAD
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware to generate the launch command. The recipe uses FlashAttention-3 with `--page-size 1`, which is required by the current prefill-aware sliding-window attention path. It also disables radix cache by default, which is the better fit for batch OCR workloads where each request usually contains a different image.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/baidu/unlimited-ocr.jsx";
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## Playground
|
||||
|
||||
Use the Playground to adjust tensor parallelism on top of the selected deployment cell.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's multimodal OCR model for document parsing. It uses a sliding-window language backbone, but SGLang serves it with a prefill-aware sliding-window path so image and prompt tokens remain visible during long decode.
|
||||
|
||||
The SGLang integration loads the standalone Unlimited-OCR architecture with SAM and CLIP vision encoders plus a DeepSeek-style language backbone. It supports OpenAI-compatible image requests and model-specific image processing options through `images_config`.
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) · [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Attention backend**: use `--attention-backend fa3 --page-size 1`. The prefill-aware SWA page table is built with token-level locations, so page size 1 is required.
|
||||
- **Radix cache**: keep `--disable-radix-cache` for batch OCR over different documents. If your workload repeatedly asks about the same image and prompt, remove this flag to allow prefix reuse through `PureSWARadixCache`.
|
||||
- **Long OCR generations**: keep the default prefill-aware SWA path enabled. It retains prompt and image KV while still applying a sliding window to generated text.
|
||||
- **Custom logit processor**: keep `--enable-custom-logit-processor` in the launch command.
|
||||
- **Image modes**: pass `images_config.image_mode` per request. Supported modes are `tiny`, `small`, `base`, `large`, and `gundam`. Multiple images are supported only for `tiny`, `small`, and `base`.
|
||||
- **Default image mode**: when `images_config.image_mode` is omitted, SGLang uses `gundam`.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 OCR request
|
||||
|
||||
<Accordion title="OCR Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/Unlimited-OCR",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "document parsing."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/your_document.png"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
temperature=0,
|
||||
extra_body={"images_config": {"image_mode": "gundam"}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Choosing an image mode
|
||||
|
||||
Use lower modes to reduce prefill cost for simple images, and use `gundam` for high-detail document parsing.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Use</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Multiple images</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>tiny</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Lowest prefill cost.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>small</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Lightweight OCR requests.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>base</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Balanced quality and cost.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>large</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Higher resolution single-image OCR.</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>gundam</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Default high-detail document parsing mode.</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
title: Ornith-1.0
|
||||
description: "Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Ornith-1.0 model cards recommend SGLang `>=0.5.9`. The Deploy panel below emits the base serve command; the reasoning and tool-call parsers from the model-card quickstarts (`--reasoning-parser qwen3` for `<think>...</think>` traces, `--tool-call-parser qwen3_coder` for Qwen-style XML tool calls) are added on top via the [Playground](#playground).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install "sglang>=0.5.9"
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick an Ornith checkpoint to generate the launch command. The non-FP8 397B recipe requires an H200 single node in this matrix. The 397B-FP8 recipe is available on H100 and H200 with TP=8; H100 also supports the 35B and 9B checkpoints. The 35B recipes use tensor parallelism 2 in this matrix. The 9B checkpoint is dense and serves on a single GPU by default; the command panel makes that default explicit with `--tp 1`.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/deepreinforce-ai/ornith-1.0.jsx";
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground layers SGLang features on top of whichever cell the Deploy panel is showing — only your overrides change, and any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
|
||||
|
||||
For Ornith-1.0 the knobs are the reasoning and tool-call parsers:
|
||||
|
||||
- **Reasoning Parser** appends `--reasoning-parser qwen3`. Ornith emits `<think>...</think>` traces; with this on, SGLang surfaces them as `message.reasoning_content` instead of leaving the tags inline in `content`.
|
||||
- **Tool Call Parser** appends `--tool-call-parser qwen3_coder`, so Qwen-style XML tool calls are returned as OpenAI-compatible `tool_calls`.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ornith-1.0](https://huggingface.co/collections/deepreinforce-ai/ornith-10) is DeepReinforce's self-improving open-source model family for agentic coding. The model cards describe the family as post-trained on top of Gemma 4 and Qwen 3.5, and the collection currently includes 397B, 35B, and 9B repos plus FP8 and GGUF variants. The model cards report results on Terminal-Bench 2.1, SWE-Bench, NL2Repo, ClawEval, and SWE Atlas benchmarks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Agentic coding specialization**: the model cards describe Ornith-1.0 as specialized for agentic coding and report coding-agent benchmark results.
|
||||
- **Self-improving training**: the model cards state that Ornith-1.0 uses reinforcement learning to optimize both solution rollouts and the scaffold that drives those rollouts.
|
||||
- **Reasoning model behavior**: assistant responses begin with a `<think>...</think>` reasoning block before the final answer; enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) to split it into `reasoning_content`.
|
||||
- **Tool calling**: emits Qwen-style XML tool calls; enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground).
|
||||
- **Long context**: model-card recipes use `--context-length 262144`.
|
||||
- **MIT license**: the Hugging Face repos are released under MIT.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "32%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "32%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Format</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Deploy Panel</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B">deepreinforce-ai/Ornith-1.0-397B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>H200 only</td>
|
||||
<td style={{padding: "9px 12px"}}>Flagship 397B MoE checkpoint; model-card baseline uses TP=8 on an H200 single node.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B-FP8">deepreinforce-ai/Ornith-1.0-397B-FP8</a></td>
|
||||
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 repo in the collection; the deploy command uses this repo id with TP=8.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B">deepreinforce-ai/Ornith-1.0-35B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>35B MoE checkpoint; the deploy command uses TP=2.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-FP8">deepreinforce-ai/Ornith-1.0-35B-FP8</a></td>
|
||||
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 repo in the collection; the deploy command uses this repo id with TP=2.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B">deepreinforce-ai/Ornith-1.0-9B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>Dense 9B checkpoint; the model card describes it as designed for efficient single-GPU deployment.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF">deepreinforce-ai/Ornith-1.0-35B-GGUF</a></td>
|
||||
<td style={{padding: "9px 12px"}}>GGUF</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
<td style={{padding: "9px 12px"}}>Listed for completeness; GGUF targets llama.cpp-style local inference, not the SGLang server recipe here.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF">deepreinforce-ai/Ornith-1.0-9B-GGUF</a></td>
|
||||
<td style={{padding: "9px 12px"}}>GGUF</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
<td style={{padding: "9px 12px"}}>Listed for completeness; the model card shows llama.cpp and Ollama examples for the GGUF build.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/blob/main/LICENSE)
|
||||
|
||||
**Resources:** [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10) · [Ornith blog](https://deep-reinforce.com/ornith_1_0.html)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Reasoning parser**: Ornith responses include `<think>...</think>`. Enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) so OpenAI-compatible responses expose the reasoning trace as `message.reasoning_content`.
|
||||
- **Tool-call parser**: enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) so `<tool_call>` blocks are returned as OpenAI-compatible tool calls.
|
||||
- **Context length**: the model-card SGLang recipes use `--context-length 262144`. Lower it if you need more memory headroom.
|
||||
- **Tensor parallelism**: the 397B model-card recipes use `--tp 8`; in this single-node matrix, non-FP8 397B is H200-only, while 397B-FP8 is available on both H100 and H200. The 35B deploy commands use `--tp 2`. The 9B model-card recipe is single-GPU by default; the command panel makes that explicit with `--tp 1`. Adjust TP to match your node and memory budget.
|
||||
- **Sampling**: model cards recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20` for normal use. Their reported benchmark setup may use different task-specific sampling parameters.
|
||||
- **Benchmarks**: benchmark numbers in the model cards are reported by DeepReinforce. They are useful for context, but the command panel leaves recipes unverified until exact runs are signed off.
|
||||
|
||||
## 3. Usage Examples
|
||||
|
||||
### 3.1 Basic Chat Completion
|
||||
|
||||
`message.reasoning_content` is only populated when the server was launched with the `--reasoning-parser qwen3` toggle (see the [Playground](#playground)); otherwise the `<think>...</think>` trace stays inline in `message.content`.
|
||||
|
||||
<Accordion title="Python client">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Ornith-1.0-9B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a compact Python function is_prime(n)."}
|
||||
],
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=1024,
|
||||
extra_body={"top_k": 20},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) and launch with the resulting command. Then use the standard OpenAI-compatible `tools` field:
|
||||
|
||||
<Accordion title="Tool-call request">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run_tests",
|
||||
"description": "Run the project's test suite.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"target": {"type": "string", "description": "Test target or command"}
|
||||
},
|
||||
"required": ["target"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Ornith-1.0-9B",
|
||||
messages=[{"role": "user", "content": "Run the unit tests for the parser module."}],
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,526 @@
|
||||
---
|
||||
title: DeepSeek-Math-V2
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-Math-V2 with SGLang - advanced mathematical reasoning model with gold-level IMO/CMO performance and theorem-proving capabilities."
|
||||
---
|
||||
|
||||
import { DeepSeekMathV2Deployment } from '/src/snippets/autoregressive/deepseek-math-v2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024
|
||||
- **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy
|
||||
- **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024
|
||||
- **Large MoE Model**: ~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
|
||||
|
||||
<DeepSeekMathV2Deployment />
|
||||
|
||||
<Warning>
|
||||
DeepSeek-Math-V2 is built on DeepSeek-V3.2 and uses DSA sparse attention. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on this model.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Hardware Requirements:**
|
||||
|
||||
- **B200 (183GB)**: BF16 tp=8
|
||||
- **B300 (275GB)**: BF16 tp=8
|
||||
|
||||
**DP Attention:**
|
||||
|
||||
- Enable DP attention for high-throughput scenarios
|
||||
- The `--dp` value commonly matches the `--tp` value
|
||||
- Trade-off: Higher throughput at the cost of slightly increased latency
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment Command
|
||||
|
||||
Deploy the model using the command generated above. Example for B200:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 Mathematical Reasoning
|
||||
|
||||
DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning.
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Mathematical reasoning problem
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-Math-V2",
|
||||
messages=[
|
||||
{"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"}
|
||||
],
|
||||
max_tokens=4096,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2.
|
||||
|
||||
This is a classic formula for the sum of the first n natural numbers. We can prove by induction.
|
||||
|
||||
Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds.
|
||||
|
||||
Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1
|
||||
)((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n.
|
||||
|
||||
...
|
||||
=============== Content =================
|
||||
We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below.
|
||||
|
||||
---
|
||||
|
||||
### 1. Proof by mathematical induction
|
||||
|
||||
**Base case (\(n=1\))**:
|
||||
\[
|
||||
1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1,
|
||||
\]
|
||||
so the formula holds for \(n=1\).
|
||||
|
||||
**Inductive hypothesis:**
|
||||
Assume that for some positive integer \(k\) the formula is true, i.e.
|
||||
\[
|
||||
1+2+\dots+k = \frac{k(k+1)}{2}.
|
||||
\]
|
||||
|
||||
**Inductive step (\(k \to k+1\))**:
|
||||
Consider the sum up to \(k+1\):
|
||||
\[
|
||||
\begin{aligned}
|
||||
1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt]
|
||||
&= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt]
|
||||
&= (k+1)\left(\frac{k}{2}+1\right)\\[4pt]
|
||||
&= (k+1)\frac{k+2}{2}\\[4pt]
|
||||
&= \frac{(k+1)(k+2)}{2}\\[4pt]
|
||||
&= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}.
|
||||
\end{aligned}
|
||||
\]
|
||||
Thus the formula also holds for \(n=k+1\).
|
||||
|
||||
By the principle of mathematical induction,
|
||||
\[
|
||||
1+2+3+\dots+n = \frac{n(n+1)}{2}
|
||||
\]
|
||||
for every positive integer \(n\).
|
||||
|
||||
---
|
||||
|
||||
### 2. Proof by pairing (Gauss’s trick)
|
||||
|
||||
Let
|
||||
\[
|
||||
S = 1 + 2 + 3 + \dots + n.
|
||||
\]
|
||||
|
||||
Write the same sum in reverse order:
|
||||
\[
|
||||
S = n + (n-1) + (n-2) + \dots + 1.
|
||||
\]
|
||||
|
||||
Add the two equalities term‑by‑term:
|
||||
\[
|
||||
\begin{aligned}
|
||||
2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\
|
||||
&= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\
|
||||
&= n\,(n+1).
|
||||
\end{aligned}
|
||||
\]
|
||||
|
||||
Therefore
|
||||
\[
|
||||
S = \frac{n(n+1)}{2}.
|
||||
\]
|
||||
|
||||
Both proofs are rigorous and show that the formula holds for all positive integers \(n\).
|
||||
```
|
||||
|
||||
### 4.3 Competition-Level Problems
|
||||
|
||||
**Example: IMO-style Problem:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# IMO-style problem
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-Math-V2",
|
||||
messages=[
|
||||
{"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."}
|
||||
],
|
||||
max_tokens=8192,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to prove that for positive real numbers a,b,c with abc = 1, we have:
|
||||
|
||||
\[
|
||||
(a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1.
|
||||
\]
|
||||
|
||||
We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite:
|
||||
|
||||
...
|
||||
=============== Content =================
|
||||
|
||||
We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove
|
||||
|
||||
\[
|
||||
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
### 1. A convenient substitution
|
||||
|
||||
Because \(abc=1\), we can write
|
||||
|
||||
\[
|
||||
a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x}
|
||||
\]
|
||||
|
||||
with positive numbers \(x,y,z\).
|
||||
(For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).)
|
||||
|
||||
---
|
||||
|
||||
### 2. Rewriting the factors
|
||||
|
||||
\[
|
||||
\begin{aligned}
|
||||
a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm]
|
||||
b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm]
|
||||
c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}.
|
||||
\end{aligned}
|
||||
\]
|
||||
|
||||
Hence the product becomes
|
||||
|
||||
\[
|
||||
P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)
|
||||
=\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}.
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
### 3. Reducing to a known inequality
|
||||
|
||||
We have to show \(P\le1\), i.e.
|
||||
|
||||
\[
|
||||
(x+z-y)(x+y-z)(y+z-x)\le xyz .
|
||||
\tag{1}
|
||||
\]
|
||||
|
||||
Set
|
||||
|
||||
\[
|
||||
p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz .
|
||||
\]
|
||||
|
||||
Notice that
|
||||
|
||||
\[
|
||||
x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x .
|
||||
\]
|
||||
|
||||
Therefore
|
||||
|
||||
\[
|
||||
\begin{aligned}
|
||||
(x+z-y)(x+y-z)(y+z-x)
|
||||
&=(p-2x)(p-2y)(p-2z)\\
|
||||
&=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\
|
||||
&=-p^{3}+4pq-8r .
|
||||
\end{aligned}
|
||||
\]
|
||||
|
||||
Inequality (1) is thus equivalent to
|
||||
|
||||
\[
|
||||
-p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r .
|
||||
\tag{2}
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
### 4. Applying Schur’s inequality
|
||||
|
||||
Schur’s inequality of third degree states that for any non‑negative \(x,y,z\)
|
||||
|
||||
\[
|
||||
p^{3}+9r\ge 4pq .
|
||||
\]
|
||||
|
||||
Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2).
|
||||
Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds.
|
||||
|
||||
Consequently (1) is true, and we obtain \(P\le1\).
|
||||
|
||||
---
|
||||
|
||||
### 5. Equality case
|
||||
|
||||
Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\).
|
||||
Then \(a=b=c=1\), and indeed the product equals \(1\).
|
||||
|
||||
---
|
||||
|
||||
Thus for all positive \(a,b,c\) with \(abc=1\),
|
||||
|
||||
\[
|
||||
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
|
||||
\]
|
||||
|
||||
∎
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
#### 5.1.1 GSM8K Benchmark
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.975
|
||||
Invalid: 0.000
|
||||
Latency: 34.358 s
|
||||
Output throughput: 540.162 token/s
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x, 183GB each)
|
||||
- Model: DeepSeek-Math-V2
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.8
|
||||
|
||||
#### 5.2.1 Latency Benchmark
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-Math-V2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 53.34
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2778
|
||||
Request throughput (req/s): 0.19
|
||||
Input token throughput (tok/s): 36.97
|
||||
Output token throughput (tok/s): 52.19
|
||||
Peak output token throughput (tok/s): 56.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 89.16
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5330.72
|
||||
Median E2E Latency (ms): 5879.28
|
||||
P90 E2E Latency (ms): 8320.33
|
||||
P99 E2E Latency (ms): 9921.29
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 183.38
|
||||
Median TTFT (ms): 177.92
|
||||
P99 TTFT (ms): 217.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.96
|
||||
Median TPOT (ms): 18.39
|
||||
P99 TPOT (ms): 19.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.57
|
||||
Median ITL (ms): 18.63
|
||||
P95 ITL (ms): 19.26
|
||||
P99 ITL (ms): 19.48
|
||||
Max ITL (ms): 24.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 Throughput Benchmark
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-Math-V2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 217.36
|
||||
Total input tokens: 301701
|
||||
Total input text tokens: 301701
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 187456
|
||||
Request throughput (req/s): 4.60
|
||||
Input token throughput (tok/s): 1388.05
|
||||
Output token throughput (tok/s): 866.67
|
||||
Peak output token throughput (tok/s): 2589.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 2254.72
|
||||
Concurrency: 89.81
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 19521.73
|
||||
Median E2E Latency (ms): 12076.76
|
||||
P90 E2E Latency (ms): 47248.87
|
||||
P99 E2E Latency (ms): 86862.79
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 790.40
|
||||
Median TTFT (ms): 456.81
|
||||
P99 TTFT (ms): 4223.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 106.52
|
||||
Median TPOT (ms): 107.24
|
||||
P99 TPOT (ms): 238.33
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 100.29
|
||||
Median ITL (ms): 38.34
|
||||
P95 ITL (ms): 237.00
|
||||
P99 ITL (ms): 347.49
|
||||
Max ITL (ms): 3642.56
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,255 @@
|
||||
---
|
||||
title: DeepSeek-OCR-2
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-OCR-2 with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
|
||||
---
|
||||
|
||||
import { DeepSeekOCR2Deployment } from '/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents).
|
||||
- **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version.
|
||||
- **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks.
|
||||
- **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details.
|
||||
|
||||
For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs.
|
||||
|
||||
<DeepSeekOCR2Deployment />
|
||||
|
||||
**Note**: DeepSeek-OCR-2 has ~3B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details.
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Single GPU Deployment:** DeepSeek-OCR-2 (~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving.
|
||||
- **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway).
|
||||
- **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`.
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
**OpenAI-compatible request example**
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "deepseek-ai/DeepSeek-OCR-2",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 512,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
**Reference**
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Recommended Prompts
|
||||
|
||||
The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).
|
||||
|
||||
**Structured document conversion** — extracts text while preserving layout:
|
||||
|
||||
```text Example
|
||||
<image>
|
||||
<|grounding|>Convert the document to markdown.
|
||||
```
|
||||
|
||||
**Free-form OCR** — extracts without layouts:
|
||||
|
||||
```text Example
|
||||
<image>
|
||||
Free OCR.
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 GPU (1x)
|
||||
- Model: DeepSeek-OCR-2
|
||||
- Tensor Parallelism: 1
|
||||
- sglang version: 0.0.0.dev1+g93fca0bbc
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models).
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-OCR-2 \
|
||||
--enable-multimodal \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-OCR-2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 3.54
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2710
|
||||
Request throughput (req/s): 2.83
|
||||
Input token throughput (tok/s): 557.53
|
||||
Output token throughput (tok/s): 787.10
|
||||
Peak output token throughput (tok/s): 818.00
|
||||
Peak concurrent requests: 5
|
||||
Total token throughput (tok/s): 1344.63
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 352.69
|
||||
Median E2E Latency (ms): 392.34
|
||||
P90 E2E Latency (ms): 540.64
|
||||
P99 E2E Latency (ms): 639.01
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 18.08
|
||||
Median TTFT (ms): 16.57
|
||||
P99 TTFT (ms): 25.67
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.18
|
||||
Median TPOT (ms): 1.21
|
||||
P99 TPOT (ms): 1.22
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.21
|
||||
Median ITL (ms): 1.21
|
||||
P95 ITL (ms): 1.28
|
||||
P99 ITL (ms): 1.44
|
||||
Max ITL (ms): 4.32
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-OCR-2 \
|
||||
--enable-multimodal \
|
||||
--tp 1 \
|
||||
--ep 1 \
|
||||
--dp 1 \
|
||||
--enable-dp-attention \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-OCR-2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 14.79
|
||||
Total input tokens: 301698
|
||||
Total input text tokens: 301698
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 185236
|
||||
Request throughput (req/s): 67.63
|
||||
Input token throughput (tok/s): 20402.54
|
||||
Output token throughput (tok/s): 12738.99
|
||||
Peak output token throughput (tok/s): 17508.00
|
||||
Peak concurrent requests: 187
|
||||
Total token throughput (tok/s): 33141.53
|
||||
Concurrency: 86.87
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1284.50
|
||||
Median E2E Latency (ms): 866.07
|
||||
P90 E2E Latency (ms): 3027.32
|
||||
P99 E2E Latency (ms): 5490.63
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 86.08
|
||||
Median TTFT (ms): 50.09
|
||||
P99 TTFT (ms): 613.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.79
|
||||
Median TPOT (ms): 6.54
|
||||
P99 TPOT (ms): 50.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.42
|
||||
Median ITL (ms): 4.64
|
||||
P95 ITL (ms): 23.65
|
||||
P99 ITL (ms): 39.62
|
||||
Max ITL (ms): 452.65
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,247 @@
|
||||
---
|
||||
title: DeepSeek-OCR
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-OCR with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced OCR**: High-accuracy text recognition from images and documents
|
||||
- **Multi-Modality**: Supports various image formats and document types
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
|
||||
|
||||
import { DeepSeekOCRDeployment } from "/src/snippets/autoregressive/deepseek-ocr-deployment.jsx";
|
||||
|
||||
<DeepSeekOCRDeployment />
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 OCR-Specific Prompts
|
||||
|
||||
DeepSeek-OCR accepts recommended prompts from the model card:
|
||||
|
||||
```text
|
||||
<image>
|
||||
<|grounding|>Convert the document to markdown.
|
||||
```
|
||||
|
||||
```text
|
||||
<image>
|
||||
Free OCR.
|
||||
```
|
||||
|
||||
**OpenAI-compatible image request example:**
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "deepseek-ai/DeepSeek-OCR",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "<image>\n<|grounding|>Convert the document to markdown."
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://example.com/your_image.jpg"}
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 512,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (1x)
|
||||
- Model: DeepSeek-OCR
|
||||
- Tensor Parallelism: 1
|
||||
- sglang version: 0.5.7
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-OCR \
|
||||
--tp 1 \
|
||||
--dtype float16 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-OCR \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 4.45
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2770
|
||||
Request throughput (req/s): 2.25
|
||||
Input token throughput (tok/s): 442.89
|
||||
Output token throughput (tok/s): 625.26
|
||||
Peak output token throughput (tok/s): 635.00
|
||||
Peak concurrent requests: 4
|
||||
Total token throughput (tok/s): 1068.16
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 443.32
|
||||
Median E2E Latency (ms): 493.29
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 21.59
|
||||
Median TTFT (ms): 20.89
|
||||
P99 TTFT (ms): 24.81
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.47
|
||||
Median TPOT (ms): 1.52
|
||||
P99 TPOT (ms): 1.53
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.52
|
||||
Median ITL (ms): 1.51
|
||||
P95 ITL (ms): 1.76
|
||||
P99 ITL (ms): 1.93
|
||||
Max ITL (ms): 8.28
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-OCR \
|
||||
--tp 1 \
|
||||
--ep 1 \
|
||||
--dp 1 \
|
||||
--enable-dp-attention \
|
||||
--dtype float16 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-OCR \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 16.24
|
||||
Total input tokens: 301698
|
||||
Total input text tokens: 301698
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 186927
|
||||
Request throughput (req/s): 61.59
|
||||
Input token throughput (tok/s): 18582.90
|
||||
Output token throughput (tok/s): 11602.84
|
||||
Peak output token throughput (tok/s): 15479.00
|
||||
Peak concurrent requests: 179
|
||||
Total token throughput (tok/s): 30185.75
|
||||
Concurrency: 85.53
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1388.60
|
||||
Median E2E Latency (ms): 901.43
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 73.36
|
||||
Median TTFT (ms): 50.21
|
||||
P99 TTFT (ms): 349.53
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.42
|
||||
Median TPOT (ms): 7.31
|
||||
P99 TPOT (ms): 27.99
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.04
|
||||
Median ITL (ms): 4.62
|
||||
P95 ITL (ms): 21.11
|
||||
P99 ITL (ms): 36.92
|
||||
Max ITL (ms): 172.15
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,993 @@
|
||||
---
|
||||
title: DeepSeek-R1
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-R1 reasoning model with SGLang - advanced step-by-step reasoning with FP8/FP4 quantization for NVIDIA and AMD GPUs."
|
||||
---
|
||||
|
||||
import { DeepSeekR1BasicDeployment } from '/src/snippets/autoregressive/deepseek-r1-basic-deployment.jsx';
|
||||
import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deepseek-r1-advanced-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X
|
||||
- **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X
|
||||
- **BF16 (upcast from FP8)**: [unsloth/DeepSeek-R1-0528-BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16)
|
||||
- **INT8 (channel-wise)**: [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8)
|
||||
- **W4A8**: [novita/Deepseek-R1-0528-W4AFP8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8)
|
||||
- **AWQ (4-bit)**: [QuixiAI/DeepSeek-R1-0528-AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ)
|
||||
- **MXFP4**: [amd/DeepSeek-R1-MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4)
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy.
|
||||
|
||||
<DeepSeekR1BasicDeployment />
|
||||
|
||||
### 3.2 Optimal Configurations
|
||||
|
||||
Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware.
|
||||
|
||||
<DeepSeekR1AdvancedDeployment />
|
||||
|
||||
### 3.3 Configuration Tips
|
||||
|
||||
DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.
|
||||
|
||||
**Recommended GPU configurations by weight type:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> The official DeepSeek-R1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
|
||||
|
||||
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels to avoid JIT overhead (~10 min):
|
||||
```bash
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code
|
||||
```
|
||||
|
||||
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node.
|
||||
|
||||
**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-0528",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--tool-call-parser deepseekv3 \
|
||||
--chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-0528",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments:
|
||||
🔧 Tool Call: None
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-0528",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
|
||||
|
||||
DeepSeek-R1 supports EAGLE-based Multi-Token Prediction (MTP), the same mechanism as DeepSeek-V3. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the complete launch command, flag reference, tuning guidance (`--speculative-num-steps`, `--speculative-eagle-topk`, `--max-running-requests`), and `bench_speculative.py` link. R1's speed benchmark commands that include `--speculative-*` flags use this mechanism.
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Limit the model's thinking token budget using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--tp 8 \
|
||||
--port 30000 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--enable-custom-logit-processor
|
||||
```
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: B200 GPU (8x)
|
||||
- Model: DeepSeek-R1-0528
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 40.00
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4205
|
||||
Request throughput (req/s): 0.25
|
||||
Input token throughput (tok/s): 152.52
|
||||
Output token throughput (tok/s): 105.24
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 257.76
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3998.40
|
||||
Median E2E Latency (ms): 3207.53
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 153.00
|
||||
Median TTFT (ms): 140.76
|
||||
P99 TTFT (ms): 214.66
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.16
|
||||
Median TPOT (ms): 9.15
|
||||
P99 TPOT (ms): 9.21
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.16
|
||||
Median ITL (ms): 9.15
|
||||
P95 ITL (ms): 9.47
|
||||
P99 ITL (ms): 9.63
|
||||
Max ITL (ms): 15.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 51.21
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40725
|
||||
Total generated tokens (retokenized): 40458
|
||||
Request throughput (req/s): 1.56
|
||||
Input token throughput (tok/s): 774.66
|
||||
Output token throughput (tok/s): 795.30
|
||||
Peak output token throughput (tok/s): 1088.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1569.96
|
||||
Concurrency: 13.93
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8918.33
|
||||
Median E2E Latency (ms): 9466.16
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 273.51
|
||||
Median TTFT (ms): 131.71
|
||||
P99 TTFT (ms): 839.57
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.56
|
||||
Median TPOT (ms): 17.46
|
||||
P99 TPOT (ms): 28.68
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 17.02
|
||||
Median ITL (ms): 14.70
|
||||
P95 ITL (ms): 16.41
|
||||
P99 ITL (ms): 112.38
|
||||
Max ITL (ms): 461.90
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 110.46
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252162
|
||||
Total generated tokens (retokenized): 251441
|
||||
Request throughput (req/s): 4.53
|
||||
Input token throughput (tok/s): 2261.80
|
||||
Output token throughput (tok/s): 2282.90
|
||||
Peak output token throughput (tok/s): 3900.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 4544.71
|
||||
Concurrency: 92.26
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20380.71
|
||||
Median E2E Latency (ms): 19391.65
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 563.14
|
||||
Median TTFT (ms): 147.62
|
||||
P99 TTFT (ms): 2632.11
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 40.11
|
||||
Median TPOT (ms): 41.98
|
||||
P99 TPOT (ms): 50.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 39.37
|
||||
Median ITL (ms): 26.36
|
||||
P95 ITL (ms): 98.16
|
||||
P99 ITL (ms): 150.08
|
||||
Max ITL (ms): 2052.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 411.34
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44452
|
||||
Total generated tokens (retokenized): 44390
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 14.83
|
||||
Output token throughput (tok/s): 108.07
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 122.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 41132.04
|
||||
Median E2E Latency (ms): 44288.71
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 125.76
|
||||
Median TTFT (ms): 126.19
|
||||
P99 TTFT (ms): 137.69
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.21
|
||||
Median TPOT (ms): 9.20
|
||||
P99 TPOT (ms): 9.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.23
|
||||
Median ITL (ms): 9.22
|
||||
P95 ITL (ms): 9.64
|
||||
P99 ITL (ms): 9.86
|
||||
Max ITL (ms): 15.18
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 348.93
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318226
|
||||
Total generated tokens (retokenized): 317630
|
||||
Request throughput (req/s): 0.23
|
||||
Input token throughput (tok/s): 113.69
|
||||
Output token throughput (tok/s): 912.02
|
||||
Peak output token throughput (tok/s): 1088.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 1025.70
|
||||
Concurrency: 14.07
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 61360.70
|
||||
Median E2E Latency (ms): 62071.20
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 176.02
|
||||
Median TTFT (ms): 153.75
|
||||
P99 TTFT (ms): 268.44
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 15.42
|
||||
Median TPOT (ms): 15.59
|
||||
P99 TPOT (ms): 16.07
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.39
|
||||
Median ITL (ms): 15.17
|
||||
P95 ITL (ms): 16.62
|
||||
P99 ITL (ms): 18.13
|
||||
Max ITL (ms): 226.59
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 589.31
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1300705
|
||||
Total generated tokens (retokenized): 1297658
|
||||
Request throughput (req/s): 0.54
|
||||
Input token throughput (tok/s): 269.70
|
||||
Output token throughput (tok/s): 2207.16
|
||||
Peak output token throughput (tok/s): 2944.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 2476.86
|
||||
Concurrency: 57.03
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 105032.36
|
||||
Median E2E Latency (ms): 108229.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 223.91
|
||||
Median TTFT (ms): 158.15
|
||||
P99 TTFT (ms): 474.86
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.94
|
||||
Median TPOT (ms): 26.72
|
||||
P99 TPOT (ms): 27.99
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.79
|
||||
Median ITL (ms): 25.37
|
||||
P95 ITL (ms): 26.70
|
||||
P99 ITL (ms): 105.49
|
||||
Max ITL (ms): 237.91
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 40.65
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4195
|
||||
Request throughput (req/s): 0.25
|
||||
Input token throughput (tok/s): 1031.65
|
||||
Output token throughput (tok/s): 103.56
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 1135.20
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4063.62
|
||||
Median E2E Latency (ms): 3296.13
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 165.91
|
||||
Median TTFT (ms): 154.96
|
||||
P99 TTFT (ms): 240.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.26
|
||||
Median TPOT (ms): 9.27
|
||||
P99 TPOT (ms): 9.42
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.28
|
||||
Median ITL (ms): 9.28
|
||||
P95 ITL (ms): 9.66
|
||||
P99 ITL (ms): 9.83
|
||||
Max ITL (ms): 14.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 56.71
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41589
|
||||
Total generated tokens (retokenized): 41490
|
||||
Request throughput (req/s): 1.41
|
||||
Input token throughput (tok/s): 5290.75
|
||||
Output token throughput (tok/s): 733.41
|
||||
Peak output token throughput (tok/s): 1024.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 6024.16
|
||||
Concurrency: 14.25
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10098.99
|
||||
Median E2E Latency (ms): 10623.46
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 486.80
|
||||
Median TTFT (ms): 189.59
|
||||
P99 TTFT (ms): 2138.73
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 19.06
|
||||
Median TPOT (ms): 19.23
|
||||
P99 TPOT (ms): 30.69
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.53
|
||||
Median ITL (ms): 15.63
|
||||
P95 ITL (ms): 16.64
|
||||
P99 ITL (ms): 109.71
|
||||
Max ITL (ms): 1471.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 115.55
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 169680
|
||||
Total generated tokens (retokenized): 169275
|
||||
Request throughput (req/s): 2.77
|
||||
Input token throughput (tok/s): 11024.93
|
||||
Output token throughput (tok/s): 1468.50
|
||||
Peak output token throughput (tok/s): 2254.00
|
||||
Peak concurrent requests: 70
|
||||
Total token throughput (tok/s): 12493.43
|
||||
Concurrency: 59.45
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 21465.98
|
||||
Median E2E Latency (ms): 20686.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 913.93
|
||||
Median TTFT (ms): 224.92
|
||||
P99 TTFT (ms): 6257.83
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 39.93
|
||||
Median TPOT (ms): 40.99
|
||||
P99 TPOT (ms): 60.91
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.83
|
||||
Median ITL (ms): 26.29
|
||||
P95 ITL (ms): 113.81
|
||||
P99 ITL (ms): 176.94
|
||||
Max ITL (ms): 5521.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
@@ -0,0 +1,637 @@
|
||||
---
|
||||
title: "DeepSeek-V3"
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-V3 MoE model with SGLang - efficient architecture with strong reasoning, coding, and tool-augmented capabilities."
|
||||
---
|
||||
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost.
|
||||
|
||||
Key highlights include:
|
||||
|
||||
- **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable.
|
||||
- **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies.
|
||||
- **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively.
|
||||
- **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-deployment.jsx";
|
||||
|
||||
<DeepSeekV3Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Recommended GPU configurations by weight type:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> The official DeepSeek-V3 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
|
||||
|
||||
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (~10 min):
|
||||
```bash
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code
|
||||
```
|
||||
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
|
||||
|
||||
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
|
||||
|
||||
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [Basic API Usage](../../../docs/get-started/quickstart)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To determine 15% of a number, follow these steps:
|
||||
|
||||
**Step 1: Understand the Problem**
|
||||
You need to find 15% of a given number. Let's assume the number is 240 for this example.
|
||||
|
||||
**Step 2: Convert the Percentage to a Decimal**
|
||||
To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100.
|
||||
|
||||
\[ 15\% = \frac{15}{100} = 0.15 \]
|
||||
|
||||
**Step 3: Multiply the Decimal by the Number**
|
||||
Now, multiply the decimal form of the percentage by the number you want to find the percentage of.
|
||||
|
||||
\[ 0.15 \times 240 \]
|
||||
|
||||
**Step 4: Perform the Multiplication**
|
||||
Calculate the product:
|
||||
|
||||
\[ 0.15 \times 240 = 36 \]
|
||||
|
||||
**Step 5: Conclusion**
|
||||
Therefore, 15% of 240 is:
|
||||
|
||||
\boxed{36}
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--tool-call-parser deepseekv3 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
|
||||
**Quick Test (curl):**
|
||||
|
||||
```shell Command
|
||||
curl "http://127.0.0.1:30000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"temperature": 0,
|
||||
"max_tokens": 100,
|
||||
"model": "deepseek-ai/DeepSeek-V3",
|
||||
"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
|
||||
"messages": [{"role": "user", "content": "How'\''s the weather in Beijing today?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
<Note>
|
||||
Use a low `temperature` (e.g. `0`) for more consistent tool call results. The `--chat-template` flag above provides an improved unified prompt for tool use.
|
||||
</Note>
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
Please attach the code blocks below to the previous Python script.
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
|
||||
|
||||
SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). With this optimization, decoding speed improves by up to **1.8×** at batch size 1 and **1.5×** at batch size 32 on H200 TP8.
|
||||
|
||||
**Enable with:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`.
|
||||
|
||||
<Note>
|
||||
For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs` to include your target batch sizes (default captured sizes for speculative decoding: 48).
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable.
|
||||
</Tip>
|
||||
|
||||
#### 4.2.4 MLA Optimizations
|
||||
|
||||
DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations:
|
||||
|
||||
- **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency.
|
||||
- **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, CutlassMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default.
|
||||
- **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8.
|
||||
- **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency.
|
||||
- **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only).
|
||||
|
||||
Overall, these optimizations achieve up to **7×** output throughput improvement vs. the baseline.
|
||||
|
||||
**Reference:** See [SGLang v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for details.
|
||||
|
||||
#### 4.2.5 Multi-Node Deployment
|
||||
|
||||
For multi-node serving and hardware-specific examples:
|
||||
|
||||
- [8× H200 / 4–8× B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
|
||||
- [8× MI300X](../../../docs/hardware-platforms/amd_gpu#running-deepseek-v3)
|
||||
- [2×8× H200 with Docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker)
|
||||
- [4×8× A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
|
||||
- [8× A100 AWQ](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
|
||||
- [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
|
||||
- [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
|
||||
- [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus)
|
||||
- [4× Atlas 800I A3 (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment)
|
||||
|
||||
**Blog references for large-scale deployment:**
|
||||
|
||||
- [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/))
|
||||
- [PD Disaggregation and Large-Scale Expert Parallelism on 96× H100](https://lmsys.org/blog/2025-05-05-large-scale-ep/)
|
||||
- [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/)
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (8x)
|
||||
- Model: DeepSeek-V3
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.7
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 81.27
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2774
|
||||
Request throughput (req/s): 0.12
|
||||
Input token throughput (tok/s): 24.27
|
||||
Output token throughput (tok/s): 34.26
|
||||
Peak output token throughput (tok/s): 65.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 58.52
|
||||
Concurrency: 1.00
|
||||
Accept length: 2.61
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8123.17
|
||||
Median E2E Latency (ms): 7982.65
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1080.76
|
||||
Median TTFT (ms): 1248.82
|
||||
P99 TTFT (ms): 1896.37
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.04
|
||||
Median TPOT (ms): 24.76
|
||||
P99 TPOT (ms): 32.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.41
|
||||
Median ITL (ms): 20.14
|
||||
P95 ITL (ms): 60.28
|
||||
P99 ITL (ms): 60.99
|
||||
Max ITL (ms): 61.49
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 406.16
|
||||
Total input tokens: 301701
|
||||
Total input text tokens: 301701
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 187542
|
||||
Request throughput (req/s): 2.46
|
||||
Input token throughput (tok/s): 742.81
|
||||
Output token throughput (tok/s): 463.80
|
||||
Peak output token throughput (tok/s): 1299.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 1206.61
|
||||
Concurrency: 87.53
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 35552.98
|
||||
Median E2E Latency (ms): 21466.07
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1521.51
|
||||
Median TTFT (ms): 476.80
|
||||
P99 TTFT (ms): 8329.50
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 214.73
|
||||
Median TPOT (ms): 152.00
|
||||
P99 TPOT (ms): 1155.85
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 182.10
|
||||
Median ITL (ms): 79.18
|
||||
P95 ITL (ms): 398.60
|
||||
P99 ITL (ms): 1488.96
|
||||
Max ITL (ms): 43465.60
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
- DeepSeek-V3
|
||||
```text Output
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 32.450 s
|
||||
Output throughput: 614.211 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
cd sglang
|
||||
bash benchmark/mmlu/download_data.sh
|
||||
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
- DeepSeek-V3
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.800
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.928
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.928
|
||||
subject: college_biology, #q:144, acc: 0.965
|
||||
subject: college_chemistry, #q:100, acc: 0.670
|
||||
subject: college_computer_science, #q:100, acc: 0.840
|
||||
subject: college_mathematics, #q:100, acc: 0.800
|
||||
subject: college_medicine, #q:173, acc: 0.861
|
||||
Total latency: 58.339
|
||||
Average accuracy: 0.871
|
||||
```
|
||||
@@ -0,0 +1,989 @@
|
||||
---
|
||||
title: DeepSeek-V3.1
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-V3.1 MoE model with SGLang - hybrid reasoning, improved tool calling, and agentic behavior for complex multi-step tasks."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture.
|
||||
|
||||
**[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases.
|
||||
|
||||
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { DeepSeekV31Deployment } from "/src/snippets/autoregressive/deepseek-v31-deployment.jsx";
|
||||
|
||||
<DeepSeekV31Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.
|
||||
|
||||
**Recommended GPU configurations by weight type:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> The official DeepSeek-V3.1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
|
||||
|
||||
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (~10 min):
|
||||
```bash
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code
|
||||
```
|
||||
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
|
||||
|
||||
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
|
||||
|
||||
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [Basic API Usage](../../../docs/get-started/quickstart)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.1-Terminus",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.
|
||||
|
||||
To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.
|
||||
|
||||
Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.
|
||||
|
||||
I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.
|
||||
|
||||
I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.
|
||||
|
||||
So, 15% of 240 is 36.
|
||||
|
||||
The problem says "step by step," so I should present it clearly.
|
||||
=============== Content =================
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
|
||||
2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
|
||||
3. Multiply the decimal by 240: \( 0.15 \times 240 \).
|
||||
4. Perform the multiplication:
|
||||
- \( 0.15 \times 200 = 30 \)
|
||||
- \( 0.15 \times 40 = 6 \)
|
||||
- Add the results: \( 30 + 6 = 36 \).
|
||||
|
||||
Alternatively, you can find 15% by breaking it into parts:
|
||||
- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
|
||||
- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
|
||||
- Add 10% and 5%: \( 24 + 12 = 36 \).
|
||||
|
||||
Thus, 15% of 240 is 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--tool-call-parser deepseekv31 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.1-Terminus",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.
|
||||
|
||||
I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.
|
||||
|
||||
The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
Please attach the code blocks below to the previous Python script.
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.1-Terminus",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "Currently, it is **22°C and sunny** in Beijing."
|
||||
```
|
||||
|
||||
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
|
||||
|
||||
DeepSeek-V3.1 shares the same architecture as DeepSeek-V3 and supports the same EAGLE-based MTP speculative decoding path. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the full configuration, tuning guidance, and `bench_speculative.py` reference. The `--speculative-num-steps`, `--speculative-eagle-topk`, and `--max-running-requests` recommendations apply equally to V3.1.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (8x)
|
||||
- Model: DeepSeek-V3.1-Terminus
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.1 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 106.24
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4201
|
||||
Request throughput (req/s): 0.09
|
||||
Input token throughput (tok/s): 57.43
|
||||
Output token throughput (tok/s): 39.72
|
||||
Peak output token throughput (tok/s): 43.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 97.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10620.29
|
||||
Median E2E Latency (ms): 8868.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 557.85
|
||||
Median TTFT (ms): 213.58
|
||||
P99 TTFT (ms): 1625.28
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 23.84
|
||||
Median TPOT (ms): 23.90
|
||||
P99 TPOT (ms): 24.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.90
|
||||
Median ITL (ms): 23.92
|
||||
P95 ITL (ms): 24.15
|
||||
P99 ITL (ms): 24.25
|
||||
Max ITL (ms): 25.44
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 107.71
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40625
|
||||
Request throughput (req/s): 0.74
|
||||
Input token throughput (tok/s): 368.28
|
||||
Output token throughput (tok/s): 378.84
|
||||
Peak output token throughput (tok/s): 508.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 747.12
|
||||
Concurrency: 13.72
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18473.65
|
||||
Median E2E Latency (ms): 19558.42
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 607.91
|
||||
Median TTFT (ms): 191.32
|
||||
P99 TTFT (ms): 2135.13
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 35.50
|
||||
Median TPOT (ms): 35.99
|
||||
P99 TPOT (ms): 43.62
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.10
|
||||
Median ITL (ms): 32.18
|
||||
P95 ITL (ms): 33.03
|
||||
P99 ITL (ms): 159.99
|
||||
Max ITL (ms): 453.99
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 207.65
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 251238
|
||||
Request throughput (req/s): 2.41
|
||||
Input token throughput (tok/s): 1203.15
|
||||
Output token throughput (tok/s): 1216.79
|
||||
Peak output token throughput (tok/s): 2100.00
|
||||
Peak concurrent requests: 106
|
||||
Total token throughput (tok/s): 2419.94
|
||||
Concurrency: 91.02
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 37800.20
|
||||
Median E2E Latency (ms): 35921.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 835.15
|
||||
Median TTFT (ms): 236.88
|
||||
P99 TTFT (ms): 2868.52
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 73.33
|
||||
Median TPOT (ms): 76.35
|
||||
P99 TPOT (ms): 97.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 73.30
|
||||
Median ITL (ms): 50.82
|
||||
P95 ITL (ms): 180.67
|
||||
P99 ITL (ms): 186.83
|
||||
Max ITL (ms): 1661.39
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 1097.29
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44313
|
||||
Request throughput (req/s): 0.01
|
||||
Input token throughput (tok/s): 5.56
|
||||
Output token throughput (tok/s): 40.52
|
||||
Peak output token throughput (tok/s): 43.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 46.08
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 109725.52
|
||||
Median E2E Latency (ms): 117748.67
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 156.67
|
||||
Median TTFT (ms): 156.19
|
||||
P99 TTFT (ms): 159.87
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.41
|
||||
Median TPOT (ms): 24.51
|
||||
P99 TPOT (ms): 24.96
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 24.65
|
||||
Median ITL (ms): 24.58
|
||||
P95 ITL (ms): 25.68
|
||||
P99 ITL (ms): 25.93
|
||||
Max ITL (ms): 29.80
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 775.02
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317426
|
||||
Request throughput (req/s): 0.10
|
||||
Input token throughput (tok/s): 51.18
|
||||
Output token throughput (tok/s): 410.70
|
||||
Peak output token throughput (tok/s): 512.00
|
||||
Peak concurrent requests: 18
|
||||
Total token throughput (tok/s): 461.89
|
||||
Concurrency: 13.86
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 134236.65
|
||||
Median E2E Latency (ms): 135181.28
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 214.35
|
||||
Median TTFT (ms): 194.12
|
||||
P99 TTFT (ms): 300.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.72
|
||||
Median TPOT (ms): 34.00
|
||||
P99 TPOT (ms): 34.75
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.69
|
||||
Median ITL (ms): 33.71
|
||||
P95 ITL (ms): 34.50
|
||||
P99 ITL (ms): 34.92
|
||||
Max ITL (ms): 164.76
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 1231.97
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1296845
|
||||
Request throughput (req/s): 0.26
|
||||
Input token throughput (tok/s): 129.01
|
||||
Output token throughput (tok/s): 1056.05
|
||||
Peak output token throughput (tok/s): 1472.00
|
||||
Peak concurrent requests: 67
|
||||
Total token throughput (tok/s): 1185.07
|
||||
Concurrency: 56.17
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 216256.25
|
||||
Median E2E Latency (ms): 224192.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 317.68
|
||||
Median TTFT (ms): 235.28
|
||||
P99 TTFT (ms): 649.39
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 53.30
|
||||
Median TPOT (ms): 55.10
|
||||
P99 TPOT (ms): 56.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 53.13
|
||||
Median ITL (ms): 52.95
|
||||
P95 ITL (ms): 56.23
|
||||
P99 ITL (ms): 181.04
|
||||
Max ITL (ms): 208.61
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 114.47
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4194
|
||||
Request throughput (req/s): 0.09
|
||||
Input token throughput (tok/s): 366.39
|
||||
Output token throughput (tok/s): 36.87
|
||||
Peak output token throughput (tok/s): 42.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 403.26
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 11442.86
|
||||
Median E2E Latency (ms): 9508.87
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 883.78
|
||||
Median TTFT (ms): 481.38
|
||||
P99 TTFT (ms): 2217.45
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.93
|
||||
Median TPOT (ms): 25.05
|
||||
P99 TPOT (ms): 26.11
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.08
|
||||
Median ITL (ms): 25.08
|
||||
P95 ITL (ms): 26.18
|
||||
P99 ITL (ms): 26.28
|
||||
Max ITL (ms): 27.41
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 162.33
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41443
|
||||
Request throughput (req/s): 0.49
|
||||
Input token throughput (tok/s): 1848.27
|
||||
Output token throughput (tok/s): 256.70
|
||||
Peak output token throughput (tok/s): 467.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 2104.97
|
||||
Concurrency: 14.52
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 29456.89
|
||||
Median E2E Latency (ms): 27628.16
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1784.30
|
||||
Median TTFT (ms): 1347.21
|
||||
P99 TTFT (ms): 5384.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 53.65
|
||||
Median TPOT (ms): 52.09
|
||||
P99 TPOT (ms): 74.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 53.23
|
||||
Median ITL (ms): 34.52
|
||||
P95 ITL (ms): 35.81
|
||||
P99 ITL (ms): 513.25
|
||||
Max ITL (ms): 2865.73
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 282.55
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169081
|
||||
Request throughput (req/s): 1.13
|
||||
Input token throughput (tok/s): 4508.6
|
||||
Output token throughput (tok/s): 601.67
|
||||
Peak output token throughput (tok/s): 1216
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 5110.27
|
||||
Concurrency: 59.81
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 52810.32
|
||||
Median E2E Latency (ms): 50981.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 786.69
|
||||
Median TTFT (ms): 499.38
|
||||
P99 TTFT (ms): 2925.98
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 97.93
|
||||
Median TPOT (ms): 103.45
|
||||
P99 TPOT (ms): 157.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 98.11
|
||||
Median ITL (ms): 55.7
|
||||
P95 ITL (ms): 240.71
|
||||
P99 ITL (ms): 1114.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,593 @@
|
||||
---
|
||||
title: DeepSeek-V4
|
||||
description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, and Pro."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
|
||||
|
||||
**NVIDIA GPUs**
|
||||
|
||||
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000).
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
**AMD GPUs (ROCm)**
|
||||
|
||||
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
|
||||
|
||||
For example:
|
||||
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710`
|
||||
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi30x-20260623`
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
|
||||
|
||||
docker run \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
--group-add video \
|
||||
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
|
||||
--shm-size 32g --ipc=host \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Note>
|
||||
For a runnable end-to-end example, see the [DeepSeek-V4-Flash demo notebook](https://github.com/sgl-project/sglang/blob/main/docs/demo/deepseek_v4_flash.ipynb).
|
||||
</Note>
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
The knobs come in two flavors:
|
||||
|
||||
- **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
|
||||
- **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash refresh adds a checkpoint with a bundled DSpark draft head:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "30%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "40%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Total params</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Active (MoE)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash">DeepSeek-V4-Flash</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>284B</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731">DeepSeek-V4-Flash-0731</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>304</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.6T</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>49B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling.
|
||||
|
||||
**Highlights:** hybrid CSA + HCA attention (~27% inference FLOPs / ~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar.
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=1.0`.
|
||||
|
||||
**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
{/* TODO: expand this section as more recipes are validated end-to-end. */}
|
||||
|
||||
**Concurrency & DeepEP dispatch buffer**
|
||||
|
||||
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together.
|
||||
|
||||
The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.
|
||||
|
||||
**Speculative decoding**
|
||||
|
||||
The original Flash and Pro recipes use EAGLE. Flash Official (0731) uses the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes.
|
||||
|
||||
For the original Flash and Pro checkpoints:
|
||||
|
||||
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
|
||||
- `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
|
||||
- `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
|
||||
- MTP runs on the v2 speculative path.
|
||||
|
||||
**Compressed attention state dtype**
|
||||
|
||||
DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path:
|
||||
|
||||
```bash Command
|
||||
SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash \
|
||||
<other args>
|
||||
```
|
||||
|
||||
This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior.
|
||||
|
||||
**EPLB + Waterfill (Experimental)**
|
||||
|
||||
For recorded/static EPLB reproduction, first record an expert-distribution file by following
|
||||
[Capture expert selection distribution in MoE models](../../../docs/basic_usage/native_api.mdx#capture-expert-selection-distribution-in-moe-models).
|
||||
For reproduction runs, use the generated `expert_distribution_recorder_*.pt` as
|
||||
the initial expert location. **Please checkout to latest main branch for this feature.**
|
||||
|
||||
For non-PD reproduction, use:
|
||||
|
||||
```bash Command
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
|
||||
--enable-waterfill
|
||||
```
|
||||
|
||||
For PD-Disagg reproduction, use `normal` mode on the prefill server and
|
||||
`low_latency` mode on the decode server. Add the same `--init-expert-location`
|
||||
flag to both commands:
|
||||
|
||||
```bash Command
|
||||
# prefill
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal \
|
||||
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
|
||||
--enable-waterfill
|
||||
|
||||
# decode
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
|
||||
--enable-waterfill
|
||||
```
|
||||
|
||||
You can also add `--ep-num-redundant-experts` and `--eplb-algorithm` to customize
|
||||
EPLB placement.
|
||||
|
||||
Waterfill also supports MegaMOE. Use `--moe-a2a-backend megamoe
|
||||
--enable-waterfill` to keep the MegaMOE backend while applying Waterfill to the
|
||||
fused shared expert slot.
|
||||
|
||||
**FP4 Indexer (Experimental)**
|
||||
|
||||
DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.
|
||||
|
||||
```bash Command
|
||||
# Please use the latest main branch for this feature.
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash \
|
||||
--tp 4 \
|
||||
--moe-runner-backend flashinfer_mxfp4 \
|
||||
--enable-deepseek-v4-fp4-indexer
|
||||
```
|
||||
|
||||
**NVFP4 Hybrid Checkpoints**
|
||||
|
||||
The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and
|
||||
[`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints
|
||||
quantize MoE experts to **NVFP4** while keeping attention and dense layers in
|
||||
**FP8**. It requires `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided.
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
Requires Blackwell (SM100+). The MTP layer in this checkpoint stays
|
||||
MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path
|
||||
automatically.
|
||||
|
||||
<a id="hopper-note" />
|
||||
|
||||
**Hopper (H100 / H200) note**
|
||||
|
||||
Two options are available for running DeepSeek-V4 on Hopper:
|
||||
|
||||
- **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
|
||||
- **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).
|
||||
|
||||
On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below.
|
||||
|
||||
PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1`
|
||||
(or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake
|
||||
can discover the IB HCAs; without IB exposure mooncake silently falls back to
|
||||
TCP, which can lead to garbled KV transfer on large checkpoints.
|
||||
|
||||
**RTX PRO 6000 (SM120 / Blackwell Desktop) note**
|
||||
|
||||
RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner.
|
||||
V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes.
|
||||
HiCache and MegaMoE are **not** supported on RTX PRO 6000.
|
||||
|
||||
**AMD (MI300X / MI355X) note**
|
||||
|
||||
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
|
||||
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
|
||||
- **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`.
|
||||
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
|
||||
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
|
||||
|
||||
**MegaMoE**
|
||||
|
||||
MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput
|
||||
on MoE layers. To enable it, use the **MegaMoE** chip in the Playground
|
||||
below — the playground will swap `--moe-a2a-backend deepep` for
|
||||
`--moe-a2a-backend megamoe` and add the relevant env vars automatically.
|
||||
|
||||
Two variants are exposed:
|
||||
- **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations).
|
||||
- **W4A4** — adds `SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1` and
|
||||
`SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1` to run the custom W4A4
|
||||
kernel (FP4 activations). Higher throughput with negligible accuracy drop
|
||||
(~89.5 GPQA on Pro).
|
||||
|
||||
Notes:
|
||||
- The W4A8 / W4A4 variants above are **Blackwell-only** (B200 / B300 / GB200 / GB300). On **Hopper (SM90, H100 / H200)** use the all-FP8 MegaMoE path described below instead.
|
||||
- MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it.
|
||||
- When running MegaMoE, don't set `--moe-runner-backend` manually.
|
||||
- Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).
|
||||
|
||||
**SM90 (Hopper) FP8 MegaMoE (Experimental)**
|
||||
|
||||
On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the
|
||||
DeepGEMM `mega_moe` runner for higher long-context / large-decode throughput on
|
||||
the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts
|
||||
stay in **FP8** — keep `SGLANG_DSV4_FP4_EXPERTS=0`. It requires a `sgl-deep-gemm`
|
||||
build with SM90 FP8 MegaMoE support. **Please use the latest image for this
|
||||
feature.**
|
||||
|
||||
Enable the MegaMoE path with `--moe-a2a-backend megamoe` — or equivalently set
|
||||
`SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1`, which auto-configures the same backend:
|
||||
|
||||
```bash Command
|
||||
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
|
||||
SGLANG_DSV4_FP4_EXPERTS=0 \
|
||||
sglang serve \
|
||||
--model-path sgl-project/DeepSeek-V4-Flash-FP8 \
|
||||
--tp 8 \
|
||||
--moe-a2a-backend megamoe \
|
||||
--chunked-prefill-size 4096
|
||||
```
|
||||
|
||||
`SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` caps the number of tokens
|
||||
the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used
|
||||
for batches at or below this cap. The right value depends on your parallelism /
|
||||
token-split scheme, and larger values reserve more HBM.
|
||||
|
||||
**GB300 PD-Disagg cross-pod MNNVL**
|
||||
|
||||
On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may
|
||||
fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If
|
||||
this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`
|
||||
to both prefill and decode `sglang serve` commands.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
|
||||
<Accordion title="Streaming with Thinking Process (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:
|
||||
|
||||
**Step 1: Understand the meaning of percent**
|
||||
"Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).
|
||||
|
||||
**Step2: Convert the percentage to a decimal or fraction**
|
||||
\( 15\% = \frac{15}{100} = 0.15 \)
|
||||
|
||||
**Step3: Multiply by the given number**
|
||||
Multiply the decimal form by 240:
|
||||
\( 0.15 \times 240 \)
|
||||
|
||||
**Step4: Perform the multiplication**
|
||||
\( 0.15 \times 240 = 36 \)
|
||||
|
||||
**Answer:** 15% of 240 is **36**.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
<Accordion title="Python Example with Thinking Process">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if getattr(delta, "tool_calls", None):
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {"name": None, "arguments": ""}
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]["name"] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>
|
||||
|
||||
<｜DSML｜tool_calls>
|
||||
<｜DSML｜invoke name="get_weather">
|
||||
<｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
|
||||
</｜DSML｜invoke>
|
||||
</｜DSML｜tool_calls>
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers.
|
||||
|
||||
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**:
|
||||
|
||||
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only.
|
||||
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
|
||||
|
||||
For AMD devices,
|
||||
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. Use `direct` IO backend + `page_first_direct` or `layer-first` mem-layout.
|
||||
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
|
||||
|
||||
The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow.
|
||||
|
||||
For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache).
|
||||
|
||||
### 3.4 DSpark (Speculative Decoding)
|
||||
|
||||
Flash Official (0731) bundles a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`.
|
||||
|
||||
Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint.
|
||||
|
||||
The verified 4×GB300 FP4 low-latency command is:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
|
||||
--tp 4 \
|
||||
--moe-runner-backend flashinfer_mxfp4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
--mem-fraction-static 0.90 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16.
|
||||
|
||||
**Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. The current 0731 checkpoint resolves to five proposed tokens, which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five.
|
||||
|
||||
Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone.
|
||||
|
||||
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
|
||||
|
||||
DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; turn DSpark off before selecting a prefill or decode role. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Ernie4.5-VL
|
||||
metatags:
|
||||
description: "Deploy Ernie4.5-VL vision-language model with SGLang - community contribution guide for Baidu's multimodal model."
|
||||
---
|
||||
|
||||
## 📝 Community Contribution Welcome
|
||||
|
||||
This guide is currently under development. We welcome community contributions!
|
||||
|
||||
If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation.
|
||||
|
||||
## 🚀 How to Contribute
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
git checkout -b add-ernie4-5-vl-guide
|
||||
# Edit this file and submit a PR
|
||||
```
|
||||
|
||||
## 📚 Reference
|
||||
|
||||
- [GLM-4.6V](../GLM/GLM-4.6V)
|
||||
|
||||
---
|
||||
|
||||
**Let's build this together!** 🌟
|
||||
@@ -0,0 +1,696 @@
|
||||
---
|
||||
title: Ernie4.5
|
||||
metatags:
|
||||
description: "Deploy Ernie4.5 with SGLang - community contribution guide for Baidu's Ernie 4.5 model deployment."
|
||||
---
|
||||
|
||||
import { Ernie45Deployment } from '/src/snippets/autoregressive/ernie-45-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios.
|
||||
|
||||
ERNIE-4.5 delivers advanced features as below:
|
||||
- **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks.
|
||||
- **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
|
||||
- **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
|
||||
- **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Ernie45Deployment />
|
||||
|
||||
## 4. API Usage
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT.
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
**Basic Python Client Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/ERNIE-4.5-21B-A3B-PT",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is artificial intelligence?"}
|
||||
],
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
```text Output
|
||||
**Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence.
|
||||
|
||||
### Key Characteristics of AI:
|
||||
...
|
||||
|
||||
### In Summary:
|
||||
AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility.
|
||||
```
|
||||
|
||||
**Streaming Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/ERNIE-4.5-21B-A3B-PT",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain quantum computing in simple terms."}
|
||||
],
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Sure! Here’s a simple explanation of quantum computing:
|
||||
|
||||
### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules**
|
||||
|
||||
1. **Classic vs. Quantum Computers**
|
||||
- **Normal computers** use **bits** (0s and 1s) to store and process information.
|
||||
- **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**).
|
||||
|
||||
2. **Superposition: The Magic Behind Speed**
|
||||
- A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air.
|
||||
- Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems.
|
||||
|
||||
3. **Entanglement: Making Qubits Link**
|
||||
- When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein).
|
||||
- Entanglement allows quantum computers to process information in **very efficient ways**.
|
||||
|
||||
4. **What Quantum Computers Are Good At**
|
||||
- **Cracking encryption** (like RSA).
|
||||
- **Factoring large numbers** (used in encryption and cryptography).
|
||||
- **Searching unsorted databases** (way faster than classical computers).
|
||||
- **Simulating quantum systems** (like molecules for drug discovery).
|
||||
- **Optimizing problems** (like logistics or finance).
|
||||
|
||||
5. **Challenges & Current State**
|
||||
- Qubits are **fragile** and easily disturbed (called **decoherence**).
|
||||
- Engineers are working to keep qubits stable long enough to do useful calculations.
|
||||
- Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers.
|
||||
|
||||
### **Final Thought**
|
||||
Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity.
|
||||
|
||||
Would you like an example of how a quantum computer might solve a problem? 😊
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (1x)
|
||||
- Model: ERNIE-4.5-21B-A3B-PT
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 58.72
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4219
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 103.90
|
||||
Output token throughput (tok/s): 71.87
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 175.77
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5869.86
|
||||
Median E2E Latency (ms): 1870.80
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4152.58
|
||||
Median TTFT (ms): 36.81
|
||||
P99 TTFT (ms): 37498.23
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.07
|
||||
Median TPOT (ms): 4.09
|
||||
P99 TPOT (ms): 4.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.08
|
||||
Median ITL (ms): 4.08
|
||||
P95 ITL (ms): 4.14
|
||||
P99 ITL (ms): 4.20
|
||||
Max ITL (ms): 4.67
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 34.30
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40773
|
||||
Request throughput (req/s): 2.33
|
||||
Input token throughput (tok/s): 1156.62
|
||||
Output token throughput (tok/s): 1189.77
|
||||
Peak output token throughput (tok/s): 1392.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2346.39
|
||||
Concurrency: 14.14
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6060.62
|
||||
Median E2E Latency (ms): 6496.70
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 78.90
|
||||
Median TTFT (ms): 45.90
|
||||
P99 TTFT (ms): 234.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.99
|
||||
Median TPOT (ms): 12.16
|
||||
P99 TPOT (ms): 14.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.75
|
||||
Median ITL (ms): 11.48
|
||||
P95 ITL (ms): 12.24
|
||||
P99 ITL (ms): 34.85
|
||||
Max ITL (ms): 105.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 66.63
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252449
|
||||
Request throughput (req/s): 7.50
|
||||
Input token throughput (tok/s): 3749.79
|
||||
Output token throughput (tok/s): 3792.28
|
||||
Peak output token throughput (tok/s): 4902.00
|
||||
Peak concurrent requests: 113
|
||||
Total token throughput (tok/s): 7542.06
|
||||
Concurrency: 90.33
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 12036.90
|
||||
Median E2E Latency (ms): 11782.16
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 104.86
|
||||
Median TTFT (ms): 84.62
|
||||
P99 TTFT (ms): 297.85
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 23.89
|
||||
Median TPOT (ms): 24.62
|
||||
P99 TPOT (ms): 26.91
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.66
|
||||
Median ITL (ms): 20.48
|
||||
P95 ITL (ms): 45.57
|
||||
P99 ITL (ms): 54.31
|
||||
Max ITL (ms): 185.12
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 185.11
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44423
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 32.96
|
||||
Output token throughput (tok/s): 240.19
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 273.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18508.84
|
||||
Median E2E Latency (ms): 19866.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 32.59
|
||||
Median TTFT (ms): 32.14
|
||||
P99 TTFT (ms): 38.58
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.13
|
||||
Median TPOT (ms): 4.13
|
||||
P99 TPOT (ms): 4.20
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.16
|
||||
Median ITL (ms): 4.12
|
||||
P95 ITL (ms): 4.31
|
||||
P99 ITL (ms): 4.36
|
||||
Max ITL (ms): 7.28
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 263.48
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317984
|
||||
Request throughput (req/s): 0.30
|
||||
Input token throughput (tok/s): 150.55
|
||||
Output token throughput (tok/s): 1208.09
|
||||
Peak output token throughput (tok/s): 1408.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 1358.64
|
||||
Concurrency: 14.35
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 47249.55
|
||||
Median E2E Latency (ms): 47828.67
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 62.77
|
||||
Median TTFT (ms): 57.10
|
||||
P99 TTFT (ms): 93.70
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.92
|
||||
Median TPOT (ms): 12.09
|
||||
P99 TPOT (ms): 12.50
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.86
|
||||
Median ITL (ms): 12.04
|
||||
P95 ITL (ms): 12.68
|
||||
P99 ITL (ms): 13.61
|
||||
Max ITL (ms): 39.94
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 428.30
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1299877
|
||||
Request throughput (req/s): 0.75
|
||||
Input token throughput (tok/s): 371.09
|
||||
Output token throughput (tok/s): 3037.63
|
||||
Peak output token throughput (tok/s): 3880.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 3408.73
|
||||
Concurrency: 57.08
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 76392.58
|
||||
Median E2E Latency (ms): 79698.73
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 92.79
|
||||
Median TTFT (ms): 78.71
|
||||
P99 TTFT (ms): 168.89
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.81
|
||||
Median TPOT (ms): 19.15
|
||||
P99 TPOT (ms): 19.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.77
|
||||
Median ITL (ms): 18.77
|
||||
P95 ITL (ms): 19.86
|
||||
P99 ITL (ms): 42.08
|
||||
Max ITL (ms): 74.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.3.1 Low Concurrency
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 18.59
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4216
|
||||
Request throughput (req/s): 0.54
|
||||
Input token throughput (tok/s): 2256.43
|
||||
Output token throughput (tok/s): 227.04
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 2483.46
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1856.72
|
||||
Median E2E Latency (ms): 1513.87
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 86.66
|
||||
Median TTFT (ms): 72.30
|
||||
P99 TTFT (ms): 167.13
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.19
|
||||
Median TPOT (ms): 4.22
|
||||
P99 TPOT (ms): 4.30
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.20
|
||||
Median ITL (ms): 4.23
|
||||
P95 ITL (ms): 4.34
|
||||
P99 ITL (ms): 4.42
|
||||
Max ITL (ms): 5.68
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 40.25
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41646
|
||||
Request throughput (req/s): 1.99
|
||||
Input token throughput (tok/s): 7454.72
|
||||
Output token throughput (tok/s): 1035.37
|
||||
Peak output token throughput (tok/s): 1310.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 8490.09
|
||||
Concurrency: 14.37
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7229.56
|
||||
Median E2E Latency (ms): 7578.95
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 137.38
|
||||
Median TTFT (ms): 122.59
|
||||
P99 TTFT (ms): 485.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.04
|
||||
Median TPOT (ms): 14.24
|
||||
P99 TPOT (ms): 20.77
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.64
|
||||
Median ITL (ms): 12.36
|
||||
P95 ITL (ms): 14.72
|
||||
P99 ITL (ms): 57.39
|
||||
Max ITL (ms): 411.31
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 78.33
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169888
|
||||
Request throughput (req/s): 4.09
|
||||
Input token throughput (tok/s): 16262.33
|
||||
Output token throughput (tok/s): 2170.20
|
||||
Peak output token throughput (tok/s): 3005.00
|
||||
Peak concurrent requests: 73
|
||||
Total token throughput (tok/s): 18432.53
|
||||
Concurrency: 58.79
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 14392.52
|
||||
Median E2E Latency (ms): 14460.70
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 184.82
|
||||
Median TTFT (ms): 155.24
|
||||
P99 TTFT (ms): 379.82
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 26.97
|
||||
Median TPOT (ms): 28.31
|
||||
P99 TPOT (ms): 33.61
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 26.79
|
||||
Median ITL (ms): 20.55
|
||||
P95 ITL (ms): 47.55
|
||||
P99 ITL (ms): 145.64
|
||||
Max ITL (ms): 287.62
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
- ERNIE-4.5-21B-A3B-PT
|
||||
```
|
||||
Accuracy: 0.865
|
||||
Invalid: 0.000
|
||||
Latency: 21.669 s
|
||||
Output throughput: 10359.790 token/s
|
||||
```
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: Chroma-1.0
|
||||
metatags:
|
||||
description: "Deploy Chroma-1.0 end-to-end speech conversation model with SGLang - real-time speech generation, voice cloning, and speech reasoning."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities:
|
||||
|
||||
- **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios.
|
||||
- **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics.
|
||||
- **End-to-End Architecture**: Provides a complete processing workflow from speech to speech.
|
||||
- **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content.
|
||||
|
||||
## 2. Architecture Overview
|
||||
|
||||
**Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by:
|
||||
|
||||
1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops.
|
||||
2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations.
|
||||
3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable.
|
||||
|
||||
Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported.
|
||||
|
||||
- **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic)
|
||||
- **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable)
|
||||
|
||||
## 3. Installation & Setup
|
||||
|
||||
We recommend following these steps to set up the environment and prepare the model.
|
||||
|
||||
### Step 1: Get the Docker Image
|
||||
|
||||
Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured.
|
||||
|
||||
```bash Command
|
||||
docker pull flashlabs/chroma:latest
|
||||
```
|
||||
|
||||
### Step 2: Download Model Weights
|
||||
|
||||
Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods:
|
||||
|
||||
**Method 1: Using Python (Recommended)**
|
||||
|
||||
```bash Command
|
||||
huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B
|
||||
```
|
||||
|
||||
**Method 2: Using Git Clone**
|
||||
|
||||
Make sure you have Git LFS installed before cloning.
|
||||
|
||||
```bash Command
|
||||
# Install Git LFS first
|
||||
git lfs install
|
||||
|
||||
# Clone the repository
|
||||
git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B
|
||||
```
|
||||
|
||||
### Step 3: Download Chroma Codes (SGLang version)
|
||||
|
||||
```bash Command
|
||||
git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git
|
||||
|
||||
cd Chroma-SGLang
|
||||
```
|
||||
|
||||
### Step 4: Run the Server
|
||||
|
||||
```bash Command
|
||||
docker run -d \
|
||||
--gpus all \
|
||||
-p 8000:8000 \
|
||||
-w /app/Chroma-SGLang \
|
||||
-v "your_Chroma-SGLang_path":/app/Chroma-SGLang \
|
||||
-v "your_chroma_path":/model \
|
||||
-e CHROMA_MODEL_PATH=/model \
|
||||
-e DP_SIZE="1" \
|
||||
flashlabs/chroma:latest \
|
||||
/opt/conda/bin/python -m uvicorn api_server:app \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--workers 1
|
||||
```
|
||||
|
||||
or run simply the following one line command
|
||||
|
||||
```bash Command
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
## 5. Client Usage Example
|
||||
|
||||
Once the server is running, you can interact with it using HTTP requests.
|
||||
|
||||
### Python Client
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
import base64
|
||||
|
||||
url = "http://localhost:8000/v1/chat/completions"
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
payload = {
|
||||
"model": "chroma",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are Chroma, a voice agent developed by FlashLabs."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "audio", "audio": "assets/question_audio.wav"}
|
||||
]
|
||||
}
|
||||
],
|
||||
"max_tokens": 1000,
|
||||
"return_audio": True
|
||||
}
|
||||
|
||||
response = requests.post(url, json=payload, headers=headers)
|
||||
result = response.json()
|
||||
|
||||
if result.get("audio"):
|
||||
audio_data = base64.b64decode(result["audio"])
|
||||
with open("output.wav", "wb") as f:
|
||||
f.write(audio_data)
|
||||
print("Audio saved to output.wav")
|
||||
```
|
||||
|
||||
### OpenAI SDK Compatible Example
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="dummy",
|
||||
base_url="http://localhost:8000/v1"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="chroma",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "audio", "audio": "assets/question_audio.wav"}
|
||||
]
|
||||
}
|
||||
],
|
||||
extra_body={
|
||||
"prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.",
|
||||
"prompt_audio": "assets/ref_audio.wav",
|
||||
"return_audio": True
|
||||
}
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
### CLI (cURL)
|
||||
|
||||
```bash Command
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "chroma",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are Chroma, a voice agent developed by FlashLabs."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio",
|
||||
"audio": "assets/question_audio.wav"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"max_tokens": 1000,
|
||||
"return_audio": true
|
||||
}' | jq -r '.audio' | base64 -d > output.wav
|
||||
```
|
||||
@@ -0,0 +1,516 @@
|
||||
---
|
||||
title: GLM-4.5
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5 with SGLang on AMD GPUs - advanced reasoning, function calling, BF16/FP8 quantization options."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM45Deployment } from "/src/snippets/autoregressive/glm-45-deployment.jsx";
|
||||
|
||||
<GLM45Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.5 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
|
||||
- Model: GLM-4.5
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
@@ -0,0 +1,582 @@
|
||||
---
|
||||
title: GLM-4.5V
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5V vision-language model with SGLang - SOTA multimodal performance, 64K context, image reasoning and video understanding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
|
||||
|
||||
GLM-4.5V introduces several key features:
|
||||
|
||||
- **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection.
|
||||
- **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences.
|
||||
- **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces.
|
||||
- **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context.
|
||||
- **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
|
||||
|
||||
import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deployment.jsx";
|
||||
|
||||
<GLM45VDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`.
|
||||
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
|
||||
- **Hardware Notes:**
|
||||
- **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
- **Additional Multimodal Parameters:**
|
||||
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
GLM-4.5V supports both image and video inputs. Here's a basic example with image input:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image in detail."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.37s
|
||||
Generated text: Auntie Anne's
|
||||
|
||||
CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
GLM-4.5V can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.86s
|
||||
Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot).
|
||||
```
|
||||
|
||||
**Video Input Example:**
|
||||
|
||||
GLM-4.5V supports video understanding by processing video URLs:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe what happens in this video."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- For video processing, ensure you have sufficient context length configured (up to 64K tokens)
|
||||
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
|
||||
- You can also provide local file paths using `file://` protocol
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.89s
|
||||
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
|
||||
```
|
||||
|
||||
#### 4.2.2 Thinking Mode
|
||||
|
||||
GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
**Disable Thinking Mode:**
|
||||
|
||||
To disable thinking mode for a specific request:
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[{"role": "user", "content": "What is the capital of France?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
GLM-4.5V supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Beyond enabling/disabling the full reasoning mode (section 4.2.2), you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[{"role": "user", "content": "Describe this image briefly."}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.1.1 MMMU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```text Output
|
||||
Benchmark time: 616.6163094160147
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.867, 'num': 30},
|
||||
'Agriculture': {'acc': 0.567, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.667, 'num': 30},
|
||||
'Art': {'acc': 0.667, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.8, 'num': 30},
|
||||
'Biology': {'acc': 0.6, 'num': 30},
|
||||
'Chemistry': {'acc': 0.533, 'num': 30},
|
||||
'Clinical_Medicine': {'acc': 0.667, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.8, 'num': 30},
|
||||
'Design': {'acc': 0.867, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30},
|
||||
'Economics': {'acc': 0.833, 'num': 30},
|
||||
'Electronics': {'acc': 0.433, 'num': 30},
|
||||
'Energy_and_Power': {'acc': 0.733, 'num': 30},
|
||||
'Finance': {'acc': 0.767, 'num': 30},
|
||||
'Geography': {'acc': 0.667, 'num': 30},
|
||||
'History': {'acc': 0.8, 'num': 30},
|
||||
'Literature': {'acc': 0.9, 'num': 30},
|
||||
'Manage': {'acc': 0.733, 'num': 30},
|
||||
'Marketing': {'acc': 0.9, 'num': 30},
|
||||
'Materials': {'acc': 0.567, 'num': 30},
|
||||
'Math': {'acc': 0.8, 'num': 30},
|
||||
'Mechanical_Engineering': {'acc': 0.767, 'num': 30},
|
||||
'Music': {'acc': 0.3, 'num': 30},
|
||||
'Overall': {'acc': 0.732, 'num': 900},
|
||||
'Overall-Art and Design': {'acc': 0.683, 'num': 120},
|
||||
'Overall-Business': {'acc': 0.82, 'num': 150},
|
||||
'Overall-Health and Medicine': {'acc': 0.787, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120},
|
||||
'Overall-Science': {'acc': 0.707, 'num': 150},
|
||||
'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210},
|
||||
'Pharmacy': {'acc': 0.9, 'num': 30},
|
||||
'Physics': {'acc': 0.933, 'num': 30},
|
||||
'Psychology': {'acc': 0.767, 'num': 30},
|
||||
'Public_Health': {'acc': 0.9, 'num': 30},
|
||||
'Sociology': {'acc': 0.667, 'num': 30}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.732
|
||||
```
|
||||
@@ -0,0 +1,914 @@
|
||||
---
|
||||
title: GLM-4.6
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6 with SGLang - 200K context window, superior coding, advanced reasoning, and enhanced agentic capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows:
|
||||
|
||||
- **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks.
|
||||
- **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages.
|
||||
- **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability.
|
||||
- **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks.
|
||||
- **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios.
|
||||
|
||||
For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM46Deployment } from "/src/snippets/autoregressive/glm-46-deployment.jsx";
|
||||
|
||||
<GLM46Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.6 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
|
||||
- Model: GLM-4.6
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 63.82
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4209
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 95.60
|
||||
Output token throughput (tok/s): 65.97
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 161.57
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6379.24
|
||||
Median E2E Latency (ms): 5085.00
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 155.57
|
||||
Median TTFT (ms): 149.79
|
||||
P99 TTFT (ms): 207.69
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.81
|
||||
Median TPOT (ms): 14.80
|
||||
P99 TPOT (ms): 14.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.82
|
||||
Median ITL (ms): 14.82
|
||||
P95 ITL (ms): 15.17
|
||||
P99 ITL (ms): 15.36
|
||||
Max ITL (ms): 25.05
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 72.06
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40725
|
||||
Total generated tokens (retokenized): 40672
|
||||
Request throughput (req/s): 1.11
|
||||
Input token throughput (tok/s): 550.47
|
||||
Output token throughput (tok/s): 565.14
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1115.61
|
||||
Concurrency: 13.71
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 12348.93
|
||||
Median E2E Latency (ms): 13164.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 196.08
|
||||
Median TTFT (ms): 155.22
|
||||
P99 TTFT (ms): 377.98
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.24
|
||||
Median TPOT (ms): 24.55
|
||||
P99 TPOT (ms): 30.42
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.92
|
||||
Median ITL (ms): 21.40
|
||||
P95 ITL (ms): 22.49
|
||||
P99 ITL (ms): 123.83
|
||||
Max ITL (ms): 486.54
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 138.50
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252162
|
||||
Total generated tokens (retokenized): 251841
|
||||
Request throughput (req/s): 3.61
|
||||
Input token throughput (tok/s): 1803.78
|
||||
Output token throughput (tok/s): 1820.61
|
||||
Peak output token throughput (tok/s): 2900.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3624.40
|
||||
Concurrency: 90.91
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 25183.97
|
||||
Median E2E Latency (ms): 23968.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 337.77
|
||||
Median TTFT (ms): 180.65
|
||||
P99 TTFT (ms): 906.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 49.97
|
||||
Median TPOT (ms): 52.20
|
||||
P99 TPOT (ms): 61.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 49.36
|
||||
Median ITL (ms): 35.05
|
||||
P95 ITL (ms): 124.91
|
||||
P99 ITL (ms): 187.69
|
||||
Max ITL (ms): 440.34
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 666.64
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44452
|
||||
Total generated tokens (retokenized): 44387
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 9.15
|
||||
Output token throughput (tok/s): 66.68
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 75.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 66661.35
|
||||
Median E2E Latency (ms): 71902.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 160.21
|
||||
Median TTFT (ms): 140.32
|
||||
P99 TTFT (ms): 295.56
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.92
|
||||
Median TPOT (ms): 14.94
|
||||
P99 TPOT (ms): 15.02
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.96
|
||||
Median ITL (ms): 14.96
|
||||
P95 ITL (ms): 15.36
|
||||
P99 ITL (ms): 15.57
|
||||
Max ITL (ms): 19.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 503.30
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318226
|
||||
Total generated tokens (retokenized): 318025
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 78.82
|
||||
Output token throughput (tok/s): 632.28
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 711.09
|
||||
Concurrency: 13.88
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 87349.22
|
||||
Median E2E Latency (ms): 88248.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 228.54
|
||||
Median TTFT (ms): 142.78
|
||||
P99 TTFT (ms): 569.84
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 21.97
|
||||
Median TPOT (ms): 22.14
|
||||
P99 TPOT (ms): 22.47
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 21.91
|
||||
Median ITL (ms): 21.80
|
||||
P95 ITL (ms): 22.30
|
||||
P99 ITL (ms): 22.78
|
||||
Max ITL (ms): 137.19
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 772.28
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1300705
|
||||
Total generated tokens (retokenized): 1299924
|
||||
Request throughput (req/s): 0.41
|
||||
Input token throughput (tok/s): 205.80
|
||||
Output token throughput (tok/s): 1684.24
|
||||
Peak output token throughput (tok/s): 2112.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 1890.05
|
||||
Concurrency: 56.17
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 135563.36
|
||||
Median E2E Latency (ms): 140888.88
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 232.45
|
||||
Median TTFT (ms): 145.59
|
||||
P99 TTFT (ms): 576.49
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.47
|
||||
Median TPOT (ms): 34.02
|
||||
P99 TPOT (ms): 35.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.30
|
||||
Median ITL (ms): 32.63
|
||||
P95 ITL (ms): 34.27
|
||||
P99 ITL (ms): 104.39
|
||||
Max ITL (ms): 155.65
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 65.11
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4210
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 644.17
|
||||
Output token throughput (tok/s): 64.66
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 708.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6508.31
|
||||
Median E2E Latency (ms): 5263.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 189.48
|
||||
Median TTFT (ms): 159.23
|
||||
P99 TTFT (ms): 304.09
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 15.02
|
||||
Median TPOT (ms): 15.03
|
||||
P99 TPOT (ms): 15.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.04
|
||||
Median ITL (ms): 15.03
|
||||
P95 ITL (ms): 15.46
|
||||
P99 ITL (ms): 15.65
|
||||
Max ITL (ms): 24.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 76.43
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41589
|
||||
Total generated tokens (retokenized): 41577
|
||||
Request throughput (req/s): 1.05
|
||||
Input token throughput (tok/s): 3925.47
|
||||
Output token throughput (tok/s): 544.15
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 4469.62
|
||||
Concurrency: 13.95
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13329.63
|
||||
Median E2E Latency (ms): 14141.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 339.88
|
||||
Median TTFT (ms): 252.75
|
||||
P99 TTFT (ms): 906.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.37
|
||||
Median TPOT (ms): 25.73
|
||||
P99 TPOT (ms): 30.94
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.04
|
||||
Median ITL (ms): 21.68
|
||||
P95 ITL (ms): 22.69
|
||||
P99 ITL (ms): 146.98
|
||||
Max ITL (ms): 483.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 136.24
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 169680
|
||||
Total generated tokens (retokenized): 169452
|
||||
Request throughput (req/s): 2.35
|
||||
Input token throughput (tok/s): 9350.32
|
||||
Output token throughput (tok/s): 1245.44
|
||||
Peak output token throughput (tok/s): 1984.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 10595.77
|
||||
Concurrency: 58.46
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 24889.40
|
||||
Median E2E Latency (ms): 25123.37
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 355.82
|
||||
Median TTFT (ms): 268.84
|
||||
P99 TTFT (ms): 858.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 46.62
|
||||
Median TPOT (ms): 49.04
|
||||
P99 TPOT (ms): 58.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 46.36
|
||||
Median ITL (ms): 32.46
|
||||
P95 ITL (ms): 135.23
|
||||
P99 ITL (ms): 204.27
|
||||
Max ITL (ms): 508.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.975
|
||||
Invalid: 0.000
|
||||
Latency: 16.574 s
|
||||
Output throughput: 1194.637 token/s
|
||||
```
|
||||
@@ -0,0 +1,512 @@
|
||||
---
|
||||
title: GLM-4.6V
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6V vision-language model with SGLang - native function calling, 128K context, multimodal document understanding and frontend replication."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios.
|
||||
|
||||
Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features:
|
||||
|
||||
- **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#4-2-3-tool-calling).
|
||||
- **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content.
|
||||
- **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text.
|
||||
- **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
### 2.1 Docker Installation (Recommended)
|
||||
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Ready to use out of the box, no manual environment configuration needed
|
||||
- Avoids dependency conflict issues
|
||||
- Easy to migrate between different environments
|
||||
|
||||
### 2.2 Build from Source
|
||||
|
||||
If you need to use the latest development version or require custom modifications, you can build from source:
|
||||
|
||||
```bash Command
|
||||
# Install SGLang using UV (recommended)
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -e "python[all]" --index-url=https://pypi.org/simple
|
||||
pip install nvidia-cudnn-cu12==9.16.0.29
|
||||
# Install ffmpeg to support video input
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
**Use Cases:**
|
||||
|
||||
- Need to customize and modify SGLang source code
|
||||
- Want to use the latest development features
|
||||
- Participate in SGLang project development
|
||||
|
||||
For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
|
||||
|
||||
import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deployment.jsx";
|
||||
|
||||
<GLM46VDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically).
|
||||
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
|
||||
- **Hardware Notes:**
|
||||
- **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
- **Additional Multimodal Parameters:**
|
||||
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
GLM-4.6V supports image and video inputs via the OpenAI-compatible API.
|
||||
|
||||
**Image Input:**
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{30000}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "default",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "image_url",
|
||||
"image_url": {{
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}}
|
||||
}},
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What is the image"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"temperature": "0",
|
||||
"max_completion_tokens": "1000",
|
||||
"max_tokens": "1000"
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print(response)
|
||||
```
|
||||
|
||||
```text Output
|
||||
{"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"</>\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"</>\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
|
||||
```
|
||||
|
||||
**Video Input:**
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{30000}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "default",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "video_url",
|
||||
"video_url": {{
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
}}
|
||||
}},
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What is in the video"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"temperature": "0",
|
||||
"max_completion_tokens": "1000",
|
||||
"max_tokens": "1000"
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print(response)
|
||||
```
|
||||
|
||||
```text Output
|
||||
{"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n* **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n* **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n* **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n* **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
|
||||
```
|
||||
|
||||
#### 4.2.2 Thinking Mode
|
||||
|
||||
GLM-4.6V supports Thinking mode. Enable the reasoning parser during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6V \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
GLM-4.6V supports tool calling with vision capabilities. Pass tools in your API request:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
openai_api_key = "EMPTY"
|
||||
openai_api_base = "http://127.0.0.1:30000/v1"
|
||||
client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)
|
||||
|
||||
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current temperature for a given location.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and country e.g. Beijing, China",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Please help me check today's weather in Beijing, and tell me whether the tool returned an image."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_bk32t88BGpSdbtDgzT044Rh4",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": 'get_weather',
|
||||
"arguments": '{"location":"Beijing, China"}'
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C."
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=messages,
|
||||
timeout=900,
|
||||
tools=tools
|
||||
)
|
||||
print(response.choices[0].message.content.strip())
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C.
|
||||
|
||||
Yes, the tool returned an image (the SGL logo).
|
||||
```
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Beyond the reasoning parser, you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request — same as the [GLM-4.6 text model approach](./GLM-4.6#4-2-3-thinking-budget):
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=[{"role": "user", "content": "Describe this image briefly."}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1. Text Benchmark: Latency, Throughput and Accuracy
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 ./benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
#### Result Output
|
||||
```text Output
|
||||
Accuracy: 0.925
|
||||
Invalid: 0.000
|
||||
Latency: 15.327 s
|
||||
Output throughput: 1788.375 token/s
|
||||
```
|
||||
|
||||
### 5.2. Multimodal Benchmark - Latency and Throughput
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--port 30000 \
|
||||
--model zai-org/GLM-4.6V \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 128 \
|
||||
--max-concurrency 8
|
||||
```
|
||||
|
||||
#### Result Output
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 8
|
||||
Successful requests: 128
|
||||
Benchmark duration (s): 89.27
|
||||
Total input tokens: 315390
|
||||
Total input text tokens: 8702
|
||||
Total input vision tokens: 306688
|
||||
Total generated tokens: 66020
|
||||
Total generated tokens (retokenized): 31037
|
||||
Request throughput (req/s): 1.43
|
||||
Input token throughput (tok/s): 3533.17
|
||||
Output token throughput (tok/s): 739.59
|
||||
Peak output token throughput (tok/s): 823.00
|
||||
Peak concurrent requests: 12
|
||||
Total token throughput (tok/s): 4272.76
|
||||
Concurrency: 7.67
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5349.20
|
||||
Median E2E Latency (ms): 5380.98
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1724.04
|
||||
Median TTFT (ms): 1688.16
|
||||
P99 TTFT (ms): 6152.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.15
|
||||
Median TPOT (ms): 7.77
|
||||
P99 TPOT (ms): 23.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.00
|
||||
Median ITL (ms): 8.44
|
||||
P95 ITL (ms): 9.23
|
||||
P99 ITL (ms): 116.02
|
||||
Max ITL (ms): 173.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
### 5.3. Multimodal Accuracy Benchmark - MMMU
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}'
|
||||
```
|
||||
|
||||
#### Result Output
|
||||
```text Output
|
||||
Benchmark time: 487.2229107860476
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.962, 'num': 26},
|
||||
'Agriculture': {'acc': 0.5, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.733, 'num': 15},
|
||||
'Art': {'acc': 0.833, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.733, 'num': 30},
|
||||
'Biology': {'acc': 0.586, 'num': 29},
|
||||
'Chemistry': {'acc': 0.654, 'num': 26},
|
||||
'Clinical_Medicine': {'acc': 0.633, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.76, 'num': 25},
|
||||
'Design': {'acc': 0.867, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30},
|
||||
'Economics': {'acc': 0.862, 'num': 29},
|
||||
'Electronics': {'acc': 0.5, 'num': 18},
|
||||
'Energy_and_Power': {'acc': 0.875, 'num': 16},
|
||||
'Finance': {'acc': 0.857, 'num': 28},
|
||||
'Geography': {'acc': 0.714, 'num': 28},
|
||||
'History': {'acc': 0.767, 'num': 30},
|
||||
'Literature': {'acc': 0.897, 'num': 29},
|
||||
'Manage': {'acc': 0.759, 'num': 29},
|
||||
'Marketing': {'acc': 1.0, 'num': 26},
|
||||
'Materials': {'acc': 0.833, 'num': 18},
|
||||
'Math': {'acc': 0.76, 'num': 25},
|
||||
'Mechanical_Engineering': {'acc': 0.619, 'num': 21},
|
||||
'Music': {'acc': 0.286, 'num': 28},
|
||||
'Overall': {'acc': 0.761, 'num': 803},
|
||||
'Overall-Art and Design': {'acc': 0.729, 'num': 118},
|
||||
'Overall-Business': {'acc': 0.884, 'num': 138},
|
||||
'Overall-Health and Medicine': {'acc': 0.773, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118},
|
||||
'Overall-Science': {'acc': 0.728, 'num': 136},
|
||||
'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143},
|
||||
'Pharmacy': {'acc': 0.933, 'num': 30},
|
||||
'Physics': {'acc': 0.929, 'num': 28},
|
||||
'Psychology': {'acc': 0.733, 'num': 30},
|
||||
'Public_Health': {'acc': 0.933, 'num': 30},
|
||||
'Sociology': {'acc': 0.724, 'num': 29}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.761
|
||||
```
|
||||
@@ -0,0 +1,935 @@
|
||||
---
|
||||
title: GLM-4.7-Flash
|
||||
metatags:
|
||||
description: "Deploy GLM-4.7-Flash 30B-A3B MoE model with SGLang - lightweight, efficient inference optimized for single-GPU deployment."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment.
|
||||
|
||||
As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency:
|
||||
|
||||
- **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference
|
||||
- **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance
|
||||
- **Superior Coding**: Strong code generation and understanding capabilities
|
||||
- **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows
|
||||
- **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios
|
||||
|
||||
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM47FlashDeployment } from "/src/snippets/autoregressive/glm-47-flash-deployment.jsx";
|
||||
|
||||
<GLM47FlashDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.7-Flash. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--reasoning-parser glm45 \
|
||||
--attention-backend triton \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47 \
|
||||
--attention-backend triton \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls (tool call deltas may stream in multiple chunks)
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
if tool_calls_accumulator:
|
||||
print("\n=============== Tool Calls =================", flush=True)
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the
|
||||
user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you.
|
||||
=============== Tool Calls =================
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 (1x)
|
||||
- Model: GLM-4.7-Flash
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--attention-backend triton \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 38.94
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.26
|
||||
Input token throughput (tok/s): 156.67
|
||||
Output token throughput (tok/s): 108.37
|
||||
Peak output token throughput (tok/s): 125.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 265.03
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3891.12
|
||||
Median E2E Latency (ms): 3061.48
|
||||
P90 E2E Latency (ms): 7172.25
|
||||
P99 E2E Latency (ms): 9042.62
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 131.36
|
||||
Median TTFT (ms): 94.55
|
||||
P99 TTFT (ms): 435.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.75
|
||||
Median TPOT (ms): 8.82
|
||||
P99 TPOT (ms): 9.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.93
|
||||
Median ITL (ms): 8.98
|
||||
P95 ITL (ms): 9.83
|
||||
P99 ITL (ms): 10.20
|
||||
Max ITL (ms): 18.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 52.73
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40775
|
||||
Request throughput (req/s): 1.52
|
||||
Input token throughput (tok/s): 752.27
|
||||
Output token throughput (tok/s): 773.83
|
||||
Peak output token throughput (tok/s): 1040.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1526.10
|
||||
Concurrency: 13.98
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9217.90
|
||||
Median E2E Latency (ms): 9642.50
|
||||
P90 E2E Latency (ms): 15147.02
|
||||
P99 E2E Latency (ms): 18237.06
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 299.02
|
||||
Median TTFT (ms): 105.98
|
||||
P99 TTFT (ms): 1109.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.03
|
||||
Median TPOT (ms): 18.00
|
||||
P99 TPOT (ms): 26.51
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 17.52
|
||||
Median ITL (ms): 16.07
|
||||
P95 ITL (ms): 18.14
|
||||
P99 ITL (ms): 89.43
|
||||
Max ITL (ms): 763.13
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 91.48
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250941
|
||||
Request throughput (req/s): 5.47
|
||||
Input token throughput (tok/s): 2730.87
|
||||
Output token throughput (tok/s): 2761.82
|
||||
Peak output token throughput (tok/s): 4199.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 5492.69
|
||||
Concurrency: 90.54
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16566.04
|
||||
Median E2E Latency (ms): 16134.36
|
||||
P90 E2E Latency (ms): 30167.60
|
||||
P99 E2E Latency (ms): 34034.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 433.94
|
||||
Median TTFT (ms): 123.26
|
||||
P99 TTFT (ms): 1760.09
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.26
|
||||
Median TPOT (ms): 33.56
|
||||
P99 TPOT (ms): 38.78
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 31.99
|
||||
Median ITL (ms): 24.06
|
||||
P95 ITL (ms): 79.62
|
||||
P99 ITL (ms): 103.03
|
||||
Max ITL (ms): 1369.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 525.43
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44451
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 11.61
|
||||
Output token throughput (tok/s): 84.62
|
||||
Peak output token throughput (tok/s): 125.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 96.23
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 52540.19
|
||||
Median E2E Latency (ms): 53694.45
|
||||
P90 E2E Latency (ms): 94742.08
|
||||
P99 E2E Latency (ms): 101224.18
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 97.45
|
||||
Median TTFT (ms): 95.28
|
||||
P99 TTFT (ms): 105.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.94
|
||||
Median TPOT (ms): 11.25
|
||||
P99 TPOT (ms): 13.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.80
|
||||
Median ITL (ms): 11.51
|
||||
P95 ITL (ms): 15.83
|
||||
P99 ITL (ms): 16.86
|
||||
Max ITL (ms): 19.96
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 473.92
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317860
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 83.70
|
||||
Output token throughput (tok/s): 671.65
|
||||
Peak output token throughput (tok/s): 1040.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 755.35
|
||||
Concurrency: 13.80
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 81746.73
|
||||
Median E2E Latency (ms): 78508.54
|
||||
P90 E2E Latency (ms): 155292.49
|
||||
P99 E2E Latency (ms): 166769.99
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 117.50
|
||||
Median TTFT (ms): 101.97
|
||||
P99 TTFT (ms): 182.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.36
|
||||
Median TPOT (ms): 20.48
|
||||
P99 TPOT (ms): 22.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.52
|
||||
Median ITL (ms): 20.42
|
||||
P95 ITL (ms): 23.41
|
||||
P99 ITL (ms): 26.29
|
||||
Max ITL (ms): 90.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 714.72
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1289431
|
||||
Request throughput (req/s): 0.45
|
||||
Input token throughput (tok/s): 222.38
|
||||
Output token throughput (tok/s): 1820.33
|
||||
Peak output token throughput (tok/s): 3200.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 2042.71
|
||||
Concurrency: 55.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 124364.58
|
||||
Median E2E Latency (ms): 129250.98
|
||||
P90 E2E Latency (ms): 219175.80
|
||||
P99 E2E Latency (ms): 247741.77
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 149.40
|
||||
Median TTFT (ms): 114.78
|
||||
P99 TTFT (ms): 288.60
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 30.51
|
||||
Median TPOT (ms): 31.75
|
||||
P99 TPOT (ms): 33.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 30.56
|
||||
Median ITL (ms): 30.82
|
||||
P95 ITL (ms): 33.20
|
||||
P99 ITL (ms): 80.54
|
||||
Max ITL (ms): 117.72
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 58.27
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 719.73
|
||||
Output token throughput (tok/s): 72.42
|
||||
Peak output token throughput (tok/s): 112.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 792.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5825.08
|
||||
Median E2E Latency (ms): 4624.26
|
||||
P90 E2E Latency (ms): 12690.22
|
||||
P99 E2E Latency (ms): 13177.96
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 296.01
|
||||
Median TTFT (ms): 195.59
|
||||
P99 TTFT (ms): 717.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.63
|
||||
Median TPOT (ms): 13.07
|
||||
P99 TPOT (ms): 16.68
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.13
|
||||
Median ITL (ms): 13.17
|
||||
P95 ITL (ms): 17.02
|
||||
P99 ITL (ms): 17.47
|
||||
Max ITL (ms): 19.84
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 89.59
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41656
|
||||
Request throughput (req/s): 0.89
|
||||
Input token throughput (tok/s): 3348.77
|
||||
Output token throughput (tok/s): 465.10
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3813.87
|
||||
Concurrency: 14.39
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16120.74
|
||||
Median E2E Latency (ms): 16246.55
|
||||
P90 E2E Latency (ms): 27279.72
|
||||
P99 E2E Latency (ms): 34577.93
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1943.94
|
||||
Median TTFT (ms): 382.19
|
||||
P99 TTFT (ms): 8980.41
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 27.87
|
||||
Median TPOT (ms): 28.26
|
||||
P99 TPOT (ms): 40.55
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 27.27
|
||||
Median ITL (ms): 21.74
|
||||
P95 ITL (ms): 23.32
|
||||
P99 ITL (ms): 232.65
|
||||
Max ITL (ms): 4282.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 167.01
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169226
|
||||
Request throughput (req/s): 1.92
|
||||
Input token throughput (tok/s): 7627.82
|
||||
Output token throughput (tok/s): 1017.93
|
||||
Peak output token throughput (tok/s): 1984.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 8645.75
|
||||
Concurrency: 59.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 31147.52
|
||||
Median E2E Latency (ms): 30603.34
|
||||
P90 E2E Latency (ms): 54889.44
|
||||
P99 E2E Latency (ms): 67665.30
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 428.87
|
||||
Median TTFT (ms): 441.69
|
||||
P99 TTFT (ms): 1232.68
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 58.06
|
||||
Median TPOT (ms): 62.79
|
||||
P99 TPOT (ms): 82.23
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 57.93
|
||||
Median ITL (ms): 33.30
|
||||
P95 ITL (ms): 247.98
|
||||
P99 ITL (ms): 409.63
|
||||
Max ITL (ms): 1421.21
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Result
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.845
|
||||
Invalid: 0.000
|
||||
Latency: 8.431 s
|
||||
Output throughput: 2195.387 token/s
|
||||
```
|
||||
@@ -0,0 +1,962 @@
|
||||
---
|
||||
title: GLM-4.7
|
||||
metatags:
|
||||
description: "Deploy GLM-4.7 with SGLang on NVIDIA Blackwell (B200, GB200) and AMD GPUs - state-of-the-art reasoning, robust tool calling, and NVFP4 weights for Blackwell."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and agent workflows.
|
||||
|
||||
GLM-4.7 brings improvements across all major domains:
|
||||
|
||||
- **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations
|
||||
- **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing
|
||||
- **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance
|
||||
- **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows
|
||||
- **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms
|
||||
|
||||
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks
|
||||
- **Multiple Quantizations**: BF16, FP8, and NVFP4 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Tuned for NVIDIA Blackwell (B200, GB200) and AMD MI300X/MI325X/MI355X GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8)
|
||||
- **NVFP4 (4-bit, NVIDIA Blackwell)**: [nvidia/GLM-4.7-NVFP4](https://huggingface.co/nvidia/GLM-4.7-NVFP4)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA H100 / H200 / B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA GB200 / B300 / GB300 (aarch64)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-cu130`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-rocm720-mi30x`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-rocm720-mi35x`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM47Deployment } from "/src/snippets/autoregressive/glm-47-deployment.jsx";
|
||||
|
||||
<GLM47Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Pick a weight format by hardware: **NVFP4** on NVIDIA Blackwell (B200, GB200), **FP8** on H100/H200/AMD, **BF16** as the full-precision fallback. The recommended tensor-parallel size per platform:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>NVFP4</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200 (8×, single node)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=2 / 4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB200 (NVL72, 4× per tray)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=2 / 4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200 (8×)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X / MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=2 / 4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4 / 8</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.7. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
For general GLM-4.x family launch guidance (AMD ROCm notes and more), see [Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang](/cookbook/autoregressive/GLM/GLM-4.5). Per-hardware bench commands and flags are inline in §5.1 below.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.7 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200, NVIDIA GB200, AMD MI300X/MI325X/MI355X (8x)
|
||||
- Model: GLM-4.7-NVFP4 on NVIDIA Blackwell; GLM-4.7-FP8 or GLM-4.7 (BF16) on AMD
|
||||
- SGLang Version: 0.5.12 (NVIDIA Blackwell), 0.5.6.post1 (AMD)
|
||||
- Best per-GPU throughput config on B200: **TP=2 NVFP4 bf16-KV** (NVFP4 weights, no EP). Numbers below come from this config.
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Four core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Throughput**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mixed RAG / agent / multi-turn conversation (used for the inline B200 / GB200 results below)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) — the Throughput (4K/1K) scenario uses `--max-concurrency 128` to match the inline B200/GB200 results below.
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 4: Throughput (4K/1K) — NVIDIA Blackwell with NVFP4**
|
||||
|
||||
The remaining sub-sections (§5.1.4.1 NVIDIA B200, §5.1.4.2 NVIDIA GB200) measure this scenario with `nvidia/GLM-4.7-NVFP4` weights and report the full `bench_serving` output verbatim. The same commands apply to other NVIDIA hardware after substituting the deployment line from §3.1.
|
||||
|
||||
> **Note**: These runs use EOS-enabled generation (no `--disable-ignore-eos`), so generated-token counts reflect natural model behavior rather than a strict fixed-OSL pin. Compare against other EOS-enabled runs at the same workload, not against fixed-output-length benchmarks.
|
||||
|
||||
#### 5.1.4.1 NVIDIA B200
|
||||
|
||||
**Model Deployment (NVIDIA B200, TP=2 NVFP4 — max tok/s/gpu config):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 5 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 1
|
||||
Successful requests: 5
|
||||
Benchmark duration (s): 25.07
|
||||
Total input tokens: 8105
|
||||
Total generated tokens: 2674
|
||||
Request throughput (req/s): 0.20
|
||||
Input token throughput (tok/s): 323.25
|
||||
Output token throughput (tok/s): 106.65
|
||||
Total token throughput (tok/s): 429.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5011.93
|
||||
Median E2E Latency (ms): 6441.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 179.61
|
||||
Median TTFT (ms): 169.05
|
||||
P99 TTFT (ms): 238.01
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.05
|
||||
Median TPOT (ms): 9.03
|
||||
P99 TPOT (ms): 9.16
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.05
|
||||
Median ITL (ms): 9.05
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.60
|
||||
Total input tokens: 179772
|
||||
Total generated tokens: 39657
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 2966.39
|
||||
Output token throughput (tok/s): 654.37
|
||||
Total token throughput (tok/s): 3620.76
|
||||
Concurrency: 14.01
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10615.87
|
||||
Median E2E Latency (ms): 9985.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 267.39
|
||||
Median TTFT (ms): 177.26
|
||||
P99 TTFT (ms): 584.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.98
|
||||
Median TPOT (ms): 21.06
|
||||
P99 TPOT (ms): 24.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.92
|
||||
Median ITL (ms): 17.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 640 \
|
||||
--max-concurrency 128 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 128
|
||||
Successful requests: 640
|
||||
Benchmark duration (s): 172.95
|
||||
Total input tokens: 1453591
|
||||
Total generated tokens: 308740
|
||||
Request throughput (req/s): 3.70
|
||||
Input token throughput (tok/s): 8404.67
|
||||
Output token throughput (tok/s): 1785.14
|
||||
Total token throughput (tok/s): 10189.80
|
||||
Concurrency: 117.85
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 31848.20
|
||||
Median E2E Latency (ms): 28554.42
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1598.40
|
||||
Median TTFT (ms): 298.88
|
||||
P99 TTFT (ms): 11015.96
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 65.94
|
||||
Median TPOT (ms): 65.81
|
||||
P99 TPOT (ms): 137.73
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 62.99
|
||||
Median ITL (ms): 35.44
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.4.2 NVIDIA GB200
|
||||
|
||||
**Model Deployment (NVIDIA GB200, TP=2 NVFP4 — max tok/s/gpu config):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 5 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 1
|
||||
Successful requests: 5
|
||||
Benchmark duration (s): 24.74
|
||||
Total input tokens: 8105
|
||||
Total generated tokens: 2674
|
||||
Request throughput (req/s): 0.20
|
||||
Input token throughput (tok/s): 327.65
|
||||
Output token throughput (tok/s): 108.10
|
||||
Total token throughput (tok/s): 435.75
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4944.47
|
||||
Median E2E Latency (ms): 6347.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 211.41
|
||||
Median TTFT (ms): 207.25
|
||||
P99 TTFT (ms): 226.46
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.86
|
||||
Median TPOT (ms): 8.84
|
||||
P99 TPOT (ms): 8.96
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.87
|
||||
Median ITL (ms): 8.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.40
|
||||
Total input tokens: 179772
|
||||
Total generated tokens: 39657
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 2976.52
|
||||
Output token throughput (tok/s): 656.61
|
||||
Total token throughput (tok/s): 3633.13
|
||||
Concurrency: 13.97
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10611.51
|
||||
Median E2E Latency (ms): 9956.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 338.14
|
||||
Median TTFT (ms): 215.25
|
||||
P99 TTFT (ms): 915.40
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.87
|
||||
Median TPOT (ms): 21.36
|
||||
P99 TPOT (ms): 27.05
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.77
|
||||
Median ITL (ms): 16.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 640 \
|
||||
--max-concurrency 128 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 128
|
||||
Successful requests: 640
|
||||
Benchmark duration (s): 181.89
|
||||
Total input tokens: 1453591
|
||||
Total generated tokens: 309221
|
||||
Request throughput (req/s): 3.52
|
||||
Input token throughput (tok/s): 7991.59
|
||||
Output token throughput (tok/s): 1700.04
|
||||
Total token throughput (tok/s): 9691.63
|
||||
Concurrency: 118.86
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 33690.47
|
||||
Median E2E Latency (ms): 30421.55
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1353.16
|
||||
Median TTFT (ms): 383.52
|
||||
P99 TTFT (ms): 8940.53
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 69.88
|
||||
Median TPOT (ms): 71.77
|
||||
P99 TPOT (ms): 131.75
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 67.23
|
||||
Median ITL (ms): 33.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **4K/1K (Throughput)**: Realistic mixed workload typical of production deployments (RAG context + medium response). Long enough input that prefill matters, long enough output that decode steady-state dominates. Used for the inline B200 / GB200 results above.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-shots 5 \
|
||||
--num-questions 1319 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Test Result (NVIDIA B200, TP=2 NVFP4)
|
||||
```text Output
|
||||
Accuracy: 0.946
|
||||
Latency: 178.284 s
|
||||
Output throughput: 769.204 token/s
|
||||
```
|
||||
|
||||
- Test Result (NVIDIA GB200, TP=2 NVFP4)
|
||||
```text Output
|
||||
Accuracy: 0.951
|
||||
Latency: 175.190 s
|
||||
Invalid: 0.000
|
||||
```
|
||||
@@ -0,0 +1,740 @@
|
||||
---
|
||||
title: GLM-5.1
|
||||
metatags:
|
||||
description: "Deploy GLM-5.1 with SGLang on NVIDIA H100/H200/B300/GB300 and AMD MI300X/MI325X/MI355X."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8)
|
||||
- **NVFP4 (4-bit quantized)**: [nvidia/GLM-5.1-NVFP4](https://huggingface.co/nvidia/GLM-5.1-NVFP4)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B300, GB300, and AMD MI300X/MI325X/MI355X GPUs.
|
||||
|
||||
import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.jsx'
|
||||
|
||||
<GLM51Deployment />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.1.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>NVFP4</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>MXFP4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **H100 and H200**: FP8 is the recommended deployment path.
|
||||
- **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300.
|
||||
- **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang.
|
||||
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
|
||||
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1-FP8 \
|
||||
--tp 8 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 B300/GB300 (NVFP4) Server Command
|
||||
|
||||
#### B300
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/GLM-5.1-NVFP4 \
|
||||
--tp 8 \
|
||||
--quantization modelopt_fp4 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.80 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### GB300
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/GLM-5.1-NVFP4 \
|
||||
--tp 4 \
|
||||
--quantization modelopt_fp4 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.80 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 MI300X/MI325X/MI355X (ROCm) Server Command
|
||||
|
||||
The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above.
|
||||
|
||||
#### MXFP4 (MI355X / gfx950)
|
||||
|
||||
On MI355X (gfx950), set `SGLANG_DSA_TRITON_PREFILL=1` to enable a faster Triton attention kernel for the prefill phase (opt-in, off by default). Keep `--dsa-prefill-backend tilelang` as shown. The EAGLE speculative-decoding flags below are optional but recommended on gfx950.
|
||||
|
||||
```shell Command
|
||||
# SGLANG_DSA_TRITON_PREFILL=1 is optional; it enables a faster Triton prefill kernel on gfx950
|
||||
SGLANG_DSA_TRITON_PREFILL=1 sglang serve \
|
||||
--model-path amd/GLM-5.1-MXFP4 \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### FP8 (Recommended)
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1-FP8 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### BF16
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.3 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.4 Advanced Usage
|
||||
|
||||
#### 4.4.1 Reasoning Parser
|
||||
|
||||
GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Thinking mode is enabled by default, no extra parameters needed
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
1. **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step.
|
||||
|
||||
2. **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$.
|
||||
|
||||
3. **Step-by-Step Breakdown:**
|
||||
* *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$.
|
||||
* *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$.
|
||||
* *Step 3: Perform the calculation.*
|
||||
* $0.15 \times 240$
|
||||
* I can break this down further to make it easy to follow:
|
||||
* $0.10 \times 240 = 24$ (which is 10%)
|
||||
* $0.05 \times 240 = 12$ (which is 5%, half of 10%)
|
||||
* $24 + 12 = 36$
|
||||
* Alternatively, standard multiplication:
|
||||
* $240 \times 15 = 3600$
|
||||
* Move decimal two places left -> $36$
|
||||
* *Step 4: State the final answer clearly.*
|
||||
|
||||
4. **Draft the Response (incorporating the steps clearly):**
|
||||
* *Introduction:* State the problem clearly.
|
||||
* *Step 1:* Explain how to convert 15% to a decimal.
|
||||
* *Step 2:* Explain the multiplication step.
|
||||
* *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value).
|
||||
* *Conclusion:* Give the final answer.
|
||||
|
||||
5. **Refine the Output (Self-Correction/Polishing during drafting):**
|
||||
* *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15.
|
||||
* *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240.
|
||||
* *Drafting Step 3 (Standard way):* 0.15 × 240 = 36.
|
||||
* *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%.
|
||||
* 10% of 240 = 24 (move the decimal point one place to the left)
|
||||
* 5% is half of 10%, so half of 24 = 12
|
||||
* Add them together: 24 + 12 = 36.
|
||||
* *Final Answer:* 15% of 240 is 36.
|
||||
|
||||
6. **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output).
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution to find 15% of 240:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal.**
|
||||
To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left).
|
||||
* 15% = 15 ÷ 100 = **0.15**
|
||||
|
||||
**Step 2: Multiply the decimal by the number.**
|
||||
Now, multiply the decimal (0.15) by the number you are finding the percentage of (240).
|
||||
* 0.15 × 240 = **36**
|
||||
|
||||
*(Alternative mental math method for Step 2)*:
|
||||
If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%:
|
||||
* **10% of 240** = 24 (just move the decimal point one place to the left)
|
||||
* **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2)
|
||||
* **Add them together**: 24 + 12 = **36**
|
||||
|
||||
**Answer:**
|
||||
15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Disable thinking mode via chat_template_kwargs
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# In Instruct mode, the model responds directly without reasoning_content
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
15% of 240 is 36.
|
||||
|
||||
Here is how to calculate it:
|
||||
1. Convert the percentage to a decimal: 15% = 0.15
|
||||
2. Multiply the decimal by the number: 0.15 × 240 = 36
|
||||
```
|
||||
|
||||
#### 4.4.2 Tool Calling
|
||||
|
||||
GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location.
|
||||
=============== Content =================
|
||||
Tool Call: get_weather
|
||||
Arguments:
|
||||
Tool Call: None
|
||||
Arguments: {
|
||||
Tool Call: None
|
||||
Arguments: "location": "Be
|
||||
Tool Call: None
|
||||
Arguments: ijing"
|
||||
Tool Call: None
|
||||
Arguments: }
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: H200 (8x)
|
||||
- Model: GLM-5.1-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: commit 947927bdb
|
||||
|
||||
#### 5.1.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5.1-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.78
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4213
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 170.54
|
||||
Output token throughput (tok/s): 117.96
|
||||
Peak output token throughput (tok/s): 148.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 288.50
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.48
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3576.31
|
||||
Median E2E Latency (ms): 2935.97
|
||||
P90 E2E Latency (ms): 5908.97
|
||||
P99 E2E Latency (ms): 8588.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 290.88
|
||||
Median TTFT (ms): 282.34
|
||||
P99 TTFT (ms): 332.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.54
|
||||
Median TPOT (ms): 6.97
|
||||
P99 TPOT (ms): 9.04
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.80
|
||||
Median ITL (ms): 6.81
|
||||
P95 ITL (ms): 13.51
|
||||
P99 ITL (ms): 26.99
|
||||
Max ITL (ms): 29.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5.1-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 411.74
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 499614
|
||||
Request throughput (req/s): 2.43
|
||||
Input token throughput (tok/s): 1220.41
|
||||
Output token throughput (tok/s): 1214.97
|
||||
Peak output token throughput (tok/s): 2648.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 2435.38
|
||||
Concurrency: 96.30
|
||||
Accept length: 3.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 39648.76
|
||||
Median E2E Latency (ms): 39058.12
|
||||
P90 E2E Latency (ms): 57009.82
|
||||
P99 E2E Latency (ms): 68880.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 20613.80
|
||||
Median TTFT (ms): 21429.21
|
||||
P99 TTFT (ms): 29543.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.73
|
||||
Median TPOT (ms): 36.52
|
||||
P99 TPOT (ms): 67.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.13
|
||||
Median ITL (ms): 16.57
|
||||
P95 ITL (ms): 86.01
|
||||
P99 ITL (ms): 164.88
|
||||
Max ITL (ms): 1307.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
<Note>
|
||||
The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate benchmark run is planned.
|
||||
</Note>
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.000
|
||||
Latency: 32.470 s
|
||||
Output throughput: 642.044 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.860
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.932
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.640
|
||||
subject: college_computer_science, #q:100, acc: 0.900
|
||||
subject: college_mathematics, #q:100, acc: 0.810
|
||||
subject: college_medicine, #q:173, acc: 0.873
|
||||
subject: college_physics, #q:102, acc: 0.912
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.928
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.897
|
||||
subject: elementary_mathematics, #q:378, acc: 0.937
|
||||
subject: formal_logic, #q:126, acc: 0.778
|
||||
subject: global_facts, #q:100, acc: 0.710
|
||||
subject: high_school_biology, #q:310, acc: 0.961
|
||||
subject: high_school_chemistry, #q:203, acc: 0.847
|
||||
subject: high_school_computer_science, #q:100, acc: 0.960
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.960
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.984
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.923
|
||||
subject: high_school_mathematics, #q:270, acc: 0.696
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.821
|
||||
subject: high_school_psychology, #q:545, acc: 0.956
|
||||
subject: high_school_statistics, #q:216, acc: 0.889
|
||||
subject: high_school_us_history, #q:204, acc: 0.941
|
||||
subject: high_school_world_history, #q:237, acc: 0.945
|
||||
subject: human_aging, #q:223, acc: 0.857
|
||||
subject: human_sexuality, #q:131, acc: 0.908
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.907
|
||||
subject: logical_fallacies, #q:163, acc: 0.933
|
||||
subject: machine_learning, #q:112, acc: 0.830
|
||||
subject: management, #q:103, acc: 0.942
|
||||
subject: marketing, #q:234, acc: 0.940
|
||||
subject: medical_genetics, #q:100, acc: 0.990
|
||||
subject: miscellaneous, #q:783, acc: 0.959
|
||||
subject: moral_disputes, #q:346, acc: 0.873
|
||||
subject: moral_scenarios, #q:895, acc: 0.837
|
||||
subject: nutrition, #q:306, acc: 0.922
|
||||
subject: philosophy, #q:311, acc: 0.897
|
||||
subject: prehistory, #q:324, acc: 0.929
|
||||
subject: professional_accounting, #q:282, acc: 0.844
|
||||
subject: professional_law, #q:1534, acc: 0.714
|
||||
subject: professional_medicine, #q:272, acc: 0.941
|
||||
subject: professional_psychology, #q:612, acc: 0.913
|
||||
subject: public_relations, #q:110, acc: 0.791
|
||||
subject: security_studies, #q:245, acc: 0.878
|
||||
subject: sociology, #q:201, acc: 0.940
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.596
|
||||
subject: world_religions, #q:171, acc: 0.936
|
||||
Total latency: 165.275
|
||||
Average accuracy: 0.877
|
||||
```
|
||||
|
||||
### 5.3 AMD GPU Benchmarks
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
|
||||
|
||||
- MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends)
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
title: GLM-5.2
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/zai-org/glm-5.2.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.2-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc`. Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5, balanced 2-1-3 = 2); high-throughput has no spec.
|
||||
</Note>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Context</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.2-FP8">GLM-5.2-FP8</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · FP8</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.2">GLM-5.2</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · BF16</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/nvidia/GLM-5.2-NVFP4">GLM-5.2-NVFP4</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).
|
||||
|
||||
**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`.
|
||||
- **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute.
|
||||
- **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).
|
||||
- **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.
|
||||
- **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points.
|
||||
- **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP.
|
||||
- **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.
|
||||
|
||||
- **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).
|
||||
|
||||
<Note>
|
||||
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
|
||||
</Note>
|
||||
|
||||
- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`).
|
||||
|
||||
**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: <level>` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`:
|
||||
|
||||
| `reasoning_effort` | Injected system line | Effect |
|
||||
|---|---|---|
|
||||
| *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** |
|
||||
| `"high"` | `Reasoning Effort: High` | dials reasoning **down** |
|
||||
| `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) |
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.2-FP8",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240.
|
||||
2. **Convert the percentage to a decimal:** 15% = 0.15
|
||||
3. **Multiply by the total:** 0.15 * 240 = 36
|
||||
(Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)
|
||||
|
||||
Answer: 15% of 240 is **36**.
|
||||
|
||||
Here is how you can calculate it:
|
||||
0.15 × 240 = 36
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `<tool_call>…<arg_key>…<arg_value>…` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
resp = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.2-FP8",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.
|
||||
|
||||
Tool calls: [
|
||||
{
|
||||
"id": "call_13fcd52146934b7781d06d4a",
|
||||
"type": "function",
|
||||
"function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern.
|
||||
|
||||
### 3.4 Claude Code Integration
|
||||
|
||||
GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then:
|
||||
|
||||
```bash Command
|
||||
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
|
||||
export ANTHROPIC_AUTH_TOKEN="dummy"
|
||||
export API_TIMEOUT_MS="3000000"
|
||||
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
|
||||
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
|
||||
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
|
||||
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"
|
||||
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"
|
||||
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"
|
||||
claude
|
||||
```
|
||||
|
||||
Two of these matter specifically for GLM-5.2:
|
||||
|
||||
- **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
|
||||
- **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side.
|
||||
|
||||
For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api).
|
||||
|
||||
### 3.5 Context Parallelism
|
||||
|
||||
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:
|
||||
```bash
|
||||
--attn-cp-size 8 \
|
||||
--enable-prefill-cp \
|
||||
--cp-strategy interleave \
|
||||
```
|
||||
which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.
|
||||
|
||||
When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with
|
||||
```bash
|
||||
--enable-dsa-cache-layer-split \
|
||||
--attn-cp-size 8 \
|
||||
--cp-strategy interleave \
|
||||
```
|
||||
With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.
|
||||
@@ -0,0 +1,675 @@
|
||||
---
|
||||
title: GLM-5
|
||||
metatags:
|
||||
description: "Deploy GLM-5 with SGLang on NVIDIA H100/H200/B200 and AMD MI300X/MI325X/MI355X — state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity.
|
||||
|
||||
With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks
|
||||
- **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2)
|
||||
- **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs.
|
||||
|
||||
import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.jsx'
|
||||
|
||||
<GLM5Deployment />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **B200 (FP8)**: Use `--ep 1 --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized DSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization.
|
||||
|
||||
- **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
|
||||
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
|
||||
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
|
||||
|
||||
<Warning>
|
||||
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
|
||||
</Warning>
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy GLM-5 with the following command (FP8 on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5-FP8 \
|
||||
--tp 8 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--enable-flashinfer-allreduce-fusion \
|
||||
--mem-fraction-static 0.85 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
|
||||
|
||||
The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Reasoning Parser
|
||||
|
||||
GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Thinking mode is enabled by default, no extra parameters needed
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user wants me to solve a math problem: "What is 15% of 240?".
|
||||
|
||||
Step 1: Understand the problem. I need to calculate a percentage of a number.
|
||||
Formula: Percentage × Number = Result.
|
||||
|
||||
Step 2: Convert the percentage to a decimal or fraction.
|
||||
15% = 15/100 or 0.15.
|
||||
|
||||
Step 3: Perform the multiplication.
|
||||
Method A: Decimal multiplication.
|
||||
0.15 × 240.
|
||||
Break it down:
|
||||
10% of 240 = 24.
|
||||
5% is half of 10%, so 12.
|
||||
15% = 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Method B: Fraction multiplication.
|
||||
15/100 × 240.
|
||||
Simplify 240/100 = 2.4.
|
||||
15 × 2.4.
|
||||
10 × 2.4 = 24.
|
||||
5 × 2.4 = 12.
|
||||
24 + 12 = 36.
|
||||
|
||||
Method C: Direct multiplication.
|
||||
240 × 0.15.
|
||||
240 × 0.10 = 24.
|
||||
240 × 0.05 = 12.
|
||||
24 + 12 = 36.
|
||||
|
||||
Step 4: Final Verification.
|
||||
Is 36 reasonable?
|
||||
10% is 24. 20% is 48.
|
||||
15% is halfway between 10% and 20%.
|
||||
Halfway between 24 and 48 is 36.
|
||||
The result is correct.
|
||||
|
||||
Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%).
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal.**
|
||||
To convert 15% to a decimal, divide by 100.
|
||||
$$15\% = \frac{15}{100} = 0.15$$
|
||||
|
||||
**Step 2: Multiply the decimal by the number.**
|
||||
Now, multiply 0.15 by 240.
|
||||
$$0.15 \times 240$$
|
||||
|
||||
**Step 3: Perform the calculation.**
|
||||
You can break this down to make it easier:
|
||||
$$0.15 = 0.10 + 0.05$$
|
||||
|
||||
* First, find 10% of 240:
|
||||
$$0.10 \times 240 = 24$$
|
||||
* Next, find 5% (which is half of 10%):
|
||||
$$\frac{24}{2} = 12$$
|
||||
* Add the two results together:
|
||||
$$24 + 12 = 36$$
|
||||
|
||||
**Answer:**
|
||||
15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Disable thinking mode via chat_template_kwargs
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# In Instruct mode, the model responds directly without reasoning_content
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
To find **15% of 240**, follow these steps:
|
||||
|
||||
### Step 1: Convert the Percentage to a Decimal
|
||||
First, convert the percentage to a decimal by dividing by 100.
|
||||
|
||||
\[
|
||||
15\% = \frac{15}{100} = 0.15
|
||||
\]
|
||||
|
||||
### Step 2: Multiply by the Number
|
||||
Next, multiply the decimal by the number you want to find the percentage of.
|
||||
|
||||
\[
|
||||
0.15 \times 240
|
||||
\]
|
||||
|
||||
### Step 3: Perform the Multiplication
|
||||
Calculate the multiplication:
|
||||
|
||||
\[
|
||||
0.15 \times 240 = 36
|
||||
\]
|
||||
|
||||
### Final Answer
|
||||
\[
|
||||
\boxed{36}
|
||||
\]
|
||||
```
|
||||
|
||||
#### 4.3.2 Tool Calling
|
||||
|
||||
GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required:
|
||||
|
||||
- location: required, should be "Beijing"
|
||||
- unit: optional (not in required array), can be "celsius" or "fahrenheit"
|
||||
|
||||
Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you.
|
||||
=============== Content =================
|
||||
Tool Call: get_weather
|
||||
Arguments:
|
||||
Tool Call: None
|
||||
Arguments: {
|
||||
Tool Call: None
|
||||
Arguments: "location": "Be
|
||||
Tool Call: None
|
||||
Arguments: ijing"
|
||||
Tool Call: None
|
||||
Arguments: }
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: H200 (8x)
|
||||
- Model: GLM-5-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: commit 947927bdb
|
||||
|
||||
#### 5.1.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.78
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4213
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 170.54
|
||||
Output token throughput (tok/s): 117.96
|
||||
Peak output token throughput (tok/s): 148.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 288.50
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.48
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3576.31
|
||||
Median E2E Latency (ms): 2935.97
|
||||
P90 E2E Latency (ms): 5908.97
|
||||
P99 E2E Latency (ms): 8588.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 290.88
|
||||
Median TTFT (ms): 282.34
|
||||
P99 TTFT (ms): 332.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.54
|
||||
Median TPOT (ms): 6.97
|
||||
P99 TPOT (ms): 9.04
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.80
|
||||
Median ITL (ms): 6.81
|
||||
P95 ITL (ms): 13.51
|
||||
P99 ITL (ms): 26.99
|
||||
Max ITL (ms): 29.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 411.74
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 499614
|
||||
Request throughput (req/s): 2.43
|
||||
Input token throughput (tok/s): 1220.41
|
||||
Output token throughput (tok/s): 1214.97
|
||||
Peak output token throughput (tok/s): 2648.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 2435.38
|
||||
Concurrency: 96.30
|
||||
Accept length: 3.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 39648.76
|
||||
Median E2E Latency (ms): 39058.12
|
||||
P90 E2E Latency (ms): 57009.82
|
||||
P99 E2E Latency (ms): 68880.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 20613.80
|
||||
Median TTFT (ms): 21429.21
|
||||
P99 TTFT (ms): 29543.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.73
|
||||
Median TPOT (ms): 36.52
|
||||
P99 TPOT (ms): 67.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.13
|
||||
Median ITL (ms): 16.57
|
||||
P95 ITL (ms): 86.01
|
||||
P99 ITL (ms): 164.88
|
||||
Max ITL (ms): 1307.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
<Note>
|
||||
The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate GLM-5.1 benchmark run is planned.
|
||||
</Note>
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.000
|
||||
Latency: 32.470 s
|
||||
Output throughput: 642.044 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.860
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.932
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.640
|
||||
subject: college_computer_science, #q:100, acc: 0.900
|
||||
subject: college_mathematics, #q:100, acc: 0.810
|
||||
subject: college_medicine, #q:173, acc: 0.873
|
||||
subject: college_physics, #q:102, acc: 0.912
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.928
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.897
|
||||
subject: elementary_mathematics, #q:378, acc: 0.937
|
||||
subject: formal_logic, #q:126, acc: 0.778
|
||||
subject: global_facts, #q:100, acc: 0.710
|
||||
subject: high_school_biology, #q:310, acc: 0.961
|
||||
subject: high_school_chemistry, #q:203, acc: 0.847
|
||||
subject: high_school_computer_science, #q:100, acc: 0.960
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.960
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.984
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.923
|
||||
subject: high_school_mathematics, #q:270, acc: 0.696
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.821
|
||||
subject: high_school_psychology, #q:545, acc: 0.956
|
||||
subject: high_school_statistics, #q:216, acc: 0.889
|
||||
subject: high_school_us_history, #q:204, acc: 0.941
|
||||
subject: high_school_world_history, #q:237, acc: 0.945
|
||||
subject: human_aging, #q:223, acc: 0.857
|
||||
subject: human_sexuality, #q:131, acc: 0.908
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.907
|
||||
subject: logical_fallacies, #q:163, acc: 0.933
|
||||
subject: machine_learning, #q:112, acc: 0.830
|
||||
subject: management, #q:103, acc: 0.942
|
||||
subject: marketing, #q:234, acc: 0.940
|
||||
subject: medical_genetics, #q:100, acc: 0.990
|
||||
subject: miscellaneous, #q:783, acc: 0.959
|
||||
subject: moral_disputes, #q:346, acc: 0.873
|
||||
subject: moral_scenarios, #q:895, acc: 0.837
|
||||
subject: nutrition, #q:306, acc: 0.922
|
||||
subject: philosophy, #q:311, acc: 0.897
|
||||
subject: prehistory, #q:324, acc: 0.929
|
||||
subject: professional_accounting, #q:282, acc: 0.844
|
||||
subject: professional_law, #q:1534, acc: 0.714
|
||||
subject: professional_medicine, #q:272, acc: 0.941
|
||||
subject: professional_psychology, #q:612, acc: 0.913
|
||||
subject: public_relations, #q:110, acc: 0.791
|
||||
subject: security_studies, #q:245, acc: 0.878
|
||||
subject: sociology, #q:201, acc: 0.940
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.596
|
||||
subject: world_religions, #q:171, acc: 0.936
|
||||
Total latency: 165.275
|
||||
Average accuracy: 0.877
|
||||
```
|
||||
|
||||
### 5.3 AMD GPU Benchmarks
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
|
||||
|
||||
- MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends)
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
|
||||
@@ -0,0 +1,829 @@
|
||||
---
|
||||
title: GLM Glyph
|
||||
metatags:
|
||||
description: "Deploy GLM-Glyph with SGLang - community contribution guide for Zhipu AI's GLM Glyph model deployment."
|
||||
---
|
||||
|
||||
import { GLMGlyphDeployment } from '/src/snippets/autoregressive/glm-glyph-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options.
|
||||
|
||||
<GLMGlyphDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count. See the [GLM-4.5 cookbook page](/cookbook/autoregressive/GLM/GLM-4.5) for the full Thinking Budget usage example.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Thinking Mode
|
||||
|
||||
Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/Glyph \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
**Disable Thinking Mode:**
|
||||
|
||||
To disable thinking mode for a specific request:
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[{"role": "user", "content": "What is the capital of France?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Glyph supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/Glyph \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Model: Glyph
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/Glyph \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 17.03
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.59
|
||||
Input token throughput (tok/s): 358.17
|
||||
Output token throughput (tok/s): 247.74
|
||||
Peak output token throughput (tok/s): 251.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 605.91
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1702.14
|
||||
Median E2E Latency (ms): 1361.72
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 22.35
|
||||
Median TTFT (ms): 22.61
|
||||
P99 TTFT (ms): 23.76
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.99
|
||||
Median TPOT (ms): 3.99
|
||||
P99 TPOT (ms): 4.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.99
|
||||
Median ITL (ms): 3.99
|
||||
P95 ITL (ms): 4.03
|
||||
P99 ITL (ms): 4.12
|
||||
Max ITL (ms): 7.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 16.27
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40804
|
||||
Request throughput (req/s): 4.92
|
||||
Input token throughput (tok/s): 2438.06
|
||||
Output token throughput (tok/s): 2507.94
|
||||
Peak output token throughput (tok/s): 3069.00
|
||||
Peak concurrent requests: 26
|
||||
Total token throughput (tok/s): 4946.00
|
||||
Concurrency: 13.44
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2733.43
|
||||
Median E2E Latency (ms): 2892.98
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 33.10
|
||||
Median TTFT (ms): 27.73
|
||||
P99 TTFT (ms): 49.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.33
|
||||
Median TPOT (ms): 5.39
|
||||
P99 TPOT (ms): 5.86
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.30
|
||||
Median ITL (ms): 4.89
|
||||
P95 ITL (ms): 5.54
|
||||
P99 ITL (ms): 21.17
|
||||
Max ITL (ms): 25.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 25.67
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252657
|
||||
Request throughput (req/s): 19.48
|
||||
Input token throughput (tok/s): 9733.69
|
||||
Output token throughput (tok/s): 9843.99
|
||||
Peak output token throughput (tok/s): 13398.00
|
||||
Peak concurrent requests: 127
|
||||
Total token throughput (tok/s): 19577.68
|
||||
Concurrency: 89.49
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4593.75
|
||||
Median E2E Latency (ms): 4431.03
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 48.66
|
||||
Median TTFT (ms): 35.88
|
||||
P99 TTFT (ms): 120.61
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.10
|
||||
Median TPOT (ms): 9.55
|
||||
P99 TPOT (ms): 11.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.01
|
||||
Median ITL (ms): 6.51
|
||||
P95 ITL (ms): 23.19
|
||||
P99 ITL (ms): 25.54
|
||||
Max ITL (ms): 52.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 201.53
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44455
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 30.27
|
||||
Output token throughput (tok/s): 220.63
|
||||
Peak output token throughput (tok/s): 251.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 250.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20151.45
|
||||
Median E2E Latency (ms): 21576.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 2362.23
|
||||
Median TTFT (ms): 23.03
|
||||
P99 TTFT (ms): 21310.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.00
|
||||
Median TPOT (ms): 4.00
|
||||
P99 TPOT (ms): 4.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.00
|
||||
Median ITL (ms): 4.00
|
||||
P95 ITL (ms): 4.05
|
||||
P99 ITL (ms): 4.08
|
||||
Max ITL (ms): 5.67
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 118.67
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 318270
|
||||
Request throughput (req/s): 0.67
|
||||
Input token throughput (tok/s): 334.27
|
||||
Output token throughput (tok/s): 2682.26
|
||||
Peak output token throughput (tok/s): 3264.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3016.53
|
||||
Concurrency: 13.74
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20387.23
|
||||
Median E2E Latency (ms): 20466.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 132.47
|
||||
Median TTFT (ms): 27.19
|
||||
P99 TTFT (ms): 583.15
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.09
|
||||
Median TPOT (ms): 5.13
|
||||
P99 TPOT (ms): 5.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.09
|
||||
Median ITL (ms): 5.08
|
||||
P95 ITL (ms): 5.18
|
||||
P99 ITL (ms): 5.57
|
||||
Max ITL (ms): 522.26
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 150.00
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1300901
|
||||
Request throughput (req/s): 2.13
|
||||
Input token throughput (tok/s): 1059.59
|
||||
Output token throughput (tok/s): 8673.49
|
||||
Peak output token throughput (tok/s): 11899.00
|
||||
Peak concurrent requests: 71
|
||||
Total token throughput (tok/s): 9733.09
|
||||
Concurrency: 54.71
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 25645.42
|
||||
Median E2E Latency (ms): 26913.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 163.75
|
||||
Median TTFT (ms): 93.67
|
||||
P99 TTFT (ms): 426.19
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.27
|
||||
Median TPOT (ms): 6.39
|
||||
P99 TPOT (ms): 6.59
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.27
|
||||
Median ITL (ms): 0.17
|
||||
P95 ITL (ms): 32.94
|
||||
P99 ITL (ms): 67.89
|
||||
Max ITL (ms): 136.00
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
#### 5.1.3.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 17.44
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.57
|
||||
Input token throughput (tok/s): 2405.19
|
||||
Output token throughput (tok/s): 242.00
|
||||
Peak output token throughput (tok/s): 250.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 2647.19
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1742.54
|
||||
Median E2E Latency (ms): 1412.47
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 53.48
|
||||
Median TTFT (ms): 45.05
|
||||
P99 TTFT (ms): 98.57
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.01
|
||||
Median TPOT (ms): 4.01
|
||||
P99 TPOT (ms): 4.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.01
|
||||
Median ITL (ms): 4.01
|
||||
P95 ITL (ms): 4.06
|
||||
P99 ITL (ms): 4.09
|
||||
Max ITL (ms): 4.95
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 16.90
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41668
|
||||
Request throughput (req/s): 4.73
|
||||
Input token throughput (tok/s): 17753.58
|
||||
Output token throughput (tok/s): 2465.75
|
||||
Peak output token throughput (tok/s): 3005.00
|
||||
Peak concurrent requests: 25
|
||||
Total token throughput (tok/s): 20219.33
|
||||
Concurrency: 13.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2890.33
|
||||
Median E2E Latency (ms): 3069.55
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 41.46
|
||||
Median TTFT (ms): 31.75
|
||||
P99 TTFT (ms): 93.18
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.52
|
||||
Median TPOT (ms): 5.58
|
||||
P99 TPOT (ms): 6.14
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.48
|
||||
Median ITL (ms): 5.13
|
||||
P95 ITL (ms): 5.93
|
||||
P99 ITL (ms): 20.76
|
||||
Max ITL (ms): 36.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 35.54
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169994
|
||||
Request throughput (req/s): 9.01
|
||||
Input token throughput (tok/s): 35848.57
|
||||
Output token throughput (tok/s): 4783.96
|
||||
Peak output token throughput (tok/s): 8396.00
|
||||
Peak concurrent requests: 80
|
||||
Total token throughput (tok/s): 40632.53
|
||||
Concurrency: 59.26
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6580.96
|
||||
Median E2E Latency (ms): 6248.74
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 345.27
|
||||
Median TTFT (ms): 96.06
|
||||
P99 TTFT (ms): 2823.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.26
|
||||
Median TPOT (ms): 12.53
|
||||
P99 TPOT (ms): 23.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.76
|
||||
Median ITL (ms): 6.57
|
||||
P95 ITL (ms): 27.66
|
||||
P99 ITL (ms): 91.24
|
||||
Max ITL (ms): 2609.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.890
|
||||
Invalid: 0.000
|
||||
Latency: 3.718 s
|
||||
Output throughput: 5245.606 token/s
|
||||
```
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
title: GLM-OCR
|
||||
metatags:
|
||||
description: "Deploy GLM-OCR with SGLang - state-of-the-art OCR performance for complex document understanding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization.
|
||||
|
||||
The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
|
||||
- **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts.
|
||||
- **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments.
|
||||
- **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems.
|
||||
|
||||
For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding.
|
||||
|
||||
import { GLMOCRDeployment } from '/src/snippets/autoregressive/glm-ocr-deployment.jsx'
|
||||
|
||||
<GLMOCRDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT.
|
||||
- **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency.
|
||||
- **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 OCR Image Processing
|
||||
|
||||
GLM-OCR supports OCR tasks on various document types. Here's a basic example:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Please extract all text from this image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-OCR",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 2.29s
|
||||
Generated text: CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
|
||||
```
|
||||
|
||||
#### 4.2.2 Complex Document Processing
|
||||
|
||||
GLM-OCR excels at processing complex documents including:
|
||||
|
||||
- **Tables**: Accurate extraction of tabular data with structure preservation
|
||||
- **Formulas**: Mathematical formula recognition
|
||||
- **Code Documents**: Source code extraction from screenshots
|
||||
- **Seals and Stamps**: Recognition of seals and stamps in documents
|
||||
- **Multi-layout Documents**: Mixed content with text, images, and tables
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
# Example: Processing a document with tables
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "YOUR_DOCUMENT_IMAGE_URL"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Please extract the table content from this document and format it as markdown."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-OCR",
|
||||
messages=messages,
|
||||
max_tokens=4096
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.1.1 OCRBench Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 -m lmms_eval \
|
||||
--model openai_compatible \
|
||||
--model_args "model_version=zai-org/GLM-OCR" \
|
||||
--tasks ocrbench \
|
||||
--batch_size 128 \
|
||||
--log_samples \
|
||||
--log_samples_suffix "openai_compatible" \
|
||||
--output_path ./logs
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stderr</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yaml</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench_accuracy</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.806</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 OmniDocBench V1.5
|
||||
|
||||
GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks.
|
||||
@@ -0,0 +1,306 @@
|
||||
---
|
||||
title: DiffusionGemma
|
||||
metatags:
|
||||
description: "Deploy DiffusionGemma with SGLang - Google's uniform-state renoising block-diffusion language model (26B-A4B MoE) served with the Gemma4Renoise sampler."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
DiffusionGemma is a uniform-state (renoising) block-diffusion language model from Google. An encoder builds causal context, and a decoder denoises a fixed-length bidirectional canvas of `canvas_length` tokens. The `Gemma4Renoise` sampler runs `max_denoising_steps` reverse steps over the canvas, feeding the previous step's logits back as self-conditioning and emitting the greedy argmax of the processed logits.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Uniform-State Renoising**: The canvas starts from random tokens and is refined each step by accepting confident positions and re-noising the rest, with no mask token.
|
||||
- **Encoder / Decoder Canvas**: The encoder produces causal context KV, the decoder attends bidirectionally over the canvas.
|
||||
- **Self-Conditioning**: Each step conditions on the previous step's logits.
|
||||
- **EntropyBound Acceptance**: Each step accepts the lowest-entropy canvas positions within an entropy budget and re-noises the rest.
|
||||
- **StableAndConfident Stopping**: A canvas stops early once it is stable and confident.
|
||||
- **MoE Architecture**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference.
|
||||
- **Multimodal Input**: Accepts text and image inputs (via a ~550M vision encoder) and generates text output.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "40.0%"}} />
|
||||
<col style={{width: "30.0%"}} />
|
||||
<col style={{width: "30.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameters</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[google/diffusiongemma-26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MoE, uniform-state diffusion (text + image)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>25.2B total / 3.8B active</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Architecture Specifications:**
|
||||
|
||||
| Spec | Value |
|
||||
| --- | --- |
|
||||
| Total Parameters | 25.2B |
|
||||
| Active Parameters | 3.8B |
|
||||
| Layers | 30 |
|
||||
| Sliding Window | 1024 tokens |
|
||||
| Context Length | Up to 256K tokens |
|
||||
| Canvas Length | 256 |
|
||||
| Vocabulary Size | 262K |
|
||||
| Experts | 8 active / 128 total + 1 shared |
|
||||
| Supported Modalities | Text, Image |
|
||||
| Vision Encoder | ~550M parameters |
|
||||
|
||||
**License:**
|
||||
|
||||
Refer to the model card for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
The checkpoint ships its own modeling code, so `--trust-remote-code` is required when serving.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The required runtime settings are applied automatically for `Gemma4Renoise` (the Triton attention backend, eager mode, and unchunked prefill, needed because the full-attention head_dim is 512 and the canvas uses bidirectional attention), so a default launch works:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path google/diffusiongemma-26B-A4B-it \
|
||||
--dllm-algorithm Gemma4Renoise \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**dLLM-Specific Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`Gemma4Renoise`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required to load the checkpoint's modeling code</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm-config`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional YAML overriding the renoise schedule</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Checkpoint defaults</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The attention backend, eager mode, and unchunked prefill are selected automatically for `Gemma4Renoise`, so they do not need to be passed on the command line.
|
||||
|
||||
Sampling is governed by the renoise schedule. Request-level `logprobs`, penalties, `logit_bias`, and grammar / structured output (`json_schema` / `regex` / `ebnf` / `structural_tag`) are not applied and are rejected with a 400. Core sampling controls (`temperature`, `top_k`, `top_p`) are accepted but have no effect. Streaming is block-level: one fully-denoised canvas per chunk.
|
||||
|
||||
**Gemma4Renoise Config** (defaults follow the checkpoint's `generation_config.json`):
|
||||
|
||||
```yaml Config
|
||||
# Number of reverse denoising steps per canvas.
|
||||
max_denoising_steps: 48
|
||||
# Optional. Makes the renoise sampling reproducible (also shared across TP ranks).
|
||||
seed: 1234
|
||||
sampler_config:
|
||||
# Entropy budget. Accept the lowest-entropy canvas positions within this bound each step (the rest are re-noised).
|
||||
entropy_bound: 0.1
|
||||
# Linear temperature schedule applied over the denoising steps.
|
||||
temperature_schedule:
|
||||
t_min: 0.4
|
||||
t_max: 0.8
|
||||
# Stop early once the canvas is stable and confident.
|
||||
stopping_config:
|
||||
confidence_threshold: 0.005
|
||||
stability_threshold: 1
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment
|
||||
|
||||
Start the server with the command from [Section 3.1](#3-1-basic-configuration).
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="google/diffusiongemma-26B-A4B-it",
|
||||
messages=[
|
||||
{"role": "user", "content": "What are the key differences between TCP and UDP?"}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 4.3 Streaming
|
||||
|
||||
Streaming emits one fully-denoised canvas per chunk.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="google/diffusiongemma-26B-A4B-it",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
Not benchmarked for speed.
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Full test splits, every item scored (no failed-request exclusions). Text MCQ benchmarks use greedy generate-and-parse, MATH uses boxed-answer extraction plus sympy equivalence. MMLU, ARC-Challenge, and MATH-500 are the mean of two independent server launches.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50.0%"}} />
|
||||
<col style={{width: "50.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Benchmark</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Score</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GSM8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>95.4%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ARC-Challenge</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>91.6%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HumanEval</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.7% pass@1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMLU</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>76.2%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMLU-Pro</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>73.7%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GSM-Symbolic</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.2%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MATH-500</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>72.1%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AIME-2026</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>10.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HMMT-Feb-2025</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>10.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPQA-main</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>59.2%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Multimodal, full standard split per task (MMMU / MMMU-Pro / MMStar / AI2D as multiple-choice, MathVista testmini, DocVQA by ANLS, ChartQA by relaxed accuracy):
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50.0%"}} />
|
||||
<col style={{width: "50.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Multimodal benchmark</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Score</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMMU (val, MC)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64.9%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMMU-Pro (standard 10-opt, MC)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>57.3%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MathVista (testmini)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>68.4%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DocVQA (val)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>85.9%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ChartQA (test)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>61.7%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI2D (test)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>78.7%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMStar (val)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>65.9%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: EmbeddingGemma
|
||||
description: Serve Google's EmbeddingGemma text embedding model with SGLang.
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
[EmbeddingGemma](https://huggingface.co/google/embeddinggemma-300m) is Google's 300M-parameter text embedding model. SGLang detects its bidirectional Gemma 3 encoder, applies normalized mean pooling, and serves embeddings through the OpenAI-compatible `/v1/embeddings` endpoint.
|
||||
|
||||
On NVIDIA CUDA, SGLang uses breakable CUDA graph (BCG) for its complete prefill by default. It also disables prefix caching and chunked prefill, which are incompatible with this bidirectional encoder.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NVIDIA CUDA GPU.
|
||||
- A Hugging Face account that has accepted the [EmbeddingGemma license](https://huggingface.co/google/embeddinggemma-300m).
|
||||
- A Hugging Face access token. Export it before starting the server so it can download the gated checkpoint:
|
||||
|
||||
```bash
|
||||
export HF_TOKEN=<your-hugging-face-token>
|
||||
```
|
||||
|
||||
Install an SGLang build that includes EmbeddingGemma support:
|
||||
|
||||
```bash
|
||||
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
```
|
||||
|
||||
## Start the server
|
||||
|
||||
The standard configuration detects EmbeddingGemma and enables embedding mode,
|
||||
BCG, and the checkpoint's BF16 dtype automatically:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path google/embeddinggemma-300m \
|
||||
--host 0.0.0.0
|
||||
```
|
||||
|
||||
### Hopper performance defaults
|
||||
|
||||
On H100 and H200, SGLang automatically selects FA3 and captures BCG through
|
||||
16,384 tokens, covering eight 2K embedding requests in one replay. No extra
|
||||
performance flags are required for this workload.
|
||||
|
||||
To capture larger aggregate prefills, raise the BCG tier explicitly:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path google/embeddinggemma-300m \
|
||||
--cuda-graph-max-bs-prefill 32768 \
|
||||
--host 0.0.0.0
|
||||
```
|
||||
|
||||
EmbeddingGemma automatically enables batch tokenization for list-valued
|
||||
embedding requests, so do not add a separate tokenizer batching flag.
|
||||
|
||||
## Create embeddings
|
||||
|
||||
Send one string or a batch of strings to the OpenAI-compatible endpoint:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:30000/v1/embeddings \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "google/embeddinggemma-300m",
|
||||
"input": [
|
||||
"A short guide to serving text embeddings.",
|
||||
"Vector search retrieves semantically similar documents."
|
||||
],
|
||||
"encoding_format": "float"
|
||||
}'
|
||||
```
|
||||
|
||||
See [OpenAI-compatible embedding APIs](/docs/basic_usage/openai_api_embeddings) for Python and OpenAI client examples.
|
||||
|
||||
## Deployment behavior
|
||||
|
||||
EmbeddingGemma performs bidirectional attention over the complete input, so reusing a prefix KV cache or splitting the input into chunked prefills would produce incorrect attention states. SGLang applies the required settings automatically:
|
||||
|
||||
- disables RadixAttention prefix caching;
|
||||
- disables chunked prefill;
|
||||
- disables the decode CUDA graph because this is an embedding-only model;
|
||||
- uses BCG for CUDA prefill;
|
||||
- uses the FlashAttention raw-K/V path when the prefill backend is FA3 or FA4 on supported Hopper and Blackwell CUDA GPUs.
|
||||
|
||||
No prefill CUDA-graph override is required for this recipe. Keep BCG enabled to use the optimized EmbeddingGemma path.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,700 @@
|
||||
---
|
||||
title: LLaDA 2.1
|
||||
metatags:
|
||||
description: "Deploy LLaDA 2.1 with SGLang - large-scale discrete diffusion language model with parallel token generation, iterative denoising, MoE architecture, and reinforcement learning for reasoning."
|
||||
---
|
||||
|
||||
import { LLaDA21Deployment } from '/src/snippets/autoregressive/llada-21-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight
|
||||
- **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores
|
||||
- **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling
|
||||
- **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following
|
||||
- **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFace</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-mini**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE (20 layers, 16 attention heads)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-flash**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:**
|
||||
|
||||
Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs.
|
||||
|
||||
<LLaDA21Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**dLLM-Specific Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`JointThreshold`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for LLaDA model loading</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.8`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-running-requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum concurrent requests</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1` (for best quality)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention computation backend</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Decoding Mode Comparison:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Threshold</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Speed</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Quality</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quality Mode (Q)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Conservative</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Moderate</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Higher benchmark scores</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Accuracy-critical tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Speed Mode (S)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Aggressive</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Very fast, relies on T2T editing</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Slightly lower</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput-critical tasks</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Hardware Requirements:**
|
||||
|
||||
- **LLaDA2.1-mini (16B)**: ~47 GB VRAM, runs on a single GPU (TP=1)
|
||||
- **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200)
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment
|
||||
|
||||
Start the server using the command generated above, for example:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-mini \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 1 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
**Simple Completion Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain what a diffusion language model is in simple terms."}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Sure! Let's break it down in simple terms.
|
||||
|
||||
A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text.
|
||||
|
||||
Here’s how it works, using a simple real-life analogy:
|
||||
|
||||
Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together.
|
||||
|
||||
Now, think of the process like this:
|
||||
|
||||
1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper).
|
||||
2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book.
|
||||
3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start.
|
||||
4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word.
|
||||
|
||||
So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story.
|
||||
|
||||
In short:
|
||||
A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned.
|
||||
```
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Streaming
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
Here are several ways to implement the Fibonacci sequence in Python:
|
||||
|
||||
## 1. Recursive Approach (Simple but Inefficient)
|
||||
|
||||
```python
|
||||
def fibonacci_recursive(n):
|
||||
"""
|
||||
Compute the nth Fibonacci number using recursion.
|
||||
|
||||
Args:
|
||||
n (int): The position in the Fibonacci sequence (0-indexed)
|
||||
|
||||
Returns:
|
||||
int: The nth Fibonacci number
|
||||
|
||||
Raises:
|
||||
ValueError: If n is negative
|
||||
"""
|
||||
if n < 0:
|
||||
raise ValueError("n must be non-negative")
|
||||
|
||||
if n <= 1:
|
||||
return n
|
||||
|
||||
return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
|
||||
|
||||
# Example usage
|
||||
print(fibonacci_recursive(10)) # Output: 55
|
||||
```
|
||||
|
||||
## 2. Iterative Approach (Efficient)
|
||||
...
|
||||
````
|
||||
|
||||
#### 4.3.2 Code Generation
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."}
|
||||
],
|
||||
max_tokens=2048
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
```python
|
||||
def is_palindrome(s):
|
||||
"""
|
||||
Check if a string is a palindrome.
|
||||
|
||||
A palindrome is a word, phrase, or sequence that reads the same backward as forward.
|
||||
This function ignores case, spaces, punctuation, and non characters characters.
|
||||
|
||||
Args:
|
||||
s (str): The string to check
|
||||
|
||||
Returns:
|
||||
bool: True if the string is a palindrome, False otherwise
|
||||
|
||||
Examples:
|
||||
>>> is_palindrome("racecar")
|
||||
True
|
||||
>>> is_palindrome("A man a plan a canal Panama")
|
||||
True
|
||||
>>> is_palindrome("race a car")
|
||||
False
|
||||
>>> is_palindrome("")
|
||||
True
|
||||
>>> is_palindrome("a")
|
||||
True
|
||||
"""
|
||||
# Remove non-alphanumeric characters and convert to lowercase
|
||||
cleaned = ''.join(char.lower() for char in s if char.isalnum())
|
||||
|
||||
# Check if the cleaned string reads the same forwards and backwards
|
||||
return cleaned == cleaned[::-1]
|
||||
|
||||
# Test cases
|
||||
def test_is_palindrome():
|
||||
"""Test the is_palindrome function with various inputs."""
|
||||
|
||||
# Test basic palindromes
|
||||
assert is_palindrome("racecar") == True
|
||||
assert is_palindrome("level") == True
|
||||
assert is_palindrome("madam") == True
|
||||
assert is_palindrome("radar") == True
|
||||
|
||||
# Test palindromes with spaces and punctuation
|
||||
assert is_palindrome("A man a plan a canal Panama") == True
|
||||
assert is_palindrome("race a car") == False
|
||||
assert is_palindrome("Was it a car or a cat I saw?") == True
|
||||
assert is_palindrome("Madam, I'm Adam") == True
|
||||
|
||||
# Test edge cases
|
||||
assert is_palindrome("") == True
|
||||
assert is_palindrome("a") == True
|
||||
assert is_palindrome("A") == True
|
||||
assert is_palindrome("Aa") == True
|
||||
|
||||
# Test non-palindromes
|
||||
assert is_palindrome("hello") == False
|
||||
assert is_palindrome("world") == False
|
||||
assert is_palindrome("python") == False
|
||||
|
||||
# Test single characters
|
||||
assert is_palindrome("1") == True
|
||||
assert is_palindrome("1") == True
|
||||
|
||||
print("All tests passed!")
|
||||
|
||||
# Run the tests
|
||||
if __name__ == "__main__":
|
||||
# Example usage
|
||||
print("Testing isalindrome function:")
|
||||
print(f"'racecar' {is_palindrome('racecar')}")
|
||||
print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}")
|
||||
print(f"'race a car': {is_palindrome('race a car')}")
|
||||
print(f"'hello': {is_palindrome('hello')}")
|
||||
|
||||
# Run tests
|
||||
test_is_palindrome()
|
||||
```
|
||||
|
||||
This implementation includes:
|
||||
|
||||
1. **Comprehensive function** `is_palindrome()` that:
|
||||
- Ignores case by converting to lowercase
|
||||
- Removes all non-alphanumeric characters (spaces, punctuation, etc.)
|
||||
- Uses string slicing (`[::-1]`) to reverse the string
|
||||
|
||||
2. **Detailed docstring** explaining:
|
||||
- What the function does
|
||||
- How it works
|
||||
- Return value
|
||||
- Examples of usage
|
||||
|
||||
3. **Extensive test cases** covering:
|
||||
- Basic palindromes
|
||||
- Palindromes with spaces and punctuation
|
||||
- Edge cases (empty string, single character)
|
||||
- Non-palindromes
|
||||
- Mixed case scenarios
|
||||
|
||||
4. **Test function** that uses assertions to verify the function works correctly
|
||||
|
||||
The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama".
|
||||
````
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 (4x)
|
||||
- SGLang Version: 0.5.8+
|
||||
|
||||
#### 5.1.1 LLaDA2.1-mini
|
||||
|
||||
**Model Deployment:**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-mini \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 1 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer
|
||||
```
|
||||
|
||||
- Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-mini \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Latency Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 9.90
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3433
|
||||
Request throughput (req/s): 1.01
|
||||
Input token throughput (tok/s): 616.26
|
||||
Output token throughput (tok/s): 426.26
|
||||
Peak output token throughput (tok/s): 1010.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 1042.53
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 988.87
|
||||
Median E2E Latency (ms): 655.27
|
||||
P90 E2E Latency (ms): 1952.50
|
||||
P99 E2E Latency (ms): 2932.19
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 152.74
|
||||
Median TTFT (ms): 150.37
|
||||
P99 TTFT (ms): 229.78
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.16
|
||||
Median TPOT (ms): 2.08
|
||||
P99 TPOT (ms): 3.72
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.10
|
||||
Median ITL (ms): 1.99
|
||||
P95 ITL (ms): 4.03
|
||||
P99 ITL (ms): 6.34
|
||||
Max ITL (ms): 26.59
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-mini \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Throughput Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 467.74
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 189717
|
||||
Request throughput (req/s): 1.07
|
||||
Input token throughput (tok/s): 534.12
|
||||
Output token throughput (tok/s): 540.17
|
||||
Peak output token throughput (tok/s): 1753.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 1074.30
|
||||
Concurrency: 90.77
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 84912.27
|
||||
Median E2E Latency (ms): 86564.26
|
||||
P90 E2E Latency (ms): 110567.26
|
||||
P99 E2E Latency (ms): 114303.38
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 83920.39
|
||||
Median TTFT (ms): 85669.54
|
||||
P99 TTFT (ms): 112969.91
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.67
|
||||
Median TPOT (ms): 1.65
|
||||
P99 TPOT (ms): 4.43
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.69
|
||||
Median ITL (ms): 1.46
|
||||
P95 ITL (ms): 3.96
|
||||
P99 ITL (ms): 4.84
|
||||
Max ITL (ms): 92.08
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 LLaDA2.1-flash
|
||||
|
||||
**Model Deployment:**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-flash \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer
|
||||
```
|
||||
|
||||
- Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Latency Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 14.46
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3276
|
||||
Request throughput (req/s): 0.69
|
||||
Input token throughput (tok/s): 421.79
|
||||
Output token throughput (tok/s): 291.75
|
||||
Peak output token throughput (tok/s): 676.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 713.53
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1445.16
|
||||
Median E2E Latency (ms): 968.06
|
||||
P90 E2E Latency (ms): 3101.86
|
||||
P99 E2E Latency (ms): 4208.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 231.63
|
||||
Median TTFT (ms): 242.67
|
||||
P99 TTFT (ms): 341.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.04
|
||||
Median TPOT (ms): 2.79
|
||||
P99 TPOT (ms): 5.33
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.05
|
||||
Median ITL (ms): 2.41
|
||||
P95 ITL (ms): 7.25
|
||||
P99 ITL (ms): 8.27
|
||||
Max ITL (ms): 29.27
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Throughput Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 671.85
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 177961
|
||||
Request throughput (req/s): 0.74
|
||||
Input token throughput (tok/s): 371.85
|
||||
Output token throughput (tok/s): 376.07
|
||||
Peak output token throughput (tok/s): 1521.00
|
||||
Peak concurrent requests: 103
|
||||
Total token throughput (tok/s): 747.92
|
||||
Concurrency: 91.28
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 122658.36
|
||||
Median E2E Latency (ms): 125265.55
|
||||
P90 E2E Latency (ms): 159554.07
|
||||
P99 E2E Latency (ms): 165174.88
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 121009.17
|
||||
Median TTFT (ms): 124437.80
|
||||
P99 TTFT (ms): 163579.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.73
|
||||
Median TPOT (ms): 2.16
|
||||
P99 TPOT (ms): 7.13
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.38
|
||||
Median ITL (ms): 1.40
|
||||
P95 ITL (ms): 6.89
|
||||
P99 ITL (ms): 8.60
|
||||
Max ITL (ms): 176.78
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.895
|
||||
Invalid: 0.000
|
||||
Latency: 100.552 s
|
||||
Output throughput: 262.094 token/s
|
||||
```
|
||||
@@ -0,0 +1,217 @@
|
||||
---
|
||||
title: Ling-2.5-1T
|
||||
metatags:
|
||||
description: "Deploy Ling-2.5-1T with SGLang - 1T parameter MoE model with 63B active parameters, trillion-scale context length up to 1M tokens, and agentic tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens.
|
||||
- **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume ~4x the output tokens.
|
||||
- **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following.
|
||||
- **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4.
|
||||
- **Context Length**: 256K -> 1M (YaRN)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ling-2.5-1T runs on the standard SGLang Docker image:
|
||||
|
||||
```bash Command
|
||||
# NVIDIA (H200 / B200 / GB200 / GB300)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/install) guide for setup instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform.
|
||||
|
||||
import { Ling251TDeployment } from '/src/snippets/autoregressive/ling-25-1t-deployment.jsx'
|
||||
|
||||
<Ling251TDeployment />
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
|
||||
- `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes.
|
||||
- Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading.
|
||||
- On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For example, launch the server on 2 H200 nodes:
|
||||
|
||||
```bash Command
|
||||
export MASTER_IP=10.10.0.1 # The IP of Node 0
|
||||
export PORT=30000
|
||||
export DIST_PORT=50000
|
||||
|
||||
# Node 0:
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path inclusionAI/Ling-2.5-1T \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--pp-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
|
||||
--mem-frac 0.95
|
||||
|
||||
|
||||
# Node 1:
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path inclusionAI/Ling-2.5-1T \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--pp-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
|
||||
--mem-frac 0.95
|
||||
```
|
||||
|
||||
Once the server is running, send requests to the master node:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
|
||||
```
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "e82af153da844ee6aed7a27a3187f2f4",
|
||||
"object": "chat.completion",
|
||||
"created": 1771216764,
|
||||
"model": "auto",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "The capital of France is **Paris**.\n\n**Additional details:**\n* It is the largest city in France.\n* It is located in the north-central part of the country along the Seine River.\n* Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).",
|
||||
"reasoning_content": null,
|
||||
"tool_calls": null
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "stop",
|
||||
"matched_stop": 156895
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 25,
|
||||
"total_tokens": 93,
|
||||
"completion_tokens": 68,
|
||||
"prompt_tokens_details": null,
|
||||
"reasoning_tokens": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For more API usage examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "inclusionAI/Ling-2.5-1T",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"description": "Search for information on the internet",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "The search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto"
|
||||
}'
|
||||
```
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "b968e45c7d414f7482c8ffc0f9c6b688",
|
||||
"object": "chat.completion",
|
||||
"created": 1771216520,
|
||||
"model": "inclusionAI/Ling-2.5-1T",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"reasoning_content": null,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_e75f711d8ad840ed9d382c9e",
|
||||
"index": 0,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"arguments": "{\"query\": \"latest news about AI\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "tool_calls",
|
||||
"matched_stop": null
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 173,
|
||||
"total_tokens": 196,
|
||||
"completion_tokens": 23,
|
||||
"prompt_tokens_details": null,
|
||||
"reasoning_tokens": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 45.410 s
|
||||
Output throughput: 560.642 token/s
|
||||
```
|
||||
@@ -0,0 +1,223 @@
|
||||
---
|
||||
title: Ling-2.6
|
||||
metatags:
|
||||
description: "Deploy the Ling-2.6 family with SGLang - Ling-2.6-flash (104B total / 7.4B active BF16 MoE) and Ling-2.6-1T (~1T FP8 MoE) with hybrid linear attention and agentic tool calling."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
The **Ling-2.6** family from inclusionAI is the next iteration of the Ling instant-model series. Continuing the architectural direction set by Ling-2.5, Ling-2.6 doubles down on **inference efficiency**, **token efficiency**, and **agent performance** — staying competitive with frontier instant models while being faster, leaner, and better suited for production agent workloads.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid Linear Attention**: A `1:7 MLA + Lightning Linear` hybrid built on top of a highly sparse MoE backbone. Compared with same-class SOTA models, Ling-2.6-flash shows up to ~4× higher prefill and decode throughput in long-context scenarios; Ling-2.6-1T is shipped in FP8 so it fits a single GB300 node with `--tp 4`.
|
||||
- **Token Efficiency**: Trained with explicit token-efficiency objectives. On the full Artificial Analysis suite, Ling-2.6-flash uses only ~15M output tokens while remaining competitive — a meaningfully stronger intelligence-per-token profile than long-reasoning peers.
|
||||
- **Agentic Capabilities**: Refined for tool use, multi-step planning, and long-horizon execution. Reaches SOTA-class results on **BFCL-V4**, **TAU2-bench**, **SWE-bench Verified**, **Claw-Eval**, and **PinchBench**, and is validated against Claude Code, Kilo Code, Qwen Code, Hermes Agent, and OpenClaw.
|
||||
- **Long Context**: Native 128K, extendable to **256K (Ling-2.6-flash)** and **256K → 1M (Ling-2.6-1T via YaRN)**.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16**: [inclusionAI/Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) — 104B total / 7.4B active
|
||||
- **FP8 (E4M3)**: [inclusionAI/Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) — ~1T total
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Ling-2.6-flash
|
||||
|
||||
Ling-2.6-flash is a 104B/7.4B-active MoE that runs comfortably on a single 4-GPU node. Use the selector below to generate the launch command for your hardware.
|
||||
|
||||
import { Ling26FlashDeployment } from '/src/snippets/autoregressive/ling-26-flash-deployment.jsx'
|
||||
|
||||
<Ling26FlashDeployment />
|
||||
|
||||
#### Configuration Tips
|
||||
|
||||
- `--trust-remote-code` is required (custom `BailingMoeV2_5ForCausalLM` modeling code).
|
||||
- `--tp-size 4` is the reference layout. On 4× H20-3e the model reaches ~340 tokens/s decode at TP=4, batch 32.
|
||||
- Native context is 128K. Enable YaRN (`--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 2.0, ...}}'`) to extend to 256K — the snippet does this for you.
|
||||
- `--tool-call-parser qwen25` matches the model's `<tool_call>...</tool_call>` schema.
|
||||
- The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6 is a controllable-reasoning model whose chat template defaults to `detailed thinking off`; the SGLang `qwen3` reasoning parser, in contrast, assumes default-thinking semantics and would mis-route normal output into `reasoning_content`. Only enable it if you specifically want `<think>...</think>` blocks split out — see [§4.3 Thinking Mode](#4-3-thinking-mode).
|
||||
- **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example.
|
||||
|
||||
### 3.2 Ling-2.6-1T
|
||||
|
||||
Ling-2.6-1T ships in **FP8 (E4M3)**, so unlike Ling-2.5-1T it fits a **single GB300 node with `--tp 4`**. On smaller GPUs (H200/B200), a 2-node deployment with `--pp-size 2` is required.
|
||||
|
||||
import { Ling261TDeployment } from '/src/snippets/autoregressive/ling-26-1t-deployment.jsx'
|
||||
|
||||
<Ling261TDeployment />
|
||||
|
||||
#### Configuration Tips
|
||||
|
||||
- `--trust-remote-code` is required for the custom modeling code.
|
||||
- `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` significantly speeds up the multi-shard FP8 weight load (26 safetensors shards + an MTP layer).
|
||||
- Use `--tool-call-parser qwen` for tool calling.
|
||||
- The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6's chat template defaults to `detailed thinking off`, while SGLang's `qwen3` reasoning parser assumes default-thinking semantics — combining the two requires a per-request workaround for tool calls (see [§4.3 Thinking Mode](#4-3-thinking-mode)). Only enable `--reasoning-parser qwen3` if you specifically want `<think>...</think>` blocks split into `reasoning_content`.
|
||||
- For 2-node deployments, set `MASTER_IP`, `PORT`, and `DIST_PORT` consistently across both nodes.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
For example, launch a Ling-2.6-1T server on a single GB300 node:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ling-2.6-1T \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
|
||||
```
|
||||
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "...",
|
||||
"object": "chat.completion",
|
||||
"model": "auto",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "The capital of France is **Paris**.",
|
||||
"reasoning_content": null,
|
||||
"tool_calls": null
|
||||
},
|
||||
"finish_reason": "stop"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"description": "Search for information on the internet",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "The search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto"
|
||||
}'
|
||||
```
|
||||
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_...",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"arguments": "{\"query\": \"latest news about AI\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"finish_reason": "tool_calls"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 Thinking Mode
|
||||
|
||||
Both Ling-2.6-flash and Ling-2.6-1T are **controllable-reasoning** models. Their chat template uses textual directives in the system message — `detailed thinking on` or `detailed thinking off` — to toggle thinking. The template **defaults to `detailed thinking off`** when neither phrase is present, and it does **not** read the Qwen3-style `enable_thinking` template variable.
|
||||
|
||||
#### Enabling thinking
|
||||
|
||||
Include `detailed thinking on` in the first system message:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [
|
||||
{"role": "system", "content": "detailed thinking on"},
|
||||
{"role": "user", "content": "If a box has 12 red balls and 8 blue balls, then 5 red balls are removed, how many balls remain?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
If you already have a system prompt, append the directive on its own line:
|
||||
|
||||
```json
|
||||
{"role": "system", "content": "You are a helpful assistant.\ndetailed thinking on"}
|
||||
```
|
||||
|
||||
When thinking is on, the model emits `<think>...</think>` blocks before its final answer. To get those split into `message.reasoning_content` automatically, also launch the server with `--reasoning-parser qwen3`.
|
||||
|
||||
#### Caveat: `--reasoning-parser qwen3` + tool calling
|
||||
|
||||
The SGLang `qwen3` reasoning parser was written for Qwen3, where models are **default-thinking** and clients opt out via `chat_template_kwargs.enable_thinking=false`. Ling-2.6 is the opposite — default-non-thinking, with toggling done in the system message. As a result, when the server is launched with **both** `--tool-call-parser qwen` and `--reasoning-parser qwen3`, every tool-call request must include `chat_template_kwargs.enable_thinking=false`, otherwise the parser routes the `<tool_call>...</tool_call>` block into `reasoning_content` instead of `message.tool_calls`:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [...],
|
||||
"tool_choice": "auto",
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
`enable_thinking` here is consumed by the SGLang reasoning parser, **not** by the chat template — Ling-2.6's template ignores it. For the simplest configuration, just omit `--reasoning-parser qwen3` and toggle thinking via the system message.
|
||||
|
||||
For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request).
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K (Ling-2.6-1T, GB300 × 4)
|
||||
|
||||
Reference run on a single GB300 node with `--tp 4`:
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.9621 (1269 / 1319)
|
||||
```
|
||||
|
||||
For Ling-2.6-flash, see the official numbers on the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash) (BFCL-V4, TAU2-bench, SWE-bench Verified, Claw-Eval, PinchBench, Artificial Analysis).
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
title: Ring-2.5-1T
|
||||
metatags:
|
||||
description: "Deploy Ring-2.5-1T with SGLang - world's first open-source 1T parameter reasoning model with hybrid linear attention, deep reasoning, and agentic tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Model**: ~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention)
|
||||
- **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens
|
||||
- **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback
|
||||
- **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training
|
||||
- **Tool Calling**: Supports function calling with XML-style tool call format
|
||||
- **Context Length**: 128K -> 256K (YaRN)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ring-2.5-1T runs on the standard SGLang Docker image:
|
||||
|
||||
```bash Command
|
||||
# NVIDIA (H200 / B200 / GB200 / GB300)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# For MI300X/325X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
|
||||
|
||||
# For MI355X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
import { Ring251TDeployment } from '/src/snippets/autoregressive/ring-25-1t-deployment.jsx'
|
||||
|
||||
<Ring251TDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
|
||||
- The model uses FP8 quantization (compressed-tensors format).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy Ring-2.5-1T with the following command (on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.5-1T",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Output Example</summary>
|
||||
|
||||
````text Output
|
||||
We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution.
|
||||
|
||||
We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36.
|
||||
|
||||
But we need to present step by step. Also ensure it's clear.
|
||||
|
||||
We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36.
|
||||
|
||||
But any method is fine.
|
||||
|
||||
We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)."
|
||||
|
||||
We'll show:
|
||||
|
||||
15% = 15/100 = 0.15
|
||||
|
||||
Then 0.15 × 240 = 36.
|
||||
|
||||
Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36.
|
||||
|
||||
Finally, answer: 36.
|
||||
|
||||
We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36."
|
||||
|
||||
We'll present as a final answer: \boxed{36}.
|
||||
|
||||
However, we need to provide step-by-step solution as per instructions. We'll write a full explanation.
|
||||
|
||||
We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36.
|
||||
|
||||
Alr.
|
||||
|
||||
I think that's it.
|
||||
|
||||
|
||||
**Step 1:** Write 15% as a fraction or decimal.
|
||||
\[ 15\% = \frac{15}{100} = 0.15\]
|
||||
|
||||
**Step 2:** Multiply the number (240) by this fraction/decimal.
|
||||
\[ 240 \times 0.15 = 36\]
|
||||
|
||||
Alternatively, using the fraction:
|
||||
\[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\]
|
||||
|
||||
**Conclusion:** 15% of 240 is 36.
|
||||
|
||||
\[ \boxed{36} \]
|
||||
````
|
||||
|
||||
</details>
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
To enable tool calling, add `--tool-call-parser qwen` when launching the server.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser qwen \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.5-1T",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
[ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K
|
||||
|
||||
- Deployment Command
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp-size 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.010
|
||||
Latency: 615.833 s
|
||||
Output throughput: 412.360 token/s
|
||||
```
|
||||
@@ -0,0 +1,374 @@
|
||||
---
|
||||
title: Ring-2.6-1T
|
||||
metatags:
|
||||
description: "Deploy Ring-2.6-1T with SGLang - a trillion-parameter InclusionAI reasoning model for agent workflows, high/xhigh reasoning effort, and tool use."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is InclusionAI's trillion-parameter flagship reasoning model for real-world complex task execution. It targets agent workflows, engineering development, scientific research analysis, enterprise automation, and other long-horizon settings where the model must plan, use tools, recover from intermediate errors, and keep context across multiple steps.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Reasoning Model**: `BailingMoeV2_5ForCausalLM` with a `bailing_hybrid` architecture, 80 hidden layers, 256 routed experts, 8 selected experts per token, and FP8 compressed-tensors weights.
|
||||
- **Agent Execution**: Designed for multi-step task decomposition, tool collaboration, context continuation, and long-horizon execution. The model card reports 87.60 on PinchBench, 63.82 on ClawEval, and 95.32 on Tau2-Bench Telecom for the `high` setting.
|
||||
- **Reasoning Effort**: The model card describes `high` and `xhigh` reasoning-effort modes. In SGLang's OpenAI-compatible chat API, use top-level `reasoning_effort: "high"` for production agent workflows. To request the model-card `xhigh` prompt path, pass it through `chat_template_kwargs.reasoning_effort`.
|
||||
- **Hybrid Attention**: Uses the Bailing hybrid stack with MLA plus Lightning linear attention kernels in SGLang.
|
||||
- **Context Length**: Native 128K in the released config. Configure YaRN separately if you need a 256K deployment.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (E4M3 compressed-tensors)**: [inclusionAI/Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ring-2.6-1T requires recent SGLang builds with Bailing hybrid model support. Start with the latest SGLang Docker image when validating this cookbook:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
Use the selector below to generate a single-node command for the tested hardware targets.
|
||||
|
||||
import { Ring261TDeployment } from '/src/snippets/autoregressive/ring-26-1t-deployment.jsx'
|
||||
|
||||
<Ring261TDeployment />
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- `--trust-remote-code` is required for the model's custom Bailing hybrid implementation.
|
||||
- Use `--tp-size 4` on a single 4-GPU GB300 node.
|
||||
- Use `--tp-size 8` on a single 8-GPU B200 node.
|
||||
- Use `--tp-size 8` on a single 8-GPU H200 node.
|
||||
- Use `--mem-fraction-static 0.95` on GB300 x4. The model uses about 238.5GB/GPU after loading, so lower values can fail during KV-pool initialization.
|
||||
- Use `--mem-fraction-static 0.8` on B200 x8.
|
||||
- Use `--mem-fraction-static 0.95` on H200 x8.
|
||||
- `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` is recommended because the model has 175 large safetensors shards.
|
||||
- Keep `--tool-call-parser glm` enabled by default for OpenAI-compatible tool calls. Ring's template emits XML `<arg_key>/<arg_value>` tool calls, which the `qwen` parser does not convert into `message.tool_calls`.
|
||||
- Keep `--reasoning-parser deepseek-r1` enabled by default so `<think>...</think>` content is split into `message.reasoning_content`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For example, launch the server on a single 4-GPU GB300 node:
|
||||
|
||||
```bash Command
|
||||
export PORT=30000
|
||||
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.6-1T \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--mem-fraction-static 0.95 \
|
||||
--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' \
|
||||
--tool-call-parser glm \
|
||||
--reasoning-parser deepseek-r1
|
||||
```
|
||||
|
||||
Send a basic chat request:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
"max_tokens": 128
|
||||
}'
|
||||
```
|
||||
|
||||
### 4.2 Reasoning Effort
|
||||
|
||||
Ring-2.6-1T exposes two reasoning-effort levels in the model card: `high` and `xhigh`. In SGLang's OpenAI-compatible chat API, start with top-level `reasoning_effort: "high"` for agent and production workflows:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}],
|
||||
"reasoning_effort": "high",
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
For the model-card `xhigh` path, pass the template value explicitly:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}],
|
||||
"chat_template_kwargs": {"reasoning_effort": "xhigh"},
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
With the default deployment command, thinking text is separated into `message.reasoning_content` when the model emits `<think>...</think>` blocks.
|
||||
|
||||
### 4.3 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto",
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request).
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x), NVIDIA H200 GPU (8x), and NVIDIA GB300 GPU (4x)
|
||||
- Model: `inclusionAI/Ring-2.6-1T`
|
||||
- Docker image: `lmsysorg/sglang:latest`
|
||||
- SGLang version tested: `0.5.11`
|
||||
- Tensor Parallelism: 8 on B200 x8 and H200 x8, 4 on GB300 x4
|
||||
|
||||
Use the deployment command from [Section 3](#3-model-deployment), then confirm that the server is healthy before running benchmarks:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/health
|
||||
curl -s http://localhost:${PORT}/v1/models
|
||||
```
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Test Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port ${PORT} \
|
||||
--model inclusionAI/Ring-2.6-1T \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results (B200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 207.18
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 29.45
|
||||
Output token throughput (tok/s): 20.37
|
||||
Total token throughput (tok/s): 49.82
|
||||
Mean E2E Latency (ms): 20715.16
|
||||
Mean TTFT (ms): 187.86
|
||||
Mean TPOT (ms): 44.65
|
||||
Mean ITL (ms): 48.76
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (GB300 x4):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 62.21
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 98.07
|
||||
Output token throughput (tok/s): 67.83
|
||||
Total token throughput (tok/s): 165.91
|
||||
Mean E2E Latency (ms): 6218.57
|
||||
Mean TTFT (ms): 233.04
|
||||
Mean TPOT (ms): 14.21
|
||||
Mean ITL (ms): 14.22
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (H200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 57.10
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.18
|
||||
Input token throughput (tok/s): 106.85
|
||||
Output token throughput (tok/s): 73.91
|
||||
Total token throughput (tok/s): 180.76
|
||||
Mean E2E Latency (ms): 5707.72
|
||||
Mean TTFT (ms): 163.35
|
||||
Mean TPOT (ms): 13.17
|
||||
Mean ITL (ms): 13.17
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Test Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port ${PORT} \
|
||||
--model inclusionAI/Ring-2.6-1T \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 100 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results (B200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 46.30
|
||||
Total input tokens: 50561
|
||||
Total generated tokens: 52444
|
||||
Request throughput (req/s): 2.16
|
||||
Input token throughput (tok/s): 1092.10
|
||||
Output token throughput (tok/s): 1132.77
|
||||
Total token throughput (tok/s): 2224.86
|
||||
Mean E2E Latency (ms): 27581.74
|
||||
Mean TTFT (ms): 1710.53
|
||||
Mean TPOT (ms): 51.27
|
||||
Mean ITL (ms): 49.43
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (GB300 x4):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 55.80
|
||||
Total input tokens: 50561
|
||||
Total generated tokens: 52444
|
||||
Request throughput (req/s): 1.79
|
||||
Input token throughput (tok/s): 906.10
|
||||
Output token throughput (tok/s): 939.84
|
||||
Total token throughput (tok/s): 1845.94
|
||||
Mean E2E Latency (ms): 33736.85
|
||||
Mean TTFT (ms): 2156.40
|
||||
Mean TPOT (ms): 63.09
|
||||
Mean ITL (ms): 60.33
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (H200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 44.51
|
||||
Total input tokens: 50561
|
||||
Total generated tokens: 52444
|
||||
Request throughput (req/s): 2.25
|
||||
Input token throughput (tok/s): 1135.88
|
||||
Output token throughput (tok/s): 1178.18
|
||||
Total token throughput (tok/s): 2314.06
|
||||
Mean E2E Latency (ms): 27177.14
|
||||
Mean TTFT (ms): 2173.08
|
||||
Mean TPOT (ms): 51.11
|
||||
Mean ITL (ms): 47.77
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.test.run_eval \
|
||||
--eval-name gsm8k \
|
||||
--host 127.0.0.1 \
|
||||
--port ${PORT} \
|
||||
--model auto \
|
||||
--num-examples 200 \
|
||||
--num-threads 64 \
|
||||
--max-tokens 2048 \
|
||||
--reasoning-effort high
|
||||
```
|
||||
|
||||
- Test Results (B200 x8):
|
||||
|
||||
```text Output
|
||||
Total latency: 100.378 s
|
||||
Score: 0.990
|
||||
Output throughput: 627.401 token/s
|
||||
```
|
||||
|
||||
- Test Results (GB300 x4):
|
||||
|
||||
```text Output
|
||||
Total latency: 98.386 s
|
||||
Score: 0.990
|
||||
Output throughput: 621.469 token/s
|
||||
```
|
||||
|
||||
- Test Results (H200 x8):
|
||||
|
||||
```text Output
|
||||
Total latency: 76.849 s
|
||||
Score: 0.990
|
||||
Output throughput: 793.125 token/s
|
||||
```
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Intern-S1
|
||||
metatags:
|
||||
description: "Deploy Intern-S1 with SGLang - community contribution guide for InternLM's Intern-S1 model deployment."
|
||||
---
|
||||
|
||||
import { InternS1Deployment } from '/src/snippets/autoregressive/intern-s1-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
Intern-S1 includes the large **Intern-S1** MoE model and the smaller **Intern-S1-mini** dense model. The command generator below covers BF16 and FP8 serving on NVIDIA H100/H200/B200/B300 platforms.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install from source:
|
||||
|
||||
```bash Command
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
<InternS1Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- FP8 checkpoints use the matching BF16 checkpoint as tokenizer path.
|
||||
- B300 deployments use `--attention-backend flashinfer`.
|
||||
- Enable `--reasoning-parser interns1` and `--tool-call-parser interns1` when your workload needs structured reasoning or tool-call parsing.
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
title: Intern-S2-Preview
|
||||
metatags:
|
||||
description: "Deploy Intern-S2-Preview with SGLang"
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Intern-S2-Preview** is an efficient 35B scientific multimodal foundation model. Beyond conventional parameter and data scaling, Intern-S2-Preview explores task scaling: increasing the difficulty, diversity, and coverage of scientific tasks to further unlock model capabilities.
|
||||
|
||||
**Resources:**
|
||||
|
||||
- HuggingFace: [internLM/Intern-S2-Preview](https://huggingface.co/internLM/Intern-S2-Preview)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
Install SGLang from source or use an NVIDIA Docker image:
|
||||
|
||||
```bash Command
|
||||
# Install from source
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
|
||||
# Or use Docker for NVIDIA GPUs
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
For how to actually launch a docker image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the [command generator](#3-model-deployment) below produces):
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware and parser configuration.
|
||||
|
||||
import { InternS2PreviewDeployment } from "/src/snippets/autoregressive/intern-s2-preview-deployment.jsx";
|
||||
|
||||
<InternS2PreviewDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Use `tp>=2` for the NVIDIA deployment commands.
|
||||
- Use `--reasoning-parser qwen3` to separate reasoning content from final content in streaming responses.
|
||||
- Use `--tool-call-parser qwen3_coder` when serving tool-calling workloads.
|
||||
- Add `--mamba-radix-cache-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP.
|
||||
- If weight loading is slow, add `--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 64}'`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, see:
|
||||
|
||||
- [Basic API Usage](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Vision Input
|
||||
|
||||
Intern-S2-Preview supports image inputs. Here is an example with an image:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="internLM/Intern-S2-Preview",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image in detail.",
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.2.2 Reasoning Parser
|
||||
|
||||
Enable streaming to read reasoning content separately from the final answer:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="internLM/Intern-S2-Preview",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
Serve with `--tool-call-parser qwen3_coder` enabled, then send OpenAI-compatible tool requests:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="internLM/Intern-S2-Preview",
|
||||
messages=[{"role": "user", "content": "What is the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
max_tokens=1024,
|
||||
)
|
||||
|
||||
print(response.choices[0].message)
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
title: InternVL3.5
|
||||
metatags:
|
||||
description: "Deploy InternVL3.5 vision-language model with SGLang - community contribution guide for OpenGVLab's multimodal model."
|
||||
---
|
||||
|
||||
|
||||
## 📝 Community Contribution Welcome
|
||||
|
||||
This guide is currently under development. We welcome community contributions!
|
||||
|
||||
If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation.
|
||||
|
||||
## 🚀 How to Contribute
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
git checkout -b add-internvl3-5-guide
|
||||
# Edit this file and submit a PR
|
||||
```
|
||||
|
||||
## 📚 Reference
|
||||
|
||||
- [GLM-4.6V](../GLM/GLM-4.6V)
|
||||
|
||||
---
|
||||
|
||||
**Let's build this together!** 🌟
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Jina-reranker-m0
|
||||
metatags:
|
||||
description: "Deploy Jina-reranker-m0 with SGLang - community contribution guide for Jina AI's reranker model deployment."
|
||||
---
|
||||
|
||||
## 📝 Community Contribution Welcome
|
||||
|
||||
This guide is currently under development. We welcome community contributions!
|
||||
|
||||
If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation.
|
||||
|
||||
## 🚀 How to Contribute
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
git checkout -b add-jina-reranker-m0-guide
|
||||
# Edit this file and submit a PR
|
||||
```
|
||||
|
||||
## 📚 Reference
|
||||
|
||||
- [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md)
|
||||
|
||||
---
|
||||
|
||||
**Let's build this together!** 🌟
|
||||
@@ -0,0 +1,359 @@
|
||||
---
|
||||
title: LFM2.5
|
||||
description: "Deploy Liquid AI's LFM2.5 with SGLang — hybrid gated short conv + GQA models from 350M to the 8B-A1B MoE, plus LFM2.5-VL vision, with reasoning and Pythonic tool calling."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
<Note>
|
||||
LFM2.5 support — the dense / MoE / VL model classes and the `lfm2` tool-call parser — ships on SGLang `main`. If your installed release predates it, install from source or use the Docker dev image.
|
||||
</Note>
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
LFM2.5 support ships in the pinned SGLang dev image:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-cu13
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:dev-cu13 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Every LFM2.5 model runs on a **single GPU (TP=1)** — pick your hardware + model variant to generate the launch command. One recipe covers all operating points per variant; the commands differ only by the parsers a model needs and, on Blackwell, the attention backend. The `lfm2` tool-call parser and each reasoning model's `--reasoning-parser` are already part of the verified command.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/LiquidAI/lfm2.5.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/LiquidAI/lfm2.5-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the dev image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
For LFM2.5 the exposed knob is the **TP override** (every variant is verified at TP=1; TP=2 is available for experimentation on the larger checkpoints). The reasoning and tool-call parsers are not playground toggles here — they are variant-intrinsic and already baked into each verified command.
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
LFM2.5 is [Liquid AI](https://www.liquid.ai/)'s family of hybrid models for on-device deployment, released under the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). It builds on the LFM2 architecture with extended pre-training — 10T → 28T tokens for the dense models, 12T → 38T for the 8B-A1B MoE — and large-scale reinforcement learning.
|
||||
|
||||
The backbone interleaves **gated short convolution blocks** with a small minority of **grouped query attention (GQA) blocks**. Each convolution block applies input-dependent multiplicative gating around a depthwise short convolution, giving fast local mixing at low compute and memory cost. The GQA blocks handle global context and long-range retrieval.
|
||||
|
||||
This minimal hybrid layout was selected by a hardware-in-the-loop architecture search under edge latency and memory budgets. On CPUs it delivers up to 2× faster prefill and decode than similarly sized models (see the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404)).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid gated short conv + GQA layout**: the 1.2B / 350M dense models are 16 layers (10 conv + 6 GQA); the 8B-A1B MoE is 24 layers (18 conv + 6 GQA). With only 6 attention layers per model, the KV cache stays small even at long context.
|
||||
- **Block details**: depthwise convolutions with kernel size 3; GQA with 8 KV groups and head size 64, plus RoPE and QK-Norm; pre-norm RMSNorm and SwiGLU MLPs throughout.
|
||||
- **Sparse MoE (8B-A1B)**: 8.3B total / 1.5B active parameters. Every layer except the first two replaces its dense MLP with a 32-expert MoE block; each token is routed to the top-4 SwiGLU experts by a normalized sigmoid router with adaptive bias load balancing.
|
||||
- **New in 2.5 (8B-A1B)**: the blocks are unchanged from LFM2-8B-A1B, but the context window grows from 32K to 128K (a RoPE base-θ increase plus long-context midtraining) and the vocabulary doubles from 65,536 to 128,000 tokens for more efficient non-Latin tokenization.
|
||||
- **Pythonic tool calling**: function calls are emitted as a Python list between `<|tool_call_start|>` and `<|tool_call_end|>` tokens. The `lfm2` tool-call parser surfaces these as standard `message.tool_calls`.
|
||||
- **Reasoning variants**: the 8B-A1B and 1.2B-Thinking checkpoints are reasoning-only models that always emit an explicit `<think>...</think>` chain-of-thought before the answer. The MoE's 1.5B active parameters keep those reasoning tokens cheap.
|
||||
- **Multilingual**: every model except the JP checkpoints covers at least English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish (some variants add more). The dedicated JP chat checkpoints focus on Japanese (Japanese + English only).
|
||||
- **Vision**: LFM2.5-VL-1.6B pairs the 1.2B language backbone with a SigLIP2 So400M NaFlex encoder for OCR, document understanding, and multilingual vision. LFM2.5-VL-450M pairs the 350M backbone with a SigLIP2 Base-86M encoder for captioning and object detection at edge sizes; bounding-box grounding and function calling are new in the 2.5 release.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "26%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "44%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Context</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Role</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-8B-A1B">LFM2.5-8B-A1B</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8.3B total / 1.5B active (MoE)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>128K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning-tuned, agentic / tool use</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct">LFM2.5-1.2B-Instruct</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>General instruct, RAG, data extraction</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking">LFM2.5-1.2B-Thinking</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning (always-on chain-of-thought)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-350M">LFM2.5-350M</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>350M (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Compact instruct, structured output</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-230M">LFM2.5-230M</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>230M (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most compact; data extraction, structured output</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606">LFM2.5-1.2B-JP-202606</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Japanese chat (latest)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP">LFM2.5-1.2B-JP</a></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Japanese chat (original)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B">LFM2.5-VL-1.6B</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.2B LM + SigLIP2 400M</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Vision-language (OCR, docs, multi-image)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-VL-450M">LFM2.5-VL-450M</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>350M LM + SigLIP2 86M</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Compact vision-language (captioning, object detection)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base">LFM2.5-1.2B-Base</a></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Pre-trained base (no post-training)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The Deploy panel above covers the eight serving variants; **LFM2.5-1.2B-JP** (original — launch without `--tool-call-parser`) and the **Base** repos (pre-trained only, no post-training — see [§3.5](#3-5-base-checkpoints)) launch the same way with the model path swapped.
|
||||
|
||||
**Choosing a variant:**
|
||||
|
||||
- **8B-A1B** — flagship for agentic and tool-calling workloads; the only 128K-context option.
|
||||
- **1.2B-Thinking** — reasoning-heavy tasks: math, tool use, programming.
|
||||
- **1.2B-Instruct** — the recommended pick for chat and creative writing.
|
||||
- **350M** — tool use, data extraction, and structured output; not recommended for math, code, or creative writing.
|
||||
- **230M** — the most compact checkpoint; same use as the 350M, not for math, code, or creative writing.
|
||||
|
||||
**License:** [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE).
|
||||
|
||||
**Resources:** [LFM2.5 announcement](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai), [LFM2.5-8B-A1B blog](https://www.liquid.ai/blog/lfm2-5-8b-a1b), [LFM docs](https://docs.liquid.ai/lfm/getting-started/welcome), [LFM2 Technical Report (arXiv:2511.23404)](https://arxiv.org/abs/2511.23404).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Reasoning parser**: LFM2.5 reasoning models wrap their chain-of-thought in `<think>...</think>` tags. The command generator passes `--reasoning-parser qwen3` for **8B-A1B** (it emits an explicit opening `<think>`) and `--reasoning-parser qwen3-thinking` for **1.2B-Thinking** (always-on reasoning). This splits the thinking process into `reasoning_content`; without it the chain-of-thought stays inline in `content`.
|
||||
- **Tool calling**: `--tool-call-parser lfm2` surfaces LFM2.5's Pythonic `<|tool_call_start|>[...]<|tool_call_end|>` calls as standard `message.tool_calls`. The original **1.2B-JP** does not expose tool calling; **Base** has no post-training (see [§3.5](#3-5-base-checkpoints)).
|
||||
- **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for the dense text models. The **8B-A1B** uses a mamba-style state cache that runs on a page-size-1 backend, so the generator picks `--attention-backend flashinfer` for it. The **VL** language model also uses that state cache and offers two backends: `--attention-backend flashinfer` (keeps prefix/radix caching — what the generator emits), or `--attention-backend trtllm_mha --disable-radix-cache` to run the language model on Blackwell `trtllm_mha` attention (`--disable-radix-cache` lifts the page-size-1 requirement, at the cost of prefix caching). Pair either with `--mm-attention-backend fa4` for the vision tower.
|
||||
- **VL vision tower (`--mm-attention-backend`)**: on sm100 the `trtllm_mha` default is fastest for text but applies *causal* attention to image tokens. For the VL model, pass `--mm-attention-backend fa4` on B200/B300 (or `fa3` on H100/H200) to restore bidirectional image-token attention and full vision quality.
|
||||
- **VL multimodal feature transport**: the generator launches the VL models with `SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1`. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected.
|
||||
- **VL-450M memory headroom (`--mem-fraction-static 0.8`)**: with the default memory fraction, the 450M's small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps `--mem-fraction-static 0.8` for VL-450M; the pool is still far larger than this model ever needs.
|
||||
- **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-radix-cache-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`.
|
||||
- **Hardware requirements**: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant.
|
||||
|
||||
**Recommended sampling parameters** — pass these explicitly on every request. Some LFM2.5 checkpoints do not ship sampling defaults in `generation_config.json`, so the server will not apply them for you. `top_k`, `min_p`, and `repetition_penalty` are not standard OpenAI `chat.completions` fields — pass them through **`extra_body`** and SGLang forwards them to its sampler. Do not set `max_tokens` unless you intend to cap output, as it can truncate a response (or a reasoning model's chain-of-thought) mid-stream.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>temperature</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>extra_body (sampler)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-8B-A1B</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.2</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 80, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-Instruct</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-Thinking</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.05</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-350M</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-230M</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-JP-202606</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-JP</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.3</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-VL-1.6B (text)</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-VL-450M (text)</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-Base</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.3</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Basic Usage
|
||||
|
||||
A single client with the recommended sampling presets applied per model (the examples in the following sections reuse this `chat` helper):
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
# Non-OpenAI fields (top_k / min_p / repetition_penalty) ride in extra_body.
|
||||
SAMPLING = {
|
||||
"LiquidAI/LFM2.5-8B-A1B": dict(temperature=0.2, extra_body={"top_k": 80, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-1.2B-Instruct": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-1.2B-Thinking": dict(temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-350M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-230M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-1.2B-JP-202606": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-VL-1.6B": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-VL-450M": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
|
||||
}
|
||||
|
||||
def chat(model, messages, **overrides):
|
||||
cfg = SAMPLING[model]
|
||||
body = cfg["extra_body"] | overrides.pop("extra_body", {})
|
||||
return client.chat.completions.create(
|
||||
model=model, messages=messages,
|
||||
temperature=cfg["temperature"], extra_body=body, **overrides,
|
||||
)
|
||||
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}],
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
The 8B-A1B and 1.2B-Thinking checkpoints emit chain-of-thought as a built-in behavior. The Deploy panel launches them with the matching `--reasoning-parser`, which separates the thinking process into `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-8B-A1B",
|
||||
[{"role": "user", "content": "If a train travels 60 km/h for 2.5 hours, how far does it go?"}],
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", msg.reasoning_content)
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
LFM2.5 writes Pythonic tool calls. With `--tool-call-parser lfm2` (already part of the launch command) they are surfaced as standard `message.tool_calls`:
|
||||
|
||||
```python Example
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"location": {"type": "string"}},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}],
|
||||
)
|
||||
for call in resp.choices[0].message.tool_calls or []:
|
||||
print(call.function.name, call.function.arguments)
|
||||
```
|
||||
|
||||
Tool calling is supported on 8B-A1B, 1.2B-Thinking, 1.2B-Instruct, 350M, 230M, 1.2B-JP-202606, VL-1.6B, and VL-450M. For the **VL** models it is text-turn-only — do not combine an image and tools in the same turn.
|
||||
|
||||
### 3.4 Vision Input
|
||||
|
||||
The VL models (VL-1.6B and VL-450M) accept images via standard OpenAI multimodal content blocks. Base64 data URIs (`data:image/jpeg;base64,...`) work in place of a URL:
|
||||
|
||||
```python Example
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-VL-1.6B",
|
||||
[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {
|
||||
"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
|
||||
{"type": "text", "text": "What is in this image?"},
|
||||
],
|
||||
}],
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 3.5 Base Checkpoints
|
||||
|
||||
Each size ships a pre-trained Base repo — [LFM2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base), [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base), [LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base), and [LFM2.5-8B-A1B-Base](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-Base) — intended for fine-tuning and continued pre-training.
|
||||
|
||||
The repos ship a ChatML-style chat template, so `chat.completions` requests format normally. The checkpoints have no post-training, though — don't expect instruction following. For raw text continuation:
|
||||
|
||||
```python Example
|
||||
comp = client.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-Base",
|
||||
prompt="The capital of France is",
|
||||
temperature=0.3,
|
||||
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(comp.choices[0].text)
|
||||
```
|
||||
@@ -0,0 +1,650 @@
|
||||
---
|
||||
title: Llama-3.1
|
||||
metatags:
|
||||
description: "Deploy Llama 3.1 (8B/70B/405B) with SGLang - 128K context, tool use, multilingual support, and speculative decoding optimization."
|
||||
---
|
||||
## 1. Model Introduction
|
||||
|
||||
Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time.
|
||||
|
||||
These models bring open intelligence to all, with several new features and improvements:
|
||||
|
||||
- **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities.
|
||||
- **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning.
|
||||
- **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools.
|
||||
- **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai.
|
||||
|
||||
For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models.
|
||||
|
||||
import { Llama31Deployment } from "/src/snippets/autoregressive/llama31-deployment.jsx";
|
||||
|
||||
<Llama31Deployment />
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Speculative Decoding (NVIDIA GPUs):**
|
||||
|
||||
- Using Speculative Decoding for latency-sensitive scenarios:
|
||||
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
|
||||
- `--speculative-num-steps 3`: Number of speculative verification rounds
|
||||
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
|
||||
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
|
||||
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B).
|
||||
|
||||
**AMD GPU Deployment:**
|
||||
|
||||
- **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB)
|
||||
- **Verified TP Configurations**:
|
||||
- MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
|
||||
- MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
|
||||
- **FP8 Model Variants**:
|
||||
- 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8`
|
||||
- 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV`
|
||||
- **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models
|
||||
|
||||
**Xeon CPU Deployment:**
|
||||
|
||||
- Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
SGLang exposes an OpenAI-compatible endpoint. First, start the server
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="Meta-Llama/Llama-3.1-405B-Instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful coding assistant."},
|
||||
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
|
||||
],
|
||||
temperature=0.2,
|
||||
max_tokens=512,
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
**Exponential Backoff Retry Function in Python**
|
||||
=====================================================
|
||||
|
||||
Below is a Python function that uses the `requests` library to retry a request with exponential backoff.
|
||||
|
||||
```python
|
||||
import requests
|
||||
import time
|
||||
import random
|
||||
|
||||
def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
|
||||
"""
|
||||
Retry a request with exponential backoff.
|
||||
|
||||
Args:
|
||||
url (str): The URL to make the request to.
|
||||
method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
|
||||
retries (int): The number of retries to attempt. Defaults to 3.
|
||||
backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
|
||||
max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.
|
||||
|
||||
Returns:
|
||||
The response object from the successful request.
|
||||
"""
|
||||
|
||||
delay = 1
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
response = requests.request(method, url)
|
||||
response.raise_for_status() # Raise an exception for HTTP errors
|
||||
return response
|
||||
except requests.RequestException as e:
|
||||
if attempt < retries:
|
||||
# Calculate the delay for this retry
|
||||
delay = min(delay * backoff_factor, max_delay)
|
||||
# Add a random jitter to the delay to prevent thundering herd problem
|
||||
delay += random.uniform(0, delay * 0.1)
|
||||
# Wait for the calculated delay before retrying
|
||||
time.sleep(delay)
|
||||
else:
|
||||
# If all retries have failed, raise the exception
|
||||
raise e
|
||||
...
|
||||
````
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Tool Calling
|
||||
|
||||
Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
|
||||
--tool-call-parser llama3 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Python Example**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.1-405B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What's the weather like in Boston today?",
|
||||
}
|
||||
],
|
||||
temperature=0.7,
|
||||
stream=True,
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
|
||||
arguments = []
|
||||
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#openai-compatible-api)
|
||||
|
||||
**Output Example**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"city": "Boston", "unit": "fahrenheit"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results**
|
||||
After getting the tool call, you can execute the function:
|
||||
|
||||
```python Example
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather like in Boston today?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Boston", "unit": "fahrenheit"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Boston", "fahrenheit")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="Meta-Llama/Llama-3.1-405B-Instruct",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA A100 GPU (8x)
|
||||
- Model: Meta-Llama/Llama-3.1-70B
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 79.81
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4208
|
||||
Request throughput (req/s): 0.13
|
||||
Input token throughput (tok/s): 76.44
|
||||
Output token throughput (tok/s): 52.88
|
||||
Peak output token throughput (tok/s): 54.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 129.32
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7977.81
|
||||
Median E2E Latency (ms): 6373.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 131.61
|
||||
Median TTFT (ms): 131.77
|
||||
P99 TTFT (ms): 163.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.63
|
||||
Median TPOT (ms): 18.63
|
||||
P99 TPOT (ms): 18.65
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.64
|
||||
Median ITL (ms): 18.64
|
||||
P95 ITL (ms): 18.69
|
||||
P99 ITL (ms): 18.74
|
||||
Max ITL (ms): 21.95
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 79.47
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 38450
|
||||
Request throughput (req/s): 1.01
|
||||
Input token throughput (tok/s): 499.17
|
||||
Output token throughput (tok/s): 513.48
|
||||
Peak output token throughput (tok/s): 674.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1012.65
|
||||
Concurrency: 13.47
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13376.67
|
||||
Median E2E Latency (ms): 14130.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 264.84
|
||||
Median TTFT (ms): 147.02
|
||||
P99 TTFT (ms): 791.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 26.09
|
||||
Median TPOT (ms): 26.08
|
||||
P99 TPOT (ms): 34.65
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.76
|
||||
Median ITL (ms): 23.95
|
||||
P95 ITL (ms): 24.72
|
||||
P99 ITL (ms): 98.32
|
||||
Max ITL (ms): 478.92
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 131.64
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 243641
|
||||
Request throughput (req/s): 3.80
|
||||
Input token throughput (tok/s): 1897.87
|
||||
Output token throughput (tok/s): 1919.38
|
||||
Peak output token throughput (tok/s): 3100.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3817.25
|
||||
Concurrency: 89.70
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 23616.71
|
||||
Median E2E Latency (ms): 22770.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 245.98
|
||||
Median TTFT (ms): 184.22
|
||||
P99 TTFT (ms): 1251.67
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 47.19
|
||||
Median TPOT (ms): 48.67
|
||||
P99 TPOT (ms): 56.37
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 46.34
|
||||
Median ITL (ms): 33.46
|
||||
P95 ITL (ms): 108.61
|
||||
P99 ITL (ms): 166.11
|
||||
Max ITL (ms): 1107.09
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B\
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 83.25
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.12
|
||||
Input token throughput (tok/s): 503.77
|
||||
Output token throughput (tok/s): 50.69
|
||||
Peak output token throughput (tok/s): 54.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 554.46
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8322.45
|
||||
Median E2E Latency (ms): 6873.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 395.25
|
||||
Median TTFT (ms): 318.02
|
||||
P99 TTFT (ms): 850.80
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.80
|
||||
Median TPOT (ms): 18.81
|
||||
P99 TPOT (ms): 19.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.83
|
||||
Median ITL (ms): 18.81
|
||||
P95 ITL (ms): 19.06
|
||||
P99 ITL (ms): 19.08
|
||||
Max ITL (ms): 23.08
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 107.12
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41603
|
||||
Request throughput (req/s): 0.75
|
||||
Input token throughput (tok/s): 2800.81
|
||||
Output token throughput (tok/s): 389.00
|
||||
Peak output token throughput (tok/s): 624.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3189.81
|
||||
Concurrency: 14.18
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18988.30
|
||||
Median E2E Latency (ms): 20290.66
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 603.42
|
||||
Median TTFT (ms): 531.82
|
||||
P99 TTFT (ms): 2607.95
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 36.94
|
||||
Median TPOT (ms): 36.73
|
||||
P99 TPOT (ms): 79.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.36
|
||||
Median ITL (ms): 25.72
|
||||
P95 ITL (ms): 27.07
|
||||
P99 ITL (ms): 439.74
|
||||
Max ITL (ms): 2529.51
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 215.66
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169035
|
||||
Request throughput (req/s): 1.48
|
||||
Input token throughput (tok/s): 5906.92
|
||||
Output token throughput (tok/s): 788.27
|
||||
Peak output token throughput (tok/s): 1920.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 6695.19
|
||||
Concurrency: 60.01
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 40443.85
|
||||
Median E2E Latency (ms): 39813.12
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 633.32
|
||||
Median TTFT (ms): 616.38
|
||||
P99 TTFT (ms): 1912.97
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 74.95
|
||||
Median TPOT (ms): 82.85
|
||||
P99 TPOT (ms): 118.46
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 75.08
|
||||
Median ITL (ms): 34.12
|
||||
P95 ITL (ms): 261.18
|
||||
P99 ITL (ms): 828.12
|
||||
Max ITL (ms): 1970.03
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.830
|
||||
Invalid: 0.000
|
||||
Latency: 11.794 s
|
||||
Output throughput: 1406.961 token/s
|
||||
```
|
||||
@@ -0,0 +1,235 @@
|
||||
---
|
||||
title: Llama-3.3-70B
|
||||
metatags:
|
||||
description: "Deploy Llama-3.3-70B-Instruct with SGLang on AMD GPUs - 128K context, enhanced reasoning, tool calling, and multilingual support."
|
||||
---
|
||||
## 1. Model Introduction
|
||||
|
||||
[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
|
||||
- **Tool Calling**: Native support for function calling and tool use scenarios
|
||||
- **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
|
||||
- **Extended Context**: 128K token context window for processing long documents and complex tasks
|
||||
- **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X
|
||||
|
||||
**License:**
|
||||
Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs.
|
||||
|
||||
### 3.1 Interactive Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
|
||||
|
||||
import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
|
||||
|
||||
<Llama33Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**AMD GPU Deployment:**
|
||||
|
||||
- All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
|
||||
- **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
|
||||
- **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
|
||||
- **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput
|
||||
|
||||
**Xeon CPU Deployment:**
|
||||
|
||||
Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Tool Calling
|
||||
|
||||
Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.3-70B-Instruct \
|
||||
--tool-call-parser llama3 \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Tokyo?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
# Check for tool calls
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
tool_call = message.tool_calls[0]
|
||||
print(f"Function: {tool_call.function.name}")
|
||||
print(f"Arguments: {tool_call.function.arguments}")
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After executing the function, send the result back
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Build conversation with tool result
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Tokyo?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Tokyo", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Tokyo", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
|
||||
```
|
||||
|
||||
#### 4.2.2 Long Context Processing
|
||||
|
||||
Leverage the 128K context window for processing long documents:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Example with long document
|
||||
long_document = "..." * 10000 # Your long document here
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=1000
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## 5. Benchmarking
|
||||
|
||||
Use the SGLang benchmarking suite to test model performance with different workload patterns:
|
||||
|
||||
### 5.1 Basic Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 1000 \
|
||||
--random-input 1024 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
### 5.2 Adjusting Benchmark Parameters
|
||||
|
||||
**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:
|
||||
|
||||
- Short conversations: `--random-input 1024 --random-output 1024`
|
||||
- Long outputs: `--random-input 1024 --random-output 8192`
|
||||
- Long inputs: `--random-input 8192 --random-output 1024`
|
||||
|
||||
**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:
|
||||
|
||||
- Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
|
||||
- Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
|
||||
- High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
|
||||
- [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
|
||||
- [SGLang Documentation](/)
|
||||
- [AMD ROCm Documentation](https://rocm.docs.amd.com/)
|
||||
@@ -0,0 +1,572 @@
|
||||
---
|
||||
title: Llama 4
|
||||
metatags:
|
||||
description: "Deploy Llama 4 Scout and Maverick with SGLang - Meta's latest generation open-source LLMs with industry-leading performance."
|
||||
---
|
||||
|
||||
import { Llama4ScoutDeployment } from '/src/snippets/autoregressive/llama4-scout-deployment.jsx';
|
||||
import { Llama4MaverickDeployment } from '/src/snippets/autoregressive/llama4-maverick-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance.
|
||||
|
||||
SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
|
||||
|
||||
Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
The highly capable Llama 4 Maverick with 17B active parameters out of ~400B total, with 128 experts.
|
||||
The efficient Llama 4 Scout also has 17B active parameters out of ~109B total, using just 16 experts.
|
||||
Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi).
|
||||
|
||||
For more details, please refer to the official llama4 Repository:https://www.llama.com/models/llama-4/
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Llama4ScoutDeployment />
|
||||
|
||||
<Llama4MaverickDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **OOM Mitigation:** Reduce `--context-length` to avoid GPU out-of-memory. Recommended: Scout up to 1M on 8×H100, up to 2.5M on 8×H200; Maverick doesn't need context-length set on 8×H200. With hybrid KV cache enabled, Scout can reach 5M on 8×H100 and 10M on 8×H200.
|
||||
- **Attention Backend Auto-Selection:** SGLang automatically picks the optimal backend. Manual override with `--attention-backend`:
|
||||
- Blackwell (B200/GB200): `trtllm_mha`
|
||||
- Hopper (H100/H200): `fa3`
|
||||
- AMD GPUs: `aiter`
|
||||
- Intel XPU: `intel_xpu`
|
||||
- Other: `triton`
|
||||
- **Chat Template:** Add `--chat-template llama-4` for chat completion tasks.
|
||||
- **Multi-Modal:** Add `--enable-multimodal` to enable image input support.
|
||||
- **Hybrid KV Cache:** Set `--swa-full-tokens-ratio` to control the ratio of SWA (local attention) KV tokens to full-attention KV tokens (default: 0.8, range: 0–1).
|
||||
- **EAGLE Speculative Decoding:** Supported for Llama 4 Scout and Maverick via EAGLE3. Enable with the interactive command generator above.
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Llama4 \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 Launch the server
|
||||
|
||||
### Llama-4-Scout
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### Llama-4-Maverick
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--tp 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
#### 4.2.3 EAGLE Speculative Decoding
|
||||
|
||||
SGLang supports Llama 4 Maverick (400B) with [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding). Enable with the EAGLE3 algorithm and the SGLang EAGLE3 draft model:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
### 5.1 Speed Benchmark (Scout)
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300x GPU
|
||||
|
||||
Model: Llama-4-Scout
|
||||
|
||||
Tensor Parallelism: 8
|
||||
|
||||
sglang version: 0.5.9
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 74.62
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4211
|
||||
Request throughput (req/s): 0.14
|
||||
Input token throughput (tok/s): 82.88
|
||||
Output token throughput (tok/s): 57.42
|
||||
Peak output token throughput (tok/s): 146.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 140.20
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7459.48
|
||||
Median E2E Latency (ms): 4489.77
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4246.98
|
||||
Median TTFT (ms): 68.57
|
||||
P99 TTFT (ms): 48091.05
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.49
|
||||
Median TPOT (ms): 7.40
|
||||
P99 TPOT (ms): 7.40
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.49
|
||||
Median ITL (ms): 7.49
|
||||
P95 ITL (ms): 7.47
|
||||
P99 ITL (ms): 7.52
|
||||
Max ITL (ms): 10.44
|
||||
==================================================
|
||||
```
|
||||
### 5.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 45.41
|
||||
Total input tokens: 49668
|
||||
Total input text tokens: 49668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40516
|
||||
Request throughput (req/s): 2.26
|
||||
Input token throughput (tok/s): 1120.46
|
||||
Output token throughput (tok/s): 1152.47
|
||||
Peak output token throughput (tok/s): 1520.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2272.84
|
||||
Concurrency: 14.76
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6089.22
|
||||
Median E2E Latency (ms): 6568.80
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 124.44
|
||||
Median TTFT (ms): 87.42
|
||||
P99 TTFT (ms): 268.72
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.88
|
||||
Median TPOT (ms): 12.00
|
||||
P99 TPOT (ms): 15.49
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.72
|
||||
Median ITL (ms): 10.54
|
||||
P95 ITL (ms): 11.22
|
||||
P99 ITL (ms): 67.88
|
||||
Max ITL (ms): 74.05
|
||||
==================================================
|
||||
```
|
||||
### 5.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 85.84
|
||||
Total input tokens: 249841
|
||||
Total input text tokens: 249841
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250498
|
||||
Request throughput (req/s): 5.84
|
||||
Input token throughput (tok/s): 2910.84
|
||||
Output token throughput (tok/s): 2944.82
|
||||
Peak output token throughput (tok/s): 4100.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5854.65
|
||||
Concurrency: 92.24
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 15844.00
|
||||
Median E2E Latency (ms): 15262.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 204.46
|
||||
Median TTFT (ms): 129.96
|
||||
P99 TTFT (ms): 528.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 41.56
|
||||
Median TPOT (ms): 42.90
|
||||
P99 TPOT (ms): 47.48
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 40.99
|
||||
Median ITL (ms): 24.46
|
||||
P95 ITL (ms): 84.46
|
||||
P99 ITL (ms): 87.64
|
||||
Max ITL (ms): 226.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark (Maverick)
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300x GPU
|
||||
|
||||
Model: Llama-4-Maverick
|
||||
|
||||
Tensor Parallelism: 8
|
||||
|
||||
sglang version: 0.5.9
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.2.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 68.08
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4202
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 89.62
|
||||
Output token throughput (tok/s): 61.99
|
||||
Peak output token throughput (tok/s): 168.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 151.61
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6805.62
|
||||
Median E2E Latency (ms): 2733.91
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4296.56
|
||||
Median TTFT (ms): 57.45
|
||||
P99 TTFT (ms): 38633.95
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.95
|
||||
Median TPOT (ms): 5.96
|
||||
P99 TPOT (ms): 5.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.96
|
||||
Median ITL (ms): 5.96
|
||||
P95 ITL (ms): 6.02
|
||||
P99 ITL (ms): 6.08
|
||||
Max ITL (ms): 7.02
|
||||
==================================================
|
||||
```
|
||||
### 5.2.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 30.72
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40923
|
||||
Request throughput (req/s): 2.60
|
||||
Input token throughput (tok/s): 1291.39
|
||||
Output token throughput (tok/s): 1328.41
|
||||
Peak output token throughput (tok/s): 1760.00
|
||||
Peak concurrent requests: 22
|
||||
Total token throughput (tok/s): 2619.80
|
||||
Concurrency: 13.92
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5345.15
|
||||
Median E2E Latency (ms): 5679.73
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 259.30
|
||||
Median TTFT (ms): 72.60
|
||||
P99 TTFT (ms): 1063.45
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.53
|
||||
Median TPOT (ms): 10.22
|
||||
P99 TPOT (ms): 20.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.99
|
||||
Median ITL (ms): 9.10
|
||||
P95 ITL (ms): 9.87
|
||||
P99 ITL (ms): 55.62
|
||||
Max ITL (ms): 868.54
|
||||
==================================================
|
||||
```
|
||||
### 5.2.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 90.95
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 251625
|
||||
Request throughput (req/s): 5.50
|
||||
Input token throughput (tok/s): 2746.77
|
||||
Output token throughput (tok/s): 2777.90
|
||||
Peak output token throughput (tok/s): 3700.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 5524.67
|
||||
Concurrency: 93.04
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16924.17
|
||||
Median E2E Latency (ms): 16294.85
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 188.19
|
||||
Median TTFT (ms): 128.96
|
||||
P99 TTFT (ms): 534.81
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.63
|
||||
Median TPOT (ms): 35.37
|
||||
P99 TPOT (ms): 38.26
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.19
|
||||
Median ITL (ms): 27.66
|
||||
P95 ITL (ms): 76.91
|
||||
P99 ITL (ms): 78.82
|
||||
Max ITL (ms): 268.17
|
||||
==================================================
|
||||
```
|
||||
### 5.3 Accuracy Benchmark
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
- Llama-4-Scout-17B-16E-Instruct
|
||||
```text Output
|
||||
Accuracy: 0.945
|
||||
Invalid: 0.000
|
||||
Latency: 12.731 s
|
||||
Output throughput: 1595.418 token/s
|
||||
```
|
||||
- Llama-4-Maverick-17B-128E-Instruct
|
||||
```text Output
|
||||
Accuracy: 0.895
|
||||
Invalid: 0.000
|
||||
Latency: 9.739 s
|
||||
Output throughput: 2405.505 token/s
|
||||
```
|
||||
|
||||
#### 5.3.2 MMLU Pro with lm-eval
|
||||
|
||||
Accuracy on MMLU Pro matches [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) on 8×H100 (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)):
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Official</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Scout-17B-16E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>74.3</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>75.2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Maverick-17B-128E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>80.5</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>80.7</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Scout:**
|
||||
|
||||
```bash Command
|
||||
# Start the server
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
|
||||
# Run lm_eval
|
||||
lm_eval --model local-chat-completions \
|
||||
--model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
|
||||
--tasks mmlu_pro \
|
||||
--batch_size 128 \
|
||||
--apply_chat_template \
|
||||
--num_fewshot 0
|
||||
```
|
||||
|
||||
**Maverick:**
|
||||
|
||||
```bash Command
|
||||
# Start the server
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
|
||||
# Run lm_eval
|
||||
lm_eval --model local-chat-completions \
|
||||
--model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
|
||||
--tasks mmlu_pro \
|
||||
--batch_size 128 \
|
||||
--apply_chat_template \
|
||||
--num_fewshot 0
|
||||
```
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
title: LongCat-2.0
|
||||
description: "Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
|
||||
# Choose the nightly wheel index for your CUDA runtime.
|
||||
SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130 # B300 / CUDA 13
|
||||
# SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129 # CUDA 12.9
|
||||
|
||||
uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]"
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# Choose the rolling nightly image for your hardware.
|
||||
SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13 # B300 / CUDA 13
|
||||
# SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev # Other supported hardware
|
||||
|
||||
docker pull "${SGLANG_DOCKER_IMAGE}"
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy:
|
||||
|
||||
- **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/meituan-longcat/longcat-2.0.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/meituan-longcat/longcat-2.0-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the LongCat sparse-attention indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on LongCat-2.0.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles.
|
||||
</Note>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Serving precision</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/meituan-longcat/LongCat-2.0-FP8">LongCat-2.0-FP8</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>Sparse MoE · LongCat Sparse Attention · n-gram embedding</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 weights, BF16 KV cache</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint.
|
||||
- **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you.
|
||||
- **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path.
|
||||
- **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster.
|
||||
- **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time.
|
||||
- **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout.
|
||||
- **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Test the deployment
|
||||
|
||||
<Accordion title="Chat completion example (cURL)">
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "meituan-longcat/LongCat-2.0-FP8",
|
||||
"messages": [
|
||||
{"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."}
|
||||
],
|
||||
"max_tokens": 32,
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Expected output">
|
||||
|
||||
```text Output
|
||||
15
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="OpenAI-compatible client (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meituan-longcat/LongCat-2.0-FP8",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.",
|
||||
}
|
||||
],
|
||||
max_tokens=32,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example output">
|
||||
|
||||
```text Output
|
||||
15
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
## 4. Validation
|
||||
|
||||
The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Evaluation</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Examples</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Accuracy</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>GSM8K</td>
|
||||
<td style={{padding: "9px 12px"}}>200</td>
|
||||
<td style={{padding: "9px 12px"}}>98.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>GSM8K</td>
|
||||
<td style={{padding: "9px 12px"}}>1314</td>
|
||||
<td style={{padding: "9px 12px"}}>95.8904109589041%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,730 @@
|
||||
---
|
||||
title: MiniMax-M2.7
|
||||
metatags:
|
||||
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs — model self-evolution, professional software engineering, and native agent teams."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search.
|
||||
|
||||
Key highlights:
|
||||
|
||||
- **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**.
|
||||
- **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems.
|
||||
- **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier).
|
||||
- **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making.
|
||||
|
||||
For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en).
|
||||
|
||||
**License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA A100 / H100 / H200 / B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA B300 / GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-cu130`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities.
|
||||
|
||||
import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-deployment.jsx'
|
||||
|
||||
<MiniMaxM27Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Key Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tool-call-parser`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tool call parser for function calling support</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-m2`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning parser for thinking mode</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-append-think`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for MiniMax model loading</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.85`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tp`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor parallelism size</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2` / `4` / `8` depending on hardware</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--ep`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Expert parallelism size</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--kv-cache-dtype`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>KV cache data type (AMD only)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`fp8_e4m3`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention backend (AMD only)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Hardware Requirements: NVIDIA**
|
||||
|
||||
- **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
|
||||
- **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8
|
||||
|
||||
**Hardware Requirements: NVIDIA GB300**
|
||||
|
||||
- **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2
|
||||
- **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput
|
||||
|
||||
**Hardware Requirements: AMD**
|
||||
|
||||
- **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
|
||||
- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
|
||||
- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
|
||||
|
||||
**Hardware Requirements: Intel Xeon CPU**
|
||||
|
||||
- It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server.
|
||||
|
||||
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2.7 \
|
||||
--tp 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**Testing Deployment:**
|
||||
|
||||
After startup, you can test the SGLang OpenAI-compatible API with the following command:
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M2.7",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
**Simple Completion Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Who won the world series in 2020?"}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output**:
|
||||
```text Output
|
||||
<think>The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly.
|
||||
|
||||
We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context.
|
||||
|
||||
We should answer concisely.
|
||||
|
||||
Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas.
|
||||
|
||||
We must avoid disallowed content, no issue.
|
||||
|
||||
Thus final.
|
||||
</think>
|
||||
|
||||
The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic.
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2.7 \
|
||||
--tp 4 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process**
|
||||
|
||||
With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream, separating <think>...</think> from content
|
||||
in_think = False
|
||||
think_printed_header = False
|
||||
content_printed_header = False
|
||||
buffer = ""
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
buffer += delta.content
|
||||
|
||||
while buffer:
|
||||
if in_think:
|
||||
# Look for closing </think> tag
|
||||
end_idx = buffer.find("</think>")
|
||||
if end_idx != -1:
|
||||
print(buffer[:end_idx], end="", flush=True)
|
||||
buffer = buffer[end_idx + len("</think>"):]
|
||||
in_think = False
|
||||
else:
|
||||
# Still in thinking, print what we have
|
||||
print(buffer, end="", flush=True)
|
||||
buffer = ""
|
||||
else:
|
||||
# Look for opening <think> tag
|
||||
start_idx = buffer.find("<think>")
|
||||
if start_idx != -1:
|
||||
# Print any content before <think>
|
||||
before = buffer[:start_idx]
|
||||
if before:
|
||||
if not content_printed_header:
|
||||
print("=============== Content =================", flush=True)
|
||||
content_printed_header = True
|
||||
print(before, end="", flush=True)
|
||||
buffer = buffer[start_idx + len("<think>"):]
|
||||
in_think = True
|
||||
if not think_printed_header:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
think_printed_header = True
|
||||
else:
|
||||
# No <think> tag, print as content
|
||||
if not content_printed_header and think_printed_header:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
content_printed_header = True
|
||||
print(buffer, end="", flush=True)
|
||||
buffer = ""
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation.
|
||||
|
||||
But also ensure we follow any policy? No issues. Just straightforward.
|
||||
|
||||
I'll provide a step-by-step solution.
|
||||
|
||||
Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.
|
||||
|
||||
Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36.
|
||||
|
||||
All good.
|
||||
|
||||
=============== Content =================
|
||||
|
||||
**Step‑by‑step solution**
|
||||
|
||||
1. **Convert the percent to a decimal (or a fraction).**
|
||||
|
||||
15% = 15/100 = 0.15 = 3/20
|
||||
|
||||
2. **Multiply the original number (240) by this decimal/fraction.**
|
||||
|
||||
Using the decimal:
|
||||
240 × 0.15 = 36
|
||||
|
||||
Or using the fraction:
|
||||
240 × 3/20 = (240/20) × 3 = 12 × 3 = 36
|
||||
|
||||
3. **Result:**
|
||||
|
||||
15% of 240 = **36**
|
||||
|
||||
*Check:*
|
||||
- 10% of 240 = 24
|
||||
- 5% of 240 = 12
|
||||
- Adding them: 24 + 12 = 36, which matches the calculation.
|
||||
```
|
||||
|
||||
**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2.7 \
|
||||
--tp 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Non-streaming request
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
|
||||
# Check for tool calls
|
||||
if message.tool_calls:
|
||||
for tool_call in message.tool_calls:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
else:
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
**Output Example**:
|
||||
```text Output
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=messages
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
```text Output
|
||||
The weather in Beijing is currently 22°C and sunny.
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
**Test Environment**:
|
||||
|
||||
- Hardware: 2× NVIDIA GB300 (275GB per die)
|
||||
- Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130`
|
||||
- Model: MiniMax-M2.7 (FP8)
|
||||
- Tensor Parallelism: 2
|
||||
- SGLang version: 0.5.10.post1
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills)
|
||||
|
||||
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True`
|
||||
|
||||
#### 5.1.1 GPQA Diamond
|
||||
|
||||
- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
|
||||
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking))
|
||||
- Evaluation command:
|
||||
```bash Command
|
||||
ns prepare_data gpqa
|
||||
|
||||
ns eval \
|
||||
--cluster=local \
|
||||
--server_type=openai \
|
||||
--model=MiniMaxAI/MiniMax-M2.7 \
|
||||
--server_address=http://localhost:30000/v1 \
|
||||
--output_dir=./m2.7-eval/ \
|
||||
--benchmarks=gpqa:8 \
|
||||
++prompt_config=eval/aai/mcq-4choices \
|
||||
++inference.tokens_to_generate=120000 \
|
||||
++inference.temperature=0.6 \
|
||||
++inference.top_p=0.95 \
|
||||
++parse_reasoning=True
|
||||
```
|
||||
- Test Results:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>84.91%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3.54%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**88.89%**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>96.46%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 AIME 2025
|
||||
|
||||
- Dataset: AIME 2025 (30 problems)
|
||||
- Prompt: `generic/math` (boxed answer format)
|
||||
- Evaluation command:
|
||||
```bash Command
|
||||
ns prepare_data aime25
|
||||
|
||||
ns eval \
|
||||
--cluster=local \
|
||||
--server_type=openai \
|
||||
--model=MiniMaxAI/MiniMax-M2.7 \
|
||||
--server_address=http://localhost:30000/v1 \
|
||||
--output_dir=./m2.7-eval/ \
|
||||
--benchmarks=aime25:8 \
|
||||
++inference.tokens_to_generate=120000 \
|
||||
++inference.temperature=0.6 \
|
||||
++inference.top_p=0.95 \
|
||||
++parse_reasoning=True
|
||||
```
|
||||
- Test Results:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.50% ± 5.56%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>2.92%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**97.08%**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.3 MMLU-Pro
|
||||
|
||||
- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
|
||||
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice)
|
||||
- Evaluation command:
|
||||
```bash Command
|
||||
ns prepare_data mmlu-pro
|
||||
|
||||
ns eval \
|
||||
--cluster=local \
|
||||
--server_type=openai \
|
||||
--model=MiniMaxAI/MiniMax-M2.7 \
|
||||
--server_address=http://localhost:30000/v1 \
|
||||
--output_dir=./m2.7-eval/ \
|
||||
--benchmarks=mmlu-pro \
|
||||
++prompt_config=eval/aai/mcq-10choices \
|
||||
++inference.tokens_to_generate=32768 \
|
||||
++inference.temperature=0.0 \
|
||||
++parse_reasoning=True
|
||||
```
|
||||
- Test Results:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (greedy)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>69.41%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>18.75%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly.
|
||||
|
||||
#### 5.1.4 GSM8K Benchmark
|
||||
- Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API
|
||||
- Test Results:
|
||||
```text Output
|
||||
GSM8K Results (8-shot CoT)
|
||||
Model: MiniMaxAI/MiniMax-M2.7
|
||||
Total: 1319
|
||||
Correct: 1218
|
||||
Accuracy: 92.34%
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
#### 5.2.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 34.33
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.29
|
||||
Input token throughput (tok/s): 177.71
|
||||
Output token throughput (tok/s): 122.92
|
||||
Total token throughput (tok/s): 300.63
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3431.21
|
||||
Median E2E Latency (ms): 2742.57
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 50.28
|
||||
Median TTFT (ms): 53.85
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.02
|
||||
Median TPOT (ms): 8.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.03
|
||||
Median ITL (ms): 8.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 100.20
|
||||
Total input tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Request throughput (req/s): 4.99
|
||||
Input token throughput (tok/s): 2493.41
|
||||
Output token throughput (tok/s): 2521.66
|
||||
Total token throughput (tok/s): 5015.07
|
||||
Concurrency: 90.19
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18072.69
|
||||
Median E2E Latency (ms): 17761.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 247.94
|
||||
Median TTFT (ms): 92.05
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 35.75
|
||||
Median TPOT (ms): 36.67
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.34
|
||||
Median ITL (ms): 30.55
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,613 @@
|
||||
---
|
||||
title: MiniMax-M2
|
||||
metatags:
|
||||
description: "Deploy MiniMax-M2 with SGLang - community contribution guide for MiniMax M2 model deployment."
|
||||
---
|
||||
|
||||
import { MiniMaxM2Deployment } from '/src/snippets/autoregressive/minimax-m2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
- **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally.
|
||||
|
||||
- **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages.
|
||||
|
||||
- **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps.
|
||||
|
||||
- **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks.
|
||||
|
||||
For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The AMD environment is currently available in SGLang via Docker image install.
|
||||
|
||||
### 2.1 AMD Docker
|
||||
#### 2.1.1 Launch docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Minimax \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 2.1.2 Make modifications inside the docker
|
||||
|
||||
```shell Command
|
||||
mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \
|
||||
/sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py
|
||||
```
|
||||
|
||||
#### 2.1.3 Fix torch compile
|
||||
Comment out the following line: @torch.compile(dynamic=True, backend=get_compiler_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax_m2.py
|
||||
```shell Command
|
||||
#@torch.compile(dynamic=True, backend=get_compiler_backend())
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<MiniMaxM2Deployment />
|
||||
|
||||
#### 3.1.1 NVIDIA GPU Deployment
|
||||
|
||||
The interactive command generator above covers AMD deployments. For NVIDIA GPUs (H100/H200/B200), use these explicit commands:
|
||||
|
||||
**4-GPU deployment (up to 400K context):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 30000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**8-GPU deployment (up to 3M context):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 30000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
### 3.2 System Requirements
|
||||
|
||||
Recommended configurations — actual requirements depend on workload:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPUs</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length Support</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4× 96 GB GPUs</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Up to 400K tokens</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8× 144 GB GPUs</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Up to 3M tokens</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### 3.3 Testing Deployment
|
||||
|
||||
After the server starts, verify with:
|
||||
|
||||
```shell Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M2",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
Server Command:
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
Test Code:
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
Output Example:
|
||||
```text Output
|
||||
<think>First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction.
|
||||
|
||||
The problem is: What is 15% of 240?
|
||||
|
||||
To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15.
|
||||
|
||||
Therefore, 15% of 240 is 240 times 0.15.
|
||||
|
||||
Let me calculate that: 240 × 0.15.
|
||||
|
||||
I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100.
|
||||
|
||||
Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600.
|
||||
|
||||
Then, divide by 100: 3600 / 100 = 36.
|
||||
|
||||
So, 15% of 240 is 36.
|
||||
|
||||
I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer.
|
||||
|
||||
Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36.
|
||||
|
||||
All methods confirm it's 36.
|
||||
|
||||
The user said "solve this problem step by step," so I should present the steps clearly.
|
||||
|
||||
Step 1: Understand that "15% of 240" means 15 per hundred of 240.
|
||||
|
||||
Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15.
|
||||
|
||||
Step 3: Multiply the number by the decimal: 240 × 0.15.
|
||||
|
||||
Step 4: Calculate the multiplication: 240 × 0.15 = 36.
|
||||
|
||||
Step 5: Therefore, 15% of 240 is 36.
|
||||
|
||||
I should also mention that percentage means per hundred, so it's straightforward.
|
||||
|
||||
Finally, I need to box the answer as per the instruction.
|
||||
|
||||
So, the final answer is 36.
|
||||
</think>
|
||||
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100.
|
||||
2. **Convert to a decimal**: 15/100 = 0.15.
|
||||
3. **Multiply by the number**: 240 × 0.15.
|
||||
4. **Calculate the result**:
|
||||
- 240 × 0.15 = 36.
|
||||
|
||||
Alternatively, you can break it down:
|
||||
- 10% of 240 is 24 (since 240 ÷ 10 = 24).
|
||||
- 5% of 240 is half of 10%, which is 12.
|
||||
- Therefore, 15% is 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Both methods confirm the result.
|
||||
|
||||
**Answer**: 36
|
||||
```
|
||||
|
||||
### 4.2.2 Tool Calling
|
||||
|
||||
Server Command:
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
Test Code:
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
Output Example:
|
||||
```text Output
|
||||
Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me.
|
||||
|
||||
Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me.
|
||||
|
||||
Looking at the tool's parameters, I can see it requires:
|
||||
1. location - which is required and should be a string representing the city name
|
||||
2. unit - which is optional and can be either "celsius" or "fahrenheit"
|
||||
|
||||
For the location parameter, I'll use "Beijing" since that's what the user asked about.
|
||||
|
||||
For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information.
|
||||
|
||||
So I need to make a tool call to get_weather with the following parameters:
|
||||
- location: "Beijing"
|
||||
- unit: "celsius"
|
||||
|
||||
This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions.
|
||||
</think>
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
### 5.1 Speed Benchmark
|
||||
**Test Environment**:
|
||||
|
||||
- Hardware: AMD MI300X GPU(4x)
|
||||
|
||||
- Model: MiniMax-M2
|
||||
|
||||
- Tensor Parallelism: 4
|
||||
|
||||
- sglang version: 0.5.7
|
||||
|
||||
**Model Deployment**:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
### 5.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 138.91
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.07
|
||||
Input token throughput (tok/s): 43.92
|
||||
Output token throughput (tok/s): 30.38
|
||||
Peak output token throughput (tok/s): 46.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 74.30
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13887.62
|
||||
Median E2E Latency (ms): 10377.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4528.94
|
||||
Median TTFT (ms): 385.23
|
||||
P99 TTFT (ms): 38338.51
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 22.21
|
||||
Median TPOT (ms): 22.24
|
||||
P99 TPOT (ms): 22.25
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 22.23
|
||||
Median ITL (ms): 22.24
|
||||
P95 ITL (ms): 22.35
|
||||
P99 ITL (ms): 22.41
|
||||
Max ITL (ms): 23.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 81.07
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40803
|
||||
Request throughput (req/s): 0.99
|
||||
Input token throughput (tok/s): 489.29
|
||||
Output token throughput (tok/s): 503.32
|
||||
Peak output token throughput (tok/s): 704.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 992.61
|
||||
Concurrency: 13.74
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13925.95
|
||||
Median E2E Latency (ms): 14348.75
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 532.32
|
||||
Median TTFT (ms): 147.69
|
||||
P99 TTFT (ms): 1978.48
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 27.49
|
||||
Median TPOT (ms): 26.56
|
||||
P99 TPOT (ms): 46.52
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 26.31
|
||||
Median ITL (ms): 23.47
|
||||
P95 ITL (ms): 24.37
|
||||
P99 ITL (ms): 125.10
|
||||
Max ITL (ms): 1192.51
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 153.71
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250982
|
||||
Request throughput (req/s): 3.25
|
||||
Input token throughput (tok/s): 1625.33
|
||||
Output token throughput (tok/s): 1643.75
|
||||
Peak output token throughput (tok/s): 2597.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3269.09
|
||||
Concurrency: 91.14
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 28017.24
|
||||
Median E2E Latency (ms): 26865.28
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 387.41
|
||||
Median TTFT (ms): 183.90
|
||||
P99 TTFT (ms): 1192.44
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 55.23
|
||||
Median TPOT (ms): 57.84
|
||||
P99 TPOT (ms): 70.23
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 54.79
|
||||
Median ITL (ms): 39.01
|
||||
P95 ITL (ms): 143.10
|
||||
P99 ITL (ms): 150.46
|
||||
Max ITL (ms): 986.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Server Command**:
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
- **Benchmark Command**:
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
- **Result**:
|
||||
- MiniMax-M2
|
||||
```text Output
|
||||
Accuracy: 0.950
|
||||
Invalid: 0.000
|
||||
Latency: 15.120 s
|
||||
Output throughput: 1306.711 token/s
|
||||
```
|
||||
@@ -0,0 +1,506 @@
|
||||
---
|
||||
title: MiniMax-M3
|
||||
description: "Deploy MiniMax-M3 with SGLang — a ~428B-param (23B activated) multimodal Mixture-of-Experts reasoning model with MiniMax Sparse Attention and 1M context, MXFP8 on NVIDIA Blackwell & AMD Instinct, bf16 on Hopper."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
# MiniMax-M3 ships in SGLang PR #27944, not yet in a tagged release — install from
|
||||
# the PR head. The serving runtime is in the base dependencies, so no extra is needed:
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git fetch origin pull/27944/head && git checkout FETCH_HEAD
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image bundles the CUDA-13 runtime and the #27944 code. Once [PR #27944](https://github.com/sgl-project/sglang/pull/27944) is merged and released, `uv pip install sglang` will pull M3 support directly.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# Pull the M3 image the command panel selects for your platform, e.g.:
|
||||
docker pull lmsysorg/sglang:dev-cu13-minimax-m3
|
||||
```
|
||||
|
||||
The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces.
|
||||
|
||||
<Note>
|
||||
These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path.
|
||||
</Note>
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/MiniMaxAI/minimax-m3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is MiniMax's native-multimodal Mixture-of-Experts reasoning model: **~428B total parameters with ~23B activated per token** (128 experts, 4 active per token), 60 layers, and a **1M-token context** over text, image, and video. Its defining feature is **MiniMax Sparse Attention (MSA)** — a block-sparse "lightning indexer" attention that keeps long-context cost low (MiniMax reports ~9× prefill / ~15× decode speedup over M2 at 1M context). This page serves the **MXFP8** variant (`MiniMaxAI/MiniMax-M3-MXFP8`, ~440 GB) on NVIDIA Blackwell and AMD Instinct; on NVIDIA Hopper (H200), use the full-precision **bfloat16** build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3) (§2.4). Released under the **MiniMax Community License**.
|
||||
|
||||
Key characteristics as served by SGLang:
|
||||
|
||||
- **Multimodal (vision + text)**: accepts interleaved text and images through the OpenAI-compatible chat API (loaded as `MiniMaxM3SparseForConditionalGeneration`). Image input via URL and base64 is validated; video input has not been tested here.
|
||||
- **Reasoning model**: emits its chain of thought wrapped in `<mm:think>...</mm:think>`. Always launch with **`--reasoning-parser auto`** — it auto-detects the right parser from the chat template, and SGLang then strips the tags and returns the trace separately in `message.reasoning_content`.
|
||||
- **Native tool calling**: a custom namespace-token XML format, parsed into standard OpenAI `tool_calls`. Always launch with **`--tool-call-parser auto`** — it auto-detects the right parser from the chat template. Single, parallel, and nested (object / array) arguments are supported.
|
||||
- **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1).
|
||||
- **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized.
|
||||
|
||||
**Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client.
|
||||
|
||||
**Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
### 2.1 MSA sparse-attention fast path (recommended for Blackwell users)
|
||||
|
||||
[MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops ~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs.
|
||||
|
||||
**Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)):
|
||||
|
||||
- **GPU**: NVIDIA SM100 family — sm_100 (B200 / GB200) and sm_103 (B300 / GB300).
|
||||
- **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import.
|
||||
- **Python**: ≥ 3.10; **OS**: Linux — works on both **x86_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits.
|
||||
|
||||
<Accordion title="Install MSA (only on a custom image) & verify the gate (Python)">
|
||||
|
||||
The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`.
|
||||
|
||||
```bash Command
|
||||
# Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation
|
||||
git clone --recursive https://github.com/MiniMax-AI/MSA.git msa
|
||||
cd msa && pip install .
|
||||
# Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback):
|
||||
python -c "from sglang.srt.layers.attention.minimax_sparse_ops.msa import msa_available; print(msa_available())"
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
The first import JIT-compiles the kernels, which can take 30 s to a few minutes on a cold `nvcc` cache — this is normal, not a hang. Subsequent server starts hit the JIT cache.
|
||||
</Note>
|
||||
|
||||
<Warning>
|
||||
**Warm the JIT cache before a multi-GPU launch.** On a *cold* cache, several tensor-parallel ranks racing to JIT-compile MSA's plan kernel can leave one rank loading a half-linked module (`AttributeError: Module has no function 'plan'` at CUDA-graph capture). Run the gate-check `python -c "..."` (or any single-process `fmha_sm100_plan` call) once before launching the server — that compiles the kernel single-process, and every rank then hits the warm cache.
|
||||
</Warning>
|
||||
|
||||
The gate requires `--attention-backend fa4` (MSA's sparse blocks are 128 tokens, so the page size must be 128). SGLang auto-forces `page_size` to 128 for the `fa4` backend — including the combined `--attention-backend fa4` the M3 recipe uses (#28976) — so `--page-size 128` is omitted from the Blackwell cells below. Force the Triton path at any time with the env var `SGLANG_DISABLE_MSA=1`. MSA is a Blackwell (SM100) kernel and does not apply to the AMD ROCm paths.
|
||||
|
||||
<Note>
|
||||
For multimodal (image) serving, keep the same text recipe above — `--attention-backend fa4` (MSA) is unchanged — and add `--mm-attention-backend flashinfer_cudnn` for the vision tower. The text and vision-tower attention backends are independent knobs; MSA only touches the language-model sparse attention, not image handling.
|
||||
</Note>
|
||||
|
||||
### 2.2 Memory and workload tuning
|
||||
|
||||
The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm_100; GB300 is sm_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**.
|
||||
|
||||
- **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context.
|
||||
- **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated.
|
||||
- **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom.
|
||||
- **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy.
|
||||
- `--trust-remote-code` is required to load the MiniMax config / processor classes.
|
||||
|
||||
### 2.3 AMD Instinct (ROCm)
|
||||
|
||||
MiniMax-M3 runs on AMD Instinct GPUs through two code paths, by architecture — both selected automatically; you still pass `--quantization mxfp8` either way:
|
||||
|
||||
- **MI350X / MI355X (gfx950, CDNA4)** has hardware MX-scaled matmul, so the **MXFP8 weights are served natively**. SGLang auto-detects the checkpoint, selects the Triton MiniMax-M3 MoE path with the packaged tuned MXFP8 configs, and enables AITER fused all-reduce for single-node tensor parallelism. The launch command is the NVIDIA recipe minus the Blackwell-only backend flags.
|
||||
- **MI300X / MI325X (gfx942, CDNA3)** has **no** hardware MX matmul. SGLang transparently **converts the MXFP8 weights to block-fp8 `[128,128]` at load time**, then serves them with the tuned ROCm block-fp8 kernels (`--attention-backend aiter`, `--moe-runner-backend triton`; the `aiter` runner also works and scores marginally higher). On a cold start the first generation can JIT-compile AITER configs and exceed the default warmup/HTTP timeout, so the recipe adds `--watchdog-timeout 3600 --skip-server-warmup`. The block-fp8 step adds only a small relative error over MXFP8's native `1×32` scaling — negligible on GSM8K (see the benchmark card).
|
||||
|
||||
Select an MI300X/MI325X or MI350X/MI355X tile in the command panel above to get the exact launch command for each path.
|
||||
|
||||
<Note>
|
||||
The AMD recipes are validated end-to-end on **text** workloads — chat, reasoning separation, and tool calling. The vision tower was not exercised on ROCm; for image input on AMD, omit the Blackwell `--mm-attention-backend flashinfer_cudnn` flag and let the encoder use the ROCm default backend, and treat vision as unvalidated on that path.
|
||||
</Note>
|
||||
|
||||
### 2.4 Serving on Hopper (H200) with the bf16 build
|
||||
|
||||
The MXFP8 kernels are Blackwell-only, so Hopper (H200) serves the full-precision bfloat16 build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3). Select **H200 + BF16** in the Deploy panel above for the exact command — it runs at `--tp 8` (the bf16 weights need a full 8-GPU node). SGLang picks the right backends for Hopper automatically, so the recipe stays minimal:
|
||||
|
||||
- **MoE runner**: Triton, auto-selected for bf16 weights.
|
||||
- **Attention**: FlashAttention-3 with page size 1. MSA (§2.1) is a Blackwell kernel, so M3's sparse step runs on the built-in Triton path here.
|
||||
- **CUDA graph**: on, with full decode-graph capture.
|
||||
|
||||
**High-concurrency throughput (optional).** On Hopper the sparse prefill runs on the Triton path as a separate eager forward, which briefly stalls the in-flight decode batch under heavy concurrent load. Adding `--enable-mixed-chunk --chunked-prefill-size 2048` merges the running decodes into the prefill step instead of preempting them, which recovers roughly **+10% output throughput** and **~10% lower median TPOT** at high concurrency on 8×H200, with no change in accuracy. Leave it off for latency-sensitive low-concurrency serving.
|
||||
|
||||
Validated on 8×H200 — reasoning and tool-call auto-detection plus long-context generation. For prefill/decode disaggregation on Hopper, see §3.4.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser auto` (or toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)). The `<mm:think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
=============== Reasoning ===============
|
||||
15% of 240. 15% = 0.15. 240 * 0.15 = 36. Quick check: 10% is 24, 5% is 12, 24 + 12 = 36.
|
||||
=============== Answer ==================
|
||||
15% of 240 is **36**.
|
||||
(10% of 240 = 24, and 5% of 240 = 12; 24 + 12 = 36.)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two sections can be rendered separately in real time:
|
||||
|
||||
<Accordion title="Streaming Reasoning (Python)">
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}],
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
print(delta.reasoning_content, end="", flush=True) # thinking stream
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True) # answer stream
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
[delta.reasoning_content — thinking stream]
|
||||
Let me solve this step by step.
|
||||
|
||||
15% of 240
|
||||
= 0.15 × 240
|
||||
= 36
|
||||
|
||||
Let me verify: 10% of 240 = 24, 5% of 240 = 12, so 15% = 24 + 12 = 36. ✓
|
||||
|
||||
[delta.content — answer stream]
|
||||
# Solving 15% of 240
|
||||
## Step 1: Convert the percentage to a decimal
|
||||
15% = 15/100 = 0.15
|
||||
## Step 2: Multiply by 240
|
||||
0.15 × 240 = 36
|
||||
## Answer
|
||||
**15% of 240 = 36**
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser auto` (or toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) — it auto-detects M3's tool-call parser from the chat template. M3 emits tool calls in a custom namespace-token XML format:
|
||||
|
||||
```text Raw model output
|
||||
]<]minimax[>[<tool_call>
|
||||
]<]minimax[>[<invoke name="get_weather">]<]minimax[>[<location>Beijing]<]minimax[>[</location>]<]minimax[>[</invoke>
|
||||
]<]minimax[>[</tool_call>
|
||||
```
|
||||
|
||||
The parser converts that into the standard OpenAI `tool_calls` structure:
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
Beyond a single flat call, the parser also supports:
|
||||
|
||||
- **Parallel calls** — multiple `<invoke>` blocks inside the single `<tool_call>` wrapper, surfaced as multiple `message.tool_calls` entries.
|
||||
- **Nested object arguments** — an `object`-typed parameter is emitted as nested XML tags and reconstructed into a JSON object.
|
||||
- **Array arguments** — an `array`-typed parameter uses repeated `<item>` children and is reconstructed into a JSON list.
|
||||
|
||||
For example, a tool with object and array parameters round-trips cleanly:
|
||||
|
||||
```text Output
|
||||
create_event {"title": "Design sync", "attendees": ["alice", "bob"], "location": {"room": "R2", "floor": 3}}
|
||||
```
|
||||
|
||||
To return a tool result, append the assistant's `tool_calls` turn plus a matching `tool` message and ask the model to continue — the follow-up answer may place text in `reasoning_content` as well as `content`, so print both.
|
||||
|
||||
### 3.3 Multimodal (Vision) Input
|
||||
|
||||
Images go through the standard OpenAI `image_url` content type. The vision tower is always loaded; for image serving add `--mm-attention-backend flashinfer_cudnn` (the vision-tower backend) to the Blackwell deployment recipe — the text `--attention-backend` is unchanged (§2.1 note). On AMD, omit `--mm-attention-backend` and let the encoder use the ROCm default (vision is unvalidated on ROCm — §2.3).
|
||||
|
||||
<Accordion title="Vision Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
|
||||
},
|
||||
},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
This image captures a striking and unusual urban scene on what appears to be a busy New York City street.
|
||||
|
||||
**Main Subject:**
|
||||
A man stands on the rear bumper of a yellow taxi cab (an SUV-style cab, likely a Ford Escape hybrid), operating a full-sized ironing board set up across the back of the vehicle. He is wearing a bright yellow long-sleeved shirt and dark pants, and is actively ironing a blue garment, holding an iron in his right hand.
|
||||
|
||||
**Vehicles:**
|
||||
- The yellow SUV taxi on the right is stationary, its rear hatch serving as the ironing platform.
|
||||
- A second yellow taxi (a sedan) drives past on the left, captured with motion blur.
|
||||
|
||||
**Setting:**
|
||||
Tall city buildings with classic urban architecture, an American flag, and white lane markings — a bustling downtown area, possibly Midtown Manhattan.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
Notes:
|
||||
|
||||
- If the server cannot fetch external URLs, embed the image as a base64 `data:image/png;base64,...` URI — SGLang decodes it server-side.
|
||||
- Multiple images per message are supported; add more `image_url` entries to the `content` list.
|
||||
- Reasoning and tool calling work the same way for multimodal requests — a vision prompt can still produce a `<mm:think>` trace and/or tool calls.
|
||||
|
||||
### 3.4 Prefill-Decode (PD) Disaggregation
|
||||
|
||||
[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. M3 needs one thing beyond a dense model: alongside the main KV cache, every sparse "lightning-indexer" layer keeps a **K-only index buffer**, and that buffer must reach the decode server too — otherwise sparse attention reads stale state. SGLang transfers it alongside the main KV — reusing the same page mapping — so M3 disaggregates correctly with no extra flags.
|
||||
|
||||
**Supported topology** (the released MiniMax-M3, whose sparse layers are all K-only):
|
||||
|
||||
- **Equal tensor parallelism** — the prefill and decode servers run the same `--tp`.
|
||||
- **Single pipeline stage** — PP = 1 (the default).
|
||||
- **mooncake or NIXL** transfer backend over RDMA / InfiniBand.
|
||||
|
||||
Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Pick your hardware:
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Blackwell · MXFP8">
|
||||
|
||||
On Blackwell the MXFP8 recipe — fa4, page size 128, deep_gemm MoE, and the MSA fast path (§2.1) — is auto-selected, so each role adds only the `--disaggregation-*` flags. This is the validated **2 × 4×B200** setup (TP4 prefill on node A, TP4 decode on node B); point `--disaggregation-ib-device` at your RDMA NIC(s).
|
||||
|
||||
```bash Prefill server (node A)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 4 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-ib-device mlx5_0 \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--disaggregation-bootstrap-port 8998
|
||||
```
|
||||
|
||||
```bash Decode server (node B)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 4 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-ib-device mlx5_0 \
|
||||
--host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Hopper · bf16">
|
||||
|
||||
On Hopper (H200) M3 runs the bf16 build (§2.4) with Triton MoE and the built-in Triton sparse path, pinned to `--page-size 128` so both roles share the page layout the sparse-index transfer relies on. This is the validated **2 × 8×H200** setup (TP8 each).
|
||||
|
||||
```bash Prefill server (node A)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 8 \
|
||||
--attention-backend triton \
|
||||
--moe-runner-backend triton \
|
||||
--page-size 128 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--disaggregation-bootstrap-port 8998
|
||||
```
|
||||
|
||||
```bash Decode server (node B)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 8 \
|
||||
--attention-backend triton \
|
||||
--moe-runner-backend triton \
|
||||
--page-size 128 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:
|
||||
|
||||
```bash PD router
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://<prefill-host>:30000 8998 \
|
||||
--decode http://<decode-host>:30001 \
|
||||
--policy round_robin \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Clients hit the router exactly like a single server — it splits each request across the two stages transparently:
|
||||
|
||||
<Accordion title="PD Client Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "What is 2 + 2?"}],
|
||||
max_tokens=64,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
2 + 2 = 4
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
**Validation.** PD disaggregation preserves output quality — the K-only sparse index transfers arrive intact and disaggregated output matches non-disaggregated serving. GSM8K is scored with the single sgl-eval harness used by the benchmark card above (full 1319-question split, chat with `--thinking`); see that card for per-platform single-node accuracy.
|
||||
|
||||
- **2 × 4×B200** (TP4+TP4, MXFP8, NIXL over InfiniBand) — output matches single-node serving. The 2-node PD serving benchmark (512-token input, 256-token output, 16 concurrent — a different workload from the card's single-node `random` isl=2048 / osl=256 / conc=64 row, so the throughput figures are not directly comparable) measured mean TTFT 1.1 s and TPOT 16.6 ms (≈ 60 tok/s per stream, ≈ 2.3k tok/s aggregate).
|
||||
- **2 × 8×H200** (TP8+TP8, bf16, mooncake) — output matches single-node serving.
|
||||
@@ -0,0 +1,518 @@
|
||||
---
|
||||
title: Devstral 2 (Mistral)
|
||||
metatags:
|
||||
description: "Deploy Devstral 2 agentic coding models with SGLang - optimized for tool use, codebase exploration, and multi-file edits with 256K context."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**.
|
||||
|
||||
The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Agentic coding**: Optimized for tool-driven coding and software engineering agents
|
||||
- **Improved performance**: A step up compared to earlier Devstral models
|
||||
- **Better generalization**: More robust across diverse prompts and coding environments
|
||||
- **Long context**: Up to a **256K** context window
|
||||
|
||||
**Use Cases:**
|
||||
AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration.
|
||||
|
||||
For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral.
|
||||
|
||||
**Models:**
|
||||
|
||||
- **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2)
|
||||
- **FP8 Instruct**:
|
||||
- **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)**
|
||||
- **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)**
|
||||
|
||||
---
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
<Warning title="Transformers version requirement">
|
||||
Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`:
|
||||
|
||||
```shell Command
|
||||
python -c "import transformers; print(transformers.__version__)"
|
||||
```
|
||||
|
||||
If your version is lower, upgrade:
|
||||
|
||||
```shell Command
|
||||
pip install -U --pre "transformers>=5.0.0rc0"
|
||||
```
|
||||
</Warning>
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B).
|
||||
|
||||
<Note>
|
||||
The TP size is set to the minimum required for the selected model size.
|
||||
</Note>
|
||||
|
||||
|
||||
import { Devstral2Deployment } from "/src/snippets/autoregressive/devstral-2-deployment.jsx";
|
||||
|
||||
<Devstral2Deployment />
|
||||
|
||||
### 3.2 Configuration tips
|
||||
|
||||
- **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable.
|
||||
- **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers.
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage (OpenAI-Compatible API)
|
||||
|
||||
SGLang exposes an OpenAI-compatible endpoint. Example:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="mistralai/Devstral-Small-2-24B-Instruct-2512",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful coding assistant."},
|
||||
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
|
||||
],
|
||||
temperature=0.2,
|
||||
max_tokens=512,
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors.
|
||||
|
||||
```python
|
||||
import time
|
||||
import requests
|
||||
from requests.exceptions import RequestException
|
||||
|
||||
def retry_with_exponential_backoff(
|
||||
url,
|
||||
max_retries=3,
|
||||
initial_delay=1,
|
||||
backoff_factor=2,
|
||||
method="GET",
|
||||
**kwargs
|
||||
):
|
||||
"""
|
||||
Retry a request with exponential backoff.
|
||||
|
||||
Parameters:
|
||||
- url: The URL to request.
|
||||
- max_retries: Maximum number of retry attempts (default: 3).
|
||||
- initial_delay: Initial delay in seconds (default: 1).
|
||||
- backoff_factor: Multiplier for the delay between retries (default: 2).
|
||||
- method: HTTP method to use (default: "GET").
|
||||
- **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.).
|
||||
|
||||
Returns:
|
||||
- Response object if the request succeeds.
|
||||
- Raises an exception if all retries fail.
|
||||
"""
|
||||
retry_count = 0
|
||||
delay = initial_delay
|
||||
|
||||
while retry_count < max_retries:
|
||||
try:
|
||||
response = requests.request(method, url, **kwargs)
|
||||
# Check if the response status code indicates success
|
||||
if response.status_code < 400:
|
||||
return response
|
||||
else:
|
||||
raise RequestException(f"HTTP {response.status_code}: {response.text}")
|
||||
|
||||
except RequestException as e:
|
||||
if retry_count == max_retries - 1:
|
||||
raise Exception(f"All retries failed. Last error: {e}")
|
||||
|
||||
print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...")
|
||||
time.sleep(delay)
|
||||
...
|
||||
```
|
||||
|
||||
### 4.2 Tool calling (optional)
|
||||
|
||||
Devstral 2 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--tp 2 \
|
||||
--tool-call-parser mistral
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Devstral-2-123B-Instruct-2512",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
|
||||
## AMD GPU Support
|
||||
|
||||
## 1. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
|
||||
### 1.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 1.2 Advanced Usage
|
||||
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
## 2.Benchmark
|
||||
|
||||
### 5.1 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 94.30
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4206
|
||||
Request throughput (req/s): 0.11
|
||||
Input token throughput (tok/s): 64.70
|
||||
Output token throughput (tok/s): 44.75
|
||||
Peak output token throughput (tok/s): 82.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 109.44
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9427.59
|
||||
Median E2E Latency (ms): 5637.23
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4253.85
|
||||
Median TTFT (ms): 116.95
|
||||
P99 TTFT (ms): 37764.48
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.28
|
||||
Median TPOT (ms): 12.29
|
||||
P99 TPOT (ms): 12.30
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 12.29
|
||||
Median ITL (ms): 12.29
|
||||
P95 ITL (ms): 12.38
|
||||
P99 ITL (ms): 12.42
|
||||
Max ITL (ms): 12.90
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 52.11
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40761
|
||||
Request throughput (req/s): 1.54
|
||||
Input token throughput (tok/s): 761.31
|
||||
Output token throughput (tok/s): 783.13
|
||||
Peak output token throughput (tok/s): 1120.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1544.44
|
||||
Concurrency: 13.60
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8856.19
|
||||
Median E2E Latency (ms): 9314.71
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 398.80
|
||||
Median TTFT (ms): 127.81
|
||||
P99 TTFT (ms): 1500.32
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.32
|
||||
Median TPOT (ms): 16.90
|
||||
P99 TPOT (ms): 32.78
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 16.61
|
||||
Median ITL (ms): 14.26
|
||||
P95 ITL (ms): 15.07
|
||||
P99 ITL (ms): 114.46
|
||||
Max ITL (ms): 1224.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 116.08
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252523
|
||||
Request throughput (req/s): 4.31
|
||||
Input token throughput (tok/s): 2152.21
|
||||
Output token throughput (tok/s): 2176.60
|
||||
Peak output token throughput (tok/s): 3600.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 4328.81
|
||||
Concurrency: 92.42
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 21456.71
|
||||
Median E2E Latency (ms): 20126.82
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 291.60
|
||||
Median TTFT (ms): 199.24
|
||||
P99 TTFT (ms): 866.02
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 42.42
|
||||
Median TPOT (ms): 45.18
|
||||
P99 TPOT (ms): 53.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 41.97
|
||||
Median ITL (ms): 27.59
|
||||
P95 ITL (ms): 130.43
|
||||
P99 ITL (ms): 137.87
|
||||
Max ITL (ms): 616.73
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### 5.2 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.3 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316 \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.922
|
||||
Invalid: 0.000
|
||||
Latency: 35.800 s
|
||||
Output throughput: 4507.697 token/s
|
||||
```
|
||||
@@ -0,0 +1,288 @@
|
||||
---
|
||||
title: Ministral-3
|
||||
metatags:
|
||||
description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
|
||||
---
|
||||
|
||||
import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.
|
||||
|
||||
The Ministral 3 14B Instruct model offers the following capabilities:
|
||||
|
||||
Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
|
||||
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
|
||||
System Prompt: Maintains strong adherence and support for system prompts.
|
||||
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
|
||||
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
|
||||
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
|
||||
Large Context Window: Supports a 256k context window.
|
||||
|
||||
For further details, please refer to the [official documentation](https://github.com/mistralai)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Ministral3Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.
|
||||
|
||||
**Pre-installation steps**: Adding the following steps after launching the docker
|
||||
```shell Command
|
||||
pip install mistral-common --upgrade
|
||||
pip install transformers==5.0.0.rc0
|
||||
```
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Ministral \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 Launch the server
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--tp 1 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: MI300X GPU (8x)
|
||||
- Model: mistralai/Ministral-3-14B-Instruct-2512
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--tp 1 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
##### Low Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 65.08
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4218
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 93.75
|
||||
Output token throughput (tok/s): 64.84
|
||||
Peak output token throughput (tok/s): 151.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 158.59
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6505.51
|
||||
Median E2E Latency (ms): 3037.37
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 3709.33
|
||||
Median TTFT (ms): 53.72
|
||||
P99 TTFT (ms): 33320.77
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.63
|
||||
Median TPOT (ms): 6.64
|
||||
P99 TPOT (ms): 6.66
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.64
|
||||
Median ITL (ms): 6.65
|
||||
P95 ITL (ms): 6.75
|
||||
P99 ITL (ms): 6.82
|
||||
Max ITL (ms): 8.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### Medium Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 31.20
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40783
|
||||
Request throughput (req/s): 2.56
|
||||
Input token throughput (tok/s): 1271.38
|
||||
Output token throughput (tok/s): 1307.82
|
||||
Peak output token throughput (tok/s): 1760.00
|
||||
Peak concurrent requests: 22
|
||||
Total token throughput (tok/s): 2579.20
|
||||
Concurrency: 13.72
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5351.07
|
||||
Median E2E Latency (ms): 5626.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 280.87
|
||||
Median TTFT (ms): 68.16
|
||||
P99 TTFT (ms): 1194.79
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.47
|
||||
Median TPOT (ms): 10.10
|
||||
P99 TPOT (ms): 20.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.96
|
||||
Median ITL (ms): 9.10
|
||||
P95 ITL (ms): 9.87
|
||||
P99 ITL (ms): 51.39
|
||||
Max ITL (ms): 888.63
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### High Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 88.75
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252547
|
||||
Request throughput (req/s): 5.63
|
||||
Input token throughput (tok/s): 2815.01
|
||||
Output token throughput (tok/s): 2846.91
|
||||
Peak output token throughput (tok/s): 4271.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5661.93
|
||||
Concurrency: 93.04
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16514.45
|
||||
Median E2E Latency (ms): 15834.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 148.57
|
||||
Median TTFT (ms): 99.15
|
||||
P99 TTFT (ms): 455.86
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.93
|
||||
Median TPOT (ms): 34.73
|
||||
P99 TPOT (ms): 38.05
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 32.45
|
||||
Median ITL (ms): 27.30
|
||||
P95 ITL (ms): 71.73
|
||||
P99 ITL (ms): 73.45
|
||||
Max ITL (ms): 328.10
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
@@ -0,0 +1,456 @@
|
||||
---
|
||||
title: Mistral Medium 3.5
|
||||
metatags:
|
||||
description: "Deploy Mistral Medium 3.5 with SGLang - 128B dense flagship merged model with hybrid reasoning, 256K context, vision input, and FP8 quantization."
|
||||
---
|
||||
|
||||
import { MistralMedium35Deployment } from '/src/snippets/autoregressive/mistral-medium-3-5-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head_dim=128)
|
||||
- **256K context window** — YARN RoPE scaling on top of the original 4K base
|
||||
- **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`)
|
||||
- **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios
|
||||
- **Function Calling**: Native tool calling and JSON output
|
||||
- **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in
|
||||
- **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali
|
||||
- **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue)
|
||||
|
||||
**Architecture:**
|
||||
|
||||
- Mistral 3 backbone with YARN RoPE for 256K context
|
||||
- Dense (no MoE), 128B parameters
|
||||
- Standard GQA attention (not MLA)
|
||||
- Pixtral-style vision encoder (48 layers, patch_size=14, spatial_merge=2, image_size=1540) trained from scratch
|
||||
- Multimodal input: text + image
|
||||
|
||||
**Models:**
|
||||
|
||||
- **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8)
|
||||
|
||||
The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present.
|
||||
|
||||
---
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
**Docker Image:** `lmsysorg/sglang:latest` covers all the GPUs in this cookbook (H100 / H200 / B200 / B300).
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5.
|
||||
|
||||
<MistralMedium35Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Tensor Parallelism**: Mistral Medium 3.5 FP8 (~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300).
|
||||
- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
|
||||
- **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output.
|
||||
- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
|
||||
- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
|
||||
- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
|
||||
- **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6).
|
||||
|
||||
### 3.3 Speculative Decoding (EAGLE)
|
||||
|
||||
Mistral ships an EAGLE draft head, [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE), that lets you run speculative decoding on top of the dense 128B target. The draft is a 2-layer GQA body sharing the target's vocab/head, FP8-quantized like the target (~4 GB), and is meant for low-concurrency latency-bound serving.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path mistralai/Mistral-Medium-3.5-128B \
|
||||
--tp 4 \
|
||||
--dtype bfloat16 \
|
||||
--tool-call-parser mistral \
|
||||
--reasoning-parser mistral \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-draft-model-path mistralai/Mistral-Medium-3.5-128B-EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- **`--dtype bfloat16` is required.** The draft `params.json` does not carry a `dtype` field, so `--dtype auto` falls back to fp32 and downcasts to fp16, which conflicts with the bf16 target when the embed/head are shared. Setting bf16 explicitly keeps both sides aligned (this is a no-op for the target — it already loads as bf16).
|
||||
- The draft uses the same vocab and lm_head as the target. Memory overhead on top of the base model is ~4 GB per TP shard.
|
||||
- `(num-steps, eagle-topk, num-draft-tokens) = (3, 1, 4)` is the recommended starting point. Tune for your workload — wider trees (higher `eagle-topk` / `num-draft-tokens`) help high-acceptance (templated) outputs, narrower trees keep latency tight on more diverse text.
|
||||
- EAGLE shines at low concurrency. At high concurrency, throughput is dominated by the target's batched forward pass and the draft's contribution shrinks; consider running without EAGLE for batch-serving workloads.
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Thinking Mode
|
||||
|
||||
Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
|
||||
],
|
||||
temperature=0.7,
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("Answer:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and
|
||||
division before addition, evaluated left to right.
|
||||
|
||||
17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391.
|
||||
144 / 12 = 12.
|
||||
Finally, 391 + 12 = 403.
|
||||
|
||||
Answer: **17 × 23 + 144 / 12 = 403**
|
||||
|
||||
Step by step:
|
||||
1. 17 × 23 = 391
|
||||
2. 144 / 12 = 12
|
||||
3. 391 + 12 = 403
|
||||
```
|
||||
|
||||
### 4.2 Instruct Mode (Reasoning Off)
|
||||
|
||||
To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is the capital of France?"},
|
||||
],
|
||||
temperature=0.1,
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
The capital of France is **Paris**. It is one of the most famous and visited cities in
|
||||
the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower,
|
||||
Louvre Museum, and Notre-Dame Cathedral.
|
||||
```
|
||||
|
||||
### 4.3 Streaming with Reasoning
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain the difference between async and threading in Python."},
|
||||
],
|
||||
temperature=0.7,
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
print("=== Reasoning ===")
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
elif delta.content:
|
||||
print("\n=== Response ===")
|
||||
print(delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
### 4.4 Tool Calling
|
||||
|
||||
Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
)
|
||||
|
||||
tool_calls = response.choices[0].message.tool_calls
|
||||
for tc in tool_calls:
|
||||
print(f"Tool: {tc.function.name}")
|
||||
print(f"Args: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Paris"}
|
||||
```
|
||||
|
||||
### 4.5 Vision (Image Input)
|
||||
|
||||
Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe what you see in this image."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.7,
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
The image features a stylized representation of the acronym "SGL." The letters
|
||||
are large, bold, and orange with a brown outline, giving them a three-dimensional
|
||||
effect. To the left of the letters, there is a graphic that resembles a neuron
|
||||
or a node with connections, also in a similar orange and brown color scheme. The
|
||||
node has a code symbol (</>) inside a square, suggesting a connection to
|
||||
programming or technology.
|
||||
```
|
||||
|
||||
### 4.6 Loading the Reference System Prompt
|
||||
|
||||
Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly:
|
||||
|
||||
```python Example
|
||||
from datetime import datetime, timedelta
|
||||
from huggingface_hub import hf_hub_download
|
||||
from openai import OpenAI
|
||||
|
||||
MODEL = "mistralai/Mistral-Medium-3.5-128B"
|
||||
|
||||
def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str:
|
||||
path = hf_hub_download(repo_id=repo_id, filename=filename)
|
||||
today = datetime.today().strftime("%Y-%m-%d")
|
||||
yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
|
||||
name = repo_id.split("/")[-1]
|
||||
with open(path) as f:
|
||||
return f.read().format(name=name, today=today, yesterday=yesterday)
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=MODEL,
|
||||
messages=[
|
||||
{"role": "system", "content": load_system_prompt(MODEL)},
|
||||
{"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."},
|
||||
],
|
||||
temperature=0.1,
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmarks
|
||||
|
||||
Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint.
|
||||
|
||||
### 5.1 Accuracy Benchmarks
|
||||
|
||||
#### GSM8K
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.945
|
||||
Invalid: 0.000
|
||||
Latency: 13.594 s
|
||||
Output throughput: 1560.660 token/s
|
||||
```
|
||||
|
||||
#### MMMU
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Overall accuracy: 0.586
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmarks
|
||||
|
||||
#### Latency (Low Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 38.86
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 2684
|
||||
Output token throughput (tok/s): 69.07
|
||||
Mean E2E Latency (ms): 3883.80
|
||||
Median TTFT (ms): 95.90
|
||||
Median TPOT (ms): 14.19
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### Throughput (High Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 117.28
|
||||
Total input tokens: 512842
|
||||
Total generated tokens: 262023
|
||||
Output token throughput (tok/s): 2234.18
|
||||
Total token throughput (tok/s): 6607.01
|
||||
Mean E2E Latency (ms): 11303.79
|
||||
Median TTFT (ms): 152.95
|
||||
Median TPOT (ms): 42.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.3 EAGLE Speculative Decoding (Latency)
|
||||
|
||||
Same 4× H200 setup, EAGLE configuration from [Section 3.3](#3-3-speculative-decoding-eagle). Single-stream latency benchmark (`--max-concurrency 1`).
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 27.64
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 2684
|
||||
Output token throughput (tok/s): 97.10
|
||||
Mean E2E Latency (ms): 2762.99
|
||||
Median TTFT (ms): 90.69
|
||||
Median TPOT (ms): 9.73
|
||||
Accept length: 1.72
|
||||
==================================================
|
||||
```
|
||||
|
||||
EAGLE delivers **~1.41× output throughput and ~29% lower E2E latency** vs. the baseline in [Section 5.2](#5-2-speed-benchmarks) on the same workload. Acceptance length of 1.72 means each draft cycle averages roughly 1.7 accepted tokens.
|
||||
@@ -0,0 +1,393 @@
|
||||
---
|
||||
title: Mistral Small 4
|
||||
metatags:
|
||||
description: "Deploy Mistral Small 4 with SGLang - unified hybrid model combining instruct, reasoning, and agentic capabilities with multimodal support."
|
||||
---
|
||||
|
||||
import { MistralSmall4Deployment } from '/src/snippets/autoregressive/mistral-small-4-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model.
|
||||
|
||||
With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request
|
||||
- **Vision**: Accepts both text and image inputs, providing insights based on visual content
|
||||
- **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities
|
||||
- **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more
|
||||
- **Context Window**: 256K context window
|
||||
- **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters)
|
||||
- **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes
|
||||
- Reasoning effort supported are only **"none" and "high"**
|
||||
|
||||
**Architecture:**
|
||||
|
||||
- Same general architecture as Mistral 3
|
||||
- MoE: 128 experts, 4 active per token
|
||||
- 119B total parameters, 6.5B activated per token
|
||||
- Multimodal input: text + image
|
||||
|
||||
**Models:**
|
||||
|
||||
- **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8)
|
||||
- **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)**
|
||||
- **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603
|
||||
- **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference
|
||||
|
||||
---
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
<Info>
|
||||
Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/install).
|
||||
</Info>
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4.
|
||||
|
||||
<MistralSmall4Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Tensor Parallelism**: Mistral Small 4 FP8 (~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (~60 GB, Blackwell only) runs with tp=1.
|
||||
- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
|
||||
- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
|
||||
- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
|
||||
- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
|
||||
- **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency.
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Thinking Mode
|
||||
|
||||
Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
|
||||
],
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("Answer:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Reasoning: First, I'll break down the problem into two parts: the multiplication and
|
||||
the division. According to the order of operations (PEMDAS/BODMAS), multiplication and
|
||||
division are performed from left to right before addition.
|
||||
|
||||
17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391
|
||||
144 / 12 = 12
|
||||
|
||||
Finally, add the results: 391 + 12 = 403
|
||||
|
||||
Answer: The solution to the problem is as follows:
|
||||
|
||||
1. First, perform the multiplication: 17 × 23.
|
||||
- 17 × 20 = 340
|
||||
- 17 × 3 = 51
|
||||
- 340 + 51 = 391
|
||||
|
||||
2. Then, perform the division: 144 / 12 = 12.
|
||||
|
||||
3. Finally, add the results:
|
||||
- 391 + 12 = 403
|
||||
|
||||
**Answer:** \boxed{403}
|
||||
```
|
||||
|
||||
### 4.2 Instruct Mode (Reasoning Off)
|
||||
|
||||
To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to reverse a string."},
|
||||
],
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
````text Output
|
||||
# Python Function to Reverse a String
|
||||
|
||||
Here are several ways to write a Python function to reverse a string:
|
||||
|
||||
## Method 1: Using String Slicing (Most Pythonic)
|
||||
```python
|
||||
def reverse_string(s):
|
||||
"""Reverse a string using slicing."""
|
||||
return s[::-1]
|
||||
```
|
||||
|
||||
## Method 2: Using a Loop
|
||||
```python Example
|
||||
def reverse_string(s):
|
||||
"""Reverse a string using a loop."""
|
||||
reversed_str = ""
|
||||
for char in s:
|
||||
reversed_str = char + reversed_str
|
||||
return reversed_str
|
||||
```
|
||||
|
||||
## Method 3: Using reversed() function
|
||||
```python Example
|
||||
def reverse_string(s):
|
||||
"""Reverse a string using reversed() function."""
|
||||
return ''.join(reversed(s))
|
||||
```
|
||||
|
||||
The first method using string slicing (`s[::-1]`) is generally the most efficient and
|
||||
recommended approach in Python.
|
||||
|
||||
Example usage:
|
||||
```python Example
|
||||
original = "Hello, World!"
|
||||
reversed_str = reverse_string(original)
|
||||
print(reversed_str) # Output: "!dlroW ,olleH"
|
||||
```
|
||||
````
|
||||
|
||||
### 4.3 Streaming with Reasoning
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain the difference between async and threading in Python."},
|
||||
],
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
print("=== Reasoning ===")
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
elif delta.content:
|
||||
print("\n=== Response ===")
|
||||
print(delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
=== Reasoning ===
|
||||
Okay, the user is asking about the difference between async and threading in Python.
|
||||
I need to break this down clearly, covering the key aspects of both, like their
|
||||
purposes, performance characteristics, and use cases...
|
||||
=== Response ===
|
||||
In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency
|
||||
models, each suited for specific use cases. Here's a breakdown of their key differences:
|
||||
|
||||
### 1. Model of Concurrency
|
||||
- **Threading**: Based on preemptive multitasking using OS threads.
|
||||
- **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield...
|
||||
```
|
||||
|
||||
### 4.4 Tool Calling
|
||||
|
||||
Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
)
|
||||
|
||||
tool_calls = response.choices[0].message.tool_calls
|
||||
for tc in tool_calls:
|
||||
print(f"Tool: {tc.function.name}")
|
||||
print(f"Args: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Paris"}
|
||||
```
|
||||
|
||||
### 4.5 Vision (Image Input)
|
||||
|
||||
Mistral Small 4 accepts image inputs alongside text:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe what you see in this image."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
The image is a copyright symbol, represented by a stylized version of the lowercase
|
||||
letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and
|
||||
the circle is orange. The design is simple yet striking, using oval and elliptical
|
||||
shapes to create a distinct symbol which signifies copyright protection.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmarks
|
||||
|
||||
### 5.1 Accuracy Benchmarks
|
||||
|
||||
#### GSM8K
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
|
||||
#### MMLU
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmarks
|
||||
|
||||
#### Latency (Low Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
|
||||
#### Throughput (High Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,556 @@
|
||||
---
|
||||
title: Kimi-K2.7-Code
|
||||
description: "Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input."
|
||||
metatags:
|
||||
description: "Deploy Kimi-K2.7-Code native multimodal agentic model with SGLang - reasoning, tool calling, and multimodal capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused agentic model by Moonshot AI, built on top of Kimi-K2.6. It improves real-world long-horizon coding task completion while reducing thinking-token usage by approximately 30% compared with Kimi-K2.6.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Coding-Focused Agentic Model**: Optimized for end-to-end coding workflows and complex software engineering tasks.
|
||||
- **Token Efficiency**: Reduces thinking-token usage by approximately 30% versus Kimi-K2.6.
|
||||
- **K2.6-Compatible Deployment**: Shares the same architecture as Kimi-K2.5/Kimi-K2.6, so the SGLang deployment method can be reused with the new model ID.
|
||||
- **Native Multimodality**: Shares Kimi-K2.6's native multimodal architecture with a MoonViT vision encoder (400M parameters) and supports image and video (experimental) input.
|
||||
|
||||
**Benchmarks:**
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Benchmark</th>
|
||||
<th>Kimi-K2.6</th>
|
||||
<th>Kimi-K2.7-Code</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Kimi Code Bench v2</td>
|
||||
<td>50.9</td>
|
||||
<td>62.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Program Bench</td>
|
||||
<td>48.3</td>
|
||||
<td>53.6</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MLS Bench Lite</td>
|
||||
<td>26.7</td>
|
||||
<td>35.1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Kimi Claw 24/7 Bench</td>
|
||||
<td>42.9</td>
|
||||
<td>46.9</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MCP Atlas</td>
|
||||
<td>69.4</td>
|
||||
<td>76.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MCP Mark Verified</td>
|
||||
<td>72.8</td>
|
||||
<td>81.1</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended Generation Parameters:**
|
||||
- Thinking Mode: `temperature=1.0`, `top_p=0.95`
|
||||
- Kimi-K2.7-Code forces thinking and preserve-thinking behavior; instant mode is not supported.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code)
|
||||
|
||||
**License:** Modified MIT for the native checkpoint.
|
||||
|
||||
For details, see the [official model card](https://huggingface.co/moonshotai/Kimi-K2.7-Code).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](/docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities.
|
||||
|
||||
import { KimiK27CodeDeployment } from '/src/snippets/autoregressive/kimi-k27-code-deployment.jsx'
|
||||
|
||||
<KimiK27CodeDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory.
|
||||
- **Context Length**: The model supports a 256K context length. Use a shorter `--context-length` when you need to reserve memory for larger batches.
|
||||
- **Transformers Version**: The model card requires `transformers>=4.57.1,<5.0.0`.
|
||||
- **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.7-Code has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
|
||||
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X.
|
||||
- **DP Attention**: Enable with `--dp <N> --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required.
|
||||
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
|
||||
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
|
||||
- **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
See [Basic API Usage](/docs/basic_usage/send_request).
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multimodal (Vision + Text) Input
|
||||
|
||||
Kimi-K2.7-Code supports native multimodal input with images:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2.7-Code",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "What is in this image? Describe it in detail."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown:
|
||||
|
||||
## Header
|
||||
- At the top left is the Auntie Anne's logo (a pretzel with a halo)
|
||||
- The store name "**Auntie Anne's**" is printed prominently at the top
|
||||
- Some text below the store name appears blurred/redacted (likely store location, address, or transaction details)
|
||||
|
||||
## Purchase Details
|
||||
- **Item**: CINNAMON SUGAR
|
||||
- **Quantity & Price**: 1 × 17,000
|
||||
- **Item Total**: 17,000
|
||||
|
||||
## Financial Summary
|
||||
- **SUB TOTAL**: 17,000
|
||||
- **GRAND TOTAL**: 17,000
|
||||
- **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah)
|
||||
- **CHANGE DUE**: 3,000
|
||||
|
||||
## Physical Description
|
||||
- The receipt is printed on white thermal paper
|
||||
- Some information in the middle section and toward the bottom is intentionally blurred/obscured
|
||||
- The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter)
|
||||
|
||||
The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash.
|
||||
```
|
||||
|
||||
#### 4.2.2 Reasoning Output
|
||||
|
||||
Kimi-K2.7-Code forces thinking mode and preserve-thinking behavior.
|
||||
|
||||
**Thinking Mode (default)** — reasoning content is automatically separated:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2.7-Code",
|
||||
messages=[
|
||||
{"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
|
||||
]
|
||||
)
|
||||
|
||||
print("====== Reasoning Content (Thinking Mode) ======")
|
||||
print(response.choices[0].message.reasoning_content)
|
||||
print("====== Response (Thinking Mode) ======")
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
====== Reasoning Content (Thinking Mode) ======
|
||||
The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion.
|
||||
|
||||
Let me think carefully:
|
||||
- 9.11 means 9 + 11/100 = 9.11
|
||||
- 9.9 means 9 + 9/10 = 9.90
|
||||
|
||||
So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11.
|
||||
|
||||
The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9).
|
||||
|
||||
So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79.
|
||||
|
||||
I should explain this clearly and address the common misconception.
|
||||
====== Response (Thinking Mode) ======
|
||||
Mathematically, **9.9 is bigger**.
|
||||
|
||||
Here's why:
|
||||
|
||||
**9.9 = 9.90**
|
||||
|
||||
When comparing decimals, you need to look at the same place values:
|
||||
- 9.11 = 9 ones, 1 tenth, and 1 hundredth
|
||||
- 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90)
|
||||
|
||||
Since **0.90 > 0.11**, it follows that **9.9 > 9.11**.
|
||||
|
||||
The difference is:
|
||||
9.9 - 9.11 = 0.79
|
||||
|
||||
**Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger.
|
||||
```
|
||||
|
||||
#### 4.2.3 Preserve Thinking
|
||||
|
||||
Kimi-K2.7-Code keeps reasoning content across multi-turn interactions. This behavior is enabled by default and cannot be disabled.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Tell me three random numbers."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.",
|
||||
"content": "473, 921, 235"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What are the other two numbers you have in mind?"
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2.7-Code",
|
||||
messages=messages,
|
||||
stream=False,
|
||||
max_tokens=4096,
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
Some OpenAI-compatible deployments use `reasoning` instead of `reasoning_content` in assistant messages. Use the field your serving stack exposes.
|
||||
|
||||
#### 4.2.4 Tool Calling
|
||||
|
||||
Kimi-K2.7-Code supports tool calling capabilities for agentic tasks:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2.7-Code",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {'name': None, 'arguments': ''}
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": "The weather in Beijing is 22°C and sunny."
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2.7-Code",
|
||||
messages=messages
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
The weather in Beijing is currently **22°C and sunny**. ☀️
|
||||
|
||||
It's a nice, warm day there—great for being outdoors!
|
||||
```
|
||||
|
||||
#### 4.2.5 Multimodal + Tool Calling (Agentic Vision)
|
||||
|
||||
Combine vision understanding with tool calling for advanced agentic tasks:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search_product",
|
||||
"description": "Search for a product by name or description",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The product name or description to search for"
|
||||
}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2.7-Code",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Can you identify this product and search for similar items?"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
|
||||
# Print reasoning process
|
||||
if msg.reasoning_content:
|
||||
print("=== Reasoning ===")
|
||||
print(msg.reasoning_content)
|
||||
|
||||
# Print response content
|
||||
if msg.content:
|
||||
print("=== Content ===")
|
||||
print(msg.content)
|
||||
|
||||
# Print tool calls
|
||||
if msg.tool_calls:
|
||||
print("=== Tool Calls ===")
|
||||
for tc in msg.tool_calls:
|
||||
print(f" Function: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=== Reasoning ===
|
||||
The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain.
|
||||
|
||||
I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items.
|
||||
=== Content ===
|
||||
Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR).
|
||||
|
||||
Let me search for this product and similar items for you.
|
||||
=== Tool Calls ===
|
||||
Function: search_product
|
||||
Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"}
|
||||
```
|
||||
|
||||
#### 4.2.6 Deployment Command Example
|
||||
|
||||
Deploy Kimi-K2.7-Code with the following command (H200/B300, reasoning and tool parsing enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path moonshotai/Kimi-K2.7-Code \
|
||||
--tp 8 \
|
||||
--reasoning-parser kimi_k2 \
|
||||
--tool-call-parser kimi_k2 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
For GB300, use `--tp 4`.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
The following results are from the official Kimi-K2.7-Code model card. They were evaluated with thinking mode enabled through Kimi Code CLI at `temperature=1.0`, `top_p=0.95`, and a 262,144-token context length unless otherwise stated.
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Category</th>
|
||||
<th>Benchmark</th>
|
||||
<th>Kimi-K2.6</th>
|
||||
<th>Kimi-K2.7-Code</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Coding</td>
|
||||
<td>Kimi Code Bench v2</td>
|
||||
<td>50.9</td>
|
||||
<td>62.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Coding</td>
|
||||
<td>Program Bench</td>
|
||||
<td>48.3</td>
|
||||
<td>53.6</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Coding</td>
|
||||
<td>MLS Bench Lite</td>
|
||||
<td>26.7</td>
|
||||
<td>35.1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Agentic</td>
|
||||
<td>Kimi Claw 24/7 Bench</td>
|
||||
<td>42.9</td>
|
||||
<td>46.9</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Agentic</td>
|
||||
<td>MCP Atlas</td>
|
||||
<td>69.4</td>
|
||||
<td>76.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Agentic</td>
|
||||
<td>MCP Mark Verified</td>
|
||||
<td>72.8</td>
|
||||
<td>81.1</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,520 @@
|
||||
---
|
||||
title: Kimi-K2
|
||||
metatags:
|
||||
description: "Deploy Kimi-K2 MoE model with SGLang - 1T total parameters, 32B active, step-by-step reasoning and tool calling capabilities."
|
||||
---
|
||||
|
||||
import { KimiK2Deployment } from '/src/snippets/autoregressive/kimi-k2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters.
|
||||
|
||||
**Model Variants:**
|
||||
|
||||
- **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM.
|
||||
- **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use.
|
||||
- **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified).
|
||||
|
||||
For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities.
|
||||
|
||||
<KimiK2Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory.
|
||||
- **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism).
|
||||
- **Data Parallel (DP)**: Enable with `--dp 4 --enable-dp-attention` for production throughput.
|
||||
- **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+).
|
||||
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content.
|
||||
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
|
||||
- **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
See [Basic API Usage](../../../docs/get-started/quickstart).
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
Enable reasoning parser for Kimi-K2-Thinking:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model moonshotai/Kimi-K2-Thinking \
|
||||
--reasoning-parser kimi_k2 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step.
|
||||
|
||||
Step 1: Understand what "percent" means.
|
||||
- "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15.
|
||||
|
||||
Step 2: Convert the percentage to a decimal.
|
||||
- 15% = 15 / 100 = 0.15
|
||||
|
||||
Step 3: Multiply the decimal by the number.
|
||||
- 0.15 * 240
|
||||
|
||||
Step 4: Perform the multiplication.
|
||||
- 0.15 * 240 = (15/100) * 240
|
||||
- = 15 * 240 / 100
|
||||
- = 3600 / 100
|
||||
- = 36
|
||||
|
||||
Alternatively, I can calculate it directly:
|
||||
- 0.15 * 240
|
||||
- 15 * 240 = 3600
|
||||
- 3600 / 100 = 36
|
||||
|
||||
Or, break it down:
|
||||
- 10% of 240 = 24
|
||||
- 5% of 240 = half of 10% = 12
|
||||
- 15% of 240 = 10% + 5% = 24 + 12 = 36
|
||||
|
||||
I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying.
|
||||
|
||||
Let me structure the answer:
|
||||
1. Convert the percentage to a decimal.
|
||||
2. Multiply the decimal by the number.
|
||||
3. Show the calculation.
|
||||
4. State the final answer.
|
||||
|
||||
This is simple and easy to follow.
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal**
|
||||
15% means 15 per 100, which is 15 ÷ 100 = **0.15**
|
||||
|
||||
**Step 2: Multiply the decimal by the number**
|
||||
0.15 × 240
|
||||
|
||||
**Step 3: Calculate the result**
|
||||
0.15 × 240 = **36**
|
||||
|
||||
**Answer:** 15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment:
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model moonshotai/Kimi-K2-Instruct \
|
||||
--tool-call-parser kimi_k2 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you.
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location":"Beijing"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K2-Thinking",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x)
|
||||
- Model: Kimi-K2-Instruct
|
||||
- sglang version: 0.5.6.post1
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-K2-Instruct \
|
||||
--tp 8 \
|
||||
--dp 4 \
|
||||
--enable-dp-attention \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model moonshotai/Kimi-K2-Instruct\
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 44.93
|
||||
Total input tokens: 1951
|
||||
Total input text tokens: 1951
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2755
|
||||
Total generated tokens (retokenized): 2748
|
||||
Request throughput (req/s): 0.22
|
||||
Input token throughput (tok/s): 43.42
|
||||
Output token throughput (tok/s): 61.32
|
||||
Peak output token throughput (tok/s): 64.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 104.74
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4489.56
|
||||
Median E2E Latency (ms): 4994.53
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 141.22
|
||||
Median TTFT (ms): 158.28
|
||||
P99 TTFT (ms): 166.90
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.40
|
||||
Median TPOT (ms): 15.63
|
||||
P99 TPOT (ms): 39.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.78
|
||||
Median ITL (ms): 15.76
|
||||
P95 ITL (ms): 16.36
|
||||
P99 ITL (ms): 16.59
|
||||
Max ITL (ms): 19.94
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-K2-Instruct \
|
||||
--tp 8 \
|
||||
--dp 4 \
|
||||
--ep 4 \
|
||||
--enable-dp-attention \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model moonshotai/Kimi-K2-Instruct\
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 174.11
|
||||
Total input tokens: 296642
|
||||
Total input text tokens: 296642
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 193831
|
||||
Total generated tokens (retokenized): 168687
|
||||
Request throughput (req/s): 5.74
|
||||
Input token throughput (tok/s): 1703.73
|
||||
Output token throughput (tok/s): 1113.25
|
||||
Peak output token throughput (tok/s): 2383.00
|
||||
Peak concurrent requests: 112
|
||||
Total token throughput (tok/s): 2816.97
|
||||
Concurrency: 89.60
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 15601.09
|
||||
Median E2E Latency (ms): 10780.52
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 457.42
|
||||
Median TTFT (ms): 221.62
|
||||
P99 TTFT (ms): 2475.32
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 97.23
|
||||
Median TPOT (ms): 85.61
|
||||
P99 TPOT (ms): 435.95
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 78.61
|
||||
Median ITL (ms): 43.66
|
||||
P95 ITL (ms): 169.53
|
||||
P99 ITL (ms): 260.91
|
||||
Max ITL (ms): 1703.21
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Server Command
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-K2-Instruct \
|
||||
--tp 8 \
|
||||
--dp 4 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
|
||||
```
|
||||
|
||||
- **Result**:
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 15.956 s
|
||||
Output throughput: 1231.699 token/s
|
||||
```
|
||||
@@ -0,0 +1,416 @@
|
||||
---
|
||||
title: Kimi-K3
|
||||
description: "Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA and AMD recipes."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:kimi-k3 # CUDA13
|
||||
docker pull lmsysorg/sglang:kimi-k3-cu12 # CUDA12
|
||||
docker pull lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727 # ROCM
|
||||
```
|
||||
|
||||
These tags publish with the public K3 launch; until then, build from the Dockerfiles linked below.
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
If you do not want to use a Docker image, reproduce the dependency installation steps from the [CUDA 13 Dockerfile](https://github.com/sgl-project/sglang/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu13.Dockerfile) or [CUDA 12 Dockerfile](https://github.com/sgl-project/sglang/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu12.Dockerfile).
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8), so it is not a separate choice.
|
||||
|
||||
**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)).
|
||||
|
||||
**Strategy** — the operating point within that shape:
|
||||
|
||||
- **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP.
|
||||
- **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x.
|
||||
- **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`).
|
||||
|
||||
`Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`.
|
||||
|
||||
**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH has no published draft checkpoint. The win is largest on short interactive traffic and fades as the prompt grows.
|
||||
|
||||
<Note>
|
||||
`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command.
|
||||
</Note>
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/moonshotai/kimi-k3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx";
|
||||
import { KimiK3MambaRatioCalculator } from "/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
### Mamba ratio calculator
|
||||
|
||||
<KimiK3MambaRatioCalculator />
|
||||
|
||||
<Accordion title="How --mamba-full-memory-ratio is calculated">
|
||||
|
||||
`--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio:
|
||||
|
||||
```text
|
||||
ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes))
|
||||
```
|
||||
|
||||
- `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two.
|
||||
- `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`.
|
||||
- `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype.
|
||||
- `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8.
|
||||
- `L` — average total request length in tokens: input + output.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<a id="playground" style={{ scrollMarginTop: "96px" }} />
|
||||
|
||||
## Advanced Features Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput.
|
||||
|
||||
K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`).
|
||||
|
||||
<Note>
|
||||
Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights
|
||||
are scheduled to release by July 27, 2026**. The recipes on this page were validated on the public
|
||||
[`sgl-project/sglang` `kimi-k3` branch](https://github.com/sgl-project/sglang/tree/kimi-k3) — the HuggingFace
|
||||
repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be
|
||||
available at launch.
|
||||
|
||||
Every cell in the Deploy panel above is currently marked **Final Verification In Progress**: the
|
||||
recipe runs, but its serving round on the final weights and current code is still open. Re-measure
|
||||
throughput and accuracy before you rely on any of them.
|
||||
</Note>
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code).
|
||||
|
||||
**Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side).
|
||||
|
||||
Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload:
|
||||
|
||||
| Lever | Effect |
|
||||
|---|---|
|
||||
| `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 |
|
||||
| `--mamba-ssm-dtype bfloat16` | ~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton |
|
||||
| `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect |
|
||||
| `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` |
|
||||
| `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) |
|
||||
|
||||
Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise).
|
||||
|
||||
**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell and it resolves to FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on — on an install without the pool, drop the flag to fall back to Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile:
|
||||
|
||||
```bash
|
||||
wget https://github.com/sgl-project/whl/releases/download/trtllm_gen_moe_cubin_20260617/trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip
|
||||
sudo mkdir -p /opt/trtllm_gen_moe_cubin_pool
|
||||
sudo unzip -q trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip -d /opt/trtllm_gen_moe_cubin_pool
|
||||
export SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/opt/trtllm_gen_moe_cubin_pool/trtllm_gen_moe_cubin_pool_20260617_v0613rc1
|
||||
```
|
||||
|
||||
Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minutes, cached).
|
||||
|
||||
**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode.
|
||||
|
||||
**Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV.
|
||||
|
||||
**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. No serving round on the final draft checkpoint has landed — measure against the same recipe running NOSPEC before adopting.
|
||||
|
||||
**Per-platform notes:**
|
||||
|
||||
| Platform | Topology | Notes |
|
||||
|---|---|---|
|
||||
| B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced |
|
||||
| GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected |
|
||||
| B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy |
|
||||
| GB200 4×4 | TP16/DCP16 | MNNVL auto-detected |
|
||||
| H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` |
|
||||
| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) |
|
||||
| MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention, graph bs up to 256; DSPARK supported |
|
||||
|
||||
**DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles:
|
||||
|
||||
- DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it.
|
||||
- Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300).
|
||||
- No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness).
|
||||
- Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype.
|
||||
- Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context.
|
||||
- Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1` or `--moe-a2a-backend` is set.
|
||||
|
||||
No cell has a serving round in this exact shape — treat them as starting points to verify.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`).
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K3",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
reasoning_effort="high", # "low" | "high" | "max" (default max)
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Pending update...
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
resp = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K3",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Pending update...
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads.
|
||||
|
||||
- On the DCP recipes (Blackwell Balanced / High-Throughput, in both the `Unified` and `Decode` roles), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 with Spec Decode off keeps DCP. Only DCP goes: the MLA KV reverts to TP-replicated, but the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP while B200 Unified keeps its `--pp-size 2` / `--ep-size`.
|
||||
- Low-Latency and the Hopper recipes take all tiers unchanged.
|
||||
|
||||
<a id="pd-disaggregation" />
|
||||
|
||||
### 3.4 PD Disaggregation
|
||||
|
||||
PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state.
|
||||
|
||||
- **Transfer**: the cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground.
|
||||
- **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers.
|
||||
- **Decode state pool**: chunk cache — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above.
|
||||
|
||||
#### Deep PP for prefill
|
||||
|
||||
Deep PP is `--tp-size 1` with one pipeline stage per GPU — `--pp-size 8` on B300/GB300, `--pp-size 16` on B200/GB200. Pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). `--tp-size 1` is also what buys context: above TP1 the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16 for the same memory.
|
||||
|
||||
- Use one stage per GPU; a shallow split still pays the in-stage all-reduce and can lose to flat TP.
|
||||
- Pays only with several requests in flight. On the 8-GPU platforms that is why `Default` stays TP8; on the 16-GPU platforms deep PP wins at the Default operating point too, so both strategies use it — measured on GB200 at ISL 8192 / concurrency 32, PP16 × TP1 reached 4550 prefill tok/s/GPU vs 3596 (PP8 × TP2), 2407 (TEP16), and 1652 (TP16). Below concurrency ~8 the pipeline cannot fill and TEP16 leads instead (1947 vs 1227) — use `--tp-size 16 --ep-size 16` there.
|
||||
- DSPARK off (`pp_size == 1` required) — on B200/GB200 that applies to `Default` as well.
|
||||
- Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side.
|
||||
|
||||
<Accordion title="Router">
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://<prefill-host>:30000 8998 \
|
||||
--decode http://<decode-host>:30100 \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--disable-circuit-breaker \
|
||||
--health-check-interval-secs 999999
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
Clients then send requests to the router (`:8000`) instead of an individual role server.
|
||||
|
||||
### 3.5 VLM Serving Profiles
|
||||
|
||||
The open-source K3 serving contract currently supports **image input only** — its
|
||||
processor rejects video and audio input.
|
||||
|
||||
#### Recommended high-speed VLM
|
||||
|
||||
The command panel now opens on the **B300 · Unified · Balanced**
|
||||
recipe below. It makes the VLM-specific performance choices explicit:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path moonshotai/Kimi-K3 \
|
||||
--tp-size 8 \
|
||||
--dcp-size 8 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--mm-feature-transport cuda_ipc \
|
||||
--mm-processor-worker-num 2 \
|
||||
--mm-io-worker-num 16 \
|
||||
--reasoning-parser kimi_k3 \
|
||||
--tool-call-parser kimi_k3 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- `--mm-feature-transport cuda_ipc` — single-node only: skips the CPU round trip, bounded pool (per-tensor CPU fallback when full), reserves up to `SGLANG_MM_FEATURE_CACHE_MB` on the base GPU. Multi-node recipes use CPU transport.
|
||||
- 2 processor / 16 I/O workers are the measured defaults; more adds contention.
|
||||
- Leave `--mm-attention-backend` unset — auto-selected, with a correctness fallback.
|
||||
- Don't add `--mm-enable-dp-encoder`; K3 already shards images across TP ranks.
|
||||
|
||||
#### VLM compatibility
|
||||
|
||||
| Feature | K3 behavior |
|
||||
|---|---|
|
||||
| PD | Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in [PD disaggregation](#pd-disaggregation). |
|
||||
| EPD | Supported on the public `kimi-k3` branch. Use an `--encoder-only` vision role and a `--language-only` prefill role; add the normal decode role for full EPD. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation). |
|
||||
| MM encoder DP | Built in. K3 shards complete images across TP ranks, so leave `--mm-enable-dp-encoder` unset in unified, PD-prefill, and encoder-only roles. |
|
||||
| CUDA IPC | Compatible with the local processor-to-scheduler path on a single-node unified or PD-prefill role. It does not replace `--encoder-transfer-backend` for EPD or the PD KV/KDA transfer, and its bounded pool consumes HBM. |
|
||||
| ViT BCG | Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. |
|
||||
|
||||
#### Should ViT BCG be enabled?
|
||||
|
||||
Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM.
|
||||
|
||||
- The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving.
|
||||
- Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes.
|
||||
- The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring.
|
||||
|
||||
#### Low-HBM VLM
|
||||
|
||||
Use this profile when keeping HBM headroom matters more than peak concurrency.
|
||||
It removes the 1 GiB CUDA IPC pool, keeps ViT BCG disabled, halves the context
|
||||
window, caps concurrency, and lowers the static-memory target:
|
||||
|
||||
```bash Command
|
||||
SGLANG_VIT_ENABLE_CUDA_GRAPH=0 \
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path moonshotai/Kimi-K3 \
|
||||
--tp-size 8 \
|
||||
--context-length 65536 \
|
||||
--enable-symm-mem \
|
||||
--mem-fraction-static 0.82 \
|
||||
--mm-feature-transport cpu \
|
||||
--mm-processor-worker-num 2 \
|
||||
--mm-io-worker-num 16 \
|
||||
--reasoning-parser kimi_k3 \
|
||||
--tool-call-parser kimi_k3 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
`--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated.
|
||||
|
||||
<a id="large-scale-presets" />
|
||||
|
||||
### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell)
|
||||
|
||||
**The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP).
|
||||
|
||||
Two presets come out of this, at `N = 8k` GPUs:
|
||||
|
||||
| Preset | What it trades | Pick it for |
|
||||
|---|---|---|
|
||||
| **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. |
|
||||
| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | Context ≥ ~16K, or per-replica concurrency past 128. |
|
||||
|
||||
- **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5.
|
||||
- The fully data-parallel extreme (`--dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape.
|
||||
|
||||
The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`:
|
||||
|
||||
```bash Command
|
||||
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path moonshotai/Kimi-K3 \
|
||||
--tp-size 32 --ep-size 32 \
|
||||
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
|
||||
--nnodes 4 --node-rank <rank> --dist-init-addr <node0-ip>:20000 \
|
||||
--moe-a2a-backend megamoe --moe-runner-backend deep_gemm \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
--mamba-radix-cache-strategy extra_buffer_lazy \
|
||||
--mem-fraction-static 0.92 \
|
||||
--reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation:
|
||||
|
||||
| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--dp-size` |
|
||||
|---|---|---|---|---|
|
||||
| 16 | 2×8 | 4×4 | 16 | 2 |
|
||||
| 32 | 4×8 | 8×4 | 32 | 4 |
|
||||
| 64 | 8×8 | 16×4 | 64 | 8 |
|
||||
|
||||
For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator).
|
||||
|
||||
Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing.
|
||||
|
||||
Decisions the preset already makes:
|
||||
|
||||
- **MegaMoE on `deep_gemm`** — the fastest a2a backend; needs the SiTU cubin pool ([§2](#2-configuration-tips)).
|
||||
- **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not.
|
||||
- **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales.
|
||||
|
||||
<Note>
|
||||
No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet.
|
||||
</Note>
|
||||
@@ -0,0 +1,297 @@
|
||||
---
|
||||
title: Kimi-Linear
|
||||
metatags:
|
||||
description: "Deploy Kimi-Linear with SGLang - community contribution guide for Moonshot AI's Kimi-Linear model deployment."
|
||||
---
|
||||
|
||||
import { KimiLinearDeployment } from '/src/snippets/autoregressive/kimi-linear-deployment.jsx';
|
||||
|
||||
## AMD GPU Support
|
||||
|
||||
## 1. Model Introduction
|
||||
Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating.
|
||||
Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention.
|
||||
Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons.
|
||||
High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT).
|
||||
|
||||
For more details, please refer to the [official Kimi Linear GitHub Repository]: https://github.com/MoonshotAI/Kimi-Linear
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<KimiLinearDeployment />
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Kimi-linear \
|
||||
lmsysorg/sglang:v0.5.7-rocm700-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 pre-installation steps inside the docker
|
||||
|
||||
```shell Command
|
||||
pip install sentencepiece tiktoken
|
||||
```
|
||||
|
||||
#### 4.2.3 Launch the server
|
||||
```shell Command
|
||||
export SGLANG_ROCM_FUSED_DECODE_MLA=0
|
||||
|
||||
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--tp 4 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
### 5.1 Speed Benchmark
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300X GPU
|
||||
|
||||
Model: Kimi-Linear-48B-A3B-Instruct
|
||||
|
||||
Tensor Parallelism: 4
|
||||
|
||||
sglang version: 0.5.7
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--tp 4 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.1.1 Low Concurrency (Latency-Optimized)
|
||||
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 23.86
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4001
|
||||
Request throughput (req/s): 0.42
|
||||
Input token throughput (tok/s): 255.70
|
||||
Output token throughput (tok/s): 176.86
|
||||
Peak output token throughput (tok/s): 190.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 432.56
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2383.93
|
||||
Median E2E Latency (ms): 1911.63
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 141.33
|
||||
Median TTFT (ms): 126.27
|
||||
P99 TTFT (ms): 294.76
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.32
|
||||
Median TPOT (ms): 5.33
|
||||
P99 TPOT (ms): 5.36
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.33
|
||||
Median ITL (ms): 5.32
|
||||
P95 ITL (ms): 5.44
|
||||
P99 ITL (ms): 5.58
|
||||
Max ITL (ms): 11.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 31.38
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 39667
|
||||
Request throughput (req/s): 2.55
|
||||
Input token throughput (tok/s): 1264.13
|
||||
Output token throughput (tok/s): 1300.37
|
||||
Peak output token throughput (tok/s): 1801.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2564.50
|
||||
Concurrency: 14.13
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5543.18
|
||||
Median E2E Latency (ms): 5755.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 175.25
|
||||
Median TTFT (ms): 137.87
|
||||
P99 TTFT (ms): 292.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.75
|
||||
Median TPOT (ms): 10.87
|
||||
P99 TPOT (ms): 16.74
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.54
|
||||
Median ITL (ms): 7.95
|
||||
P95 ITL (ms): 13.68
|
||||
P99 ITL (ms): 116.80
|
||||
Max ITL (ms): 299.89
|
||||
==================================================
|
||||
|
||||
```
|
||||
|
||||
### 5.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 79.71
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 228448
|
||||
Request throughput (req/s): 6.27
|
||||
Input token throughput (tok/s): 3134.20
|
||||
Output token throughput (tok/s): 3169.72
|
||||
Peak output token throughput (tok/s): 6109.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 6303.92
|
||||
Concurrency: 94.80
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 15113.92
|
||||
Median E2E Latency (ms): 13851.52
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 564.46
|
||||
Median TTFT (ms): 226.04
|
||||
P99 TTFT (ms): 2683.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 29.63
|
||||
Median TPOT (ms): 31.28
|
||||
P99 TPOT (ms): 38.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 28.85
|
||||
Median ITL (ms): 16.29
|
||||
P95 ITL (ms): 123.42
|
||||
P99 ITL (ms): 157.80
|
||||
Max ITL (ms): 2481.11
|
||||
==================================================
|
||||
```
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Server Command
|
||||
|
||||
```shell Command
|
||||
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
|
||||
--tp 4 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
- **Result**:
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.705
|
||||
Invalid: 0.000
|
||||
Latency: 11.855 s
|
||||
Output throughput: 3224.982 token/s
|
||||
```
|
||||
@@ -0,0 +1,654 @@
|
||||
---
|
||||
title: Nemotron 3 Nano Omni
|
||||
metatags:
|
||||
description: "Deploy NVIDIA Nemotron 3 Nano Omni multimodal MoE model with SGLang - text, image, video, and audio inputs with reasoning and tool calling."
|
||||
---
|
||||
|
||||
import { Nemotron3NanoOmniDeployment } from '/src/snippets/autoregressive/nemotron3-nano-omni-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
`NVIDIA Nemotron 3 Nano Omni` is a 30B-parameter hybrid MoE multimodal model that activates only 3B parameters per forward pass, combining vision and audio encoders into a unified architecture. Part of the Nemotron 3 family, it is designed to power multimodal sub-agents that perceive and reason across vision, audio, and language in a single inference loop — eliminating the fragmented stacks of separate models for each modality.
|
||||
|
||||
Architecture and key features:
|
||||
|
||||
- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture for efficient routing and sequence modeling.
|
||||
- **30B total / 3B active parameters:** Delivers strong multimodal accuracy at a fraction of the cost of dense models.
|
||||
- **1M token context window:** Sustains coherent agent state across extended multimodal workflows — screen history, document content, and audio context remain in view without re-ingestion.
|
||||
- **Unified vision and audio encoders:** One model replaces fragmented multimodal stacks; vision and audio perception happen in the same forward pass.
|
||||
- **3D Convolution (Conv3D):** Efficient temporal-spatial processing for video inputs.
|
||||
- **Efficient Video Sampling (EVS):** Enables longer video processing at the same compute budget via temporal-aware perception and adaptive frame sampling.
|
||||
- **FP8 and NVFP4 quantization:** FP8 supports deployment from workstation (RTX 6000, DGX Spark) to cloud (H100, H200, B200, A100, L40S); NVFP4 requires Blackwell hardware.
|
||||
- **9x higher throughput** than other open omni models at the same interactivity level.
|
||||
- **~20% higher multimodal intelligence** compared to the best open alternative.
|
||||
- **Post-trained with multi-environment reinforcement learning** via NVIDIA NeMo RL and NeMo Gym across text, image, audio, and video environments, improving instruction following and convergence to correct multimodal answers.
|
||||
|
||||
**Modalities:** Input: text, image, video, audio — Output: text
|
||||
|
||||
**Supported GPUs:** NVIDIA B200, H100, H200, A100, L40S, DGX Spark, RTX 6000
|
||||
|
||||
Available model variants on HuggingFace:
|
||||
- [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16)
|
||||
- [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8)
|
||||
- [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4)
|
||||
|
||||
**Agentic workloads this model enables:**
|
||||
- **Computer Use Agent:** Perception loop for agents navigating GUIs — reads screens, understands UI state over time, validates outcomes. Collapses vision and reasoning into a single loop.
|
||||
- **Document Intelligence:** Interprets documents, charts, tables, screenshots, and mixed media inputs for enterprise analysis and compliance workflows.
|
||||
- **Audio & Video Understanding Agents:** Maintains continuous audio-video context for customer service, research, and monitoring workflows, tying what was said, shown, and documented into a single reasoning stream.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Install SGLang via pip or from source:
|
||||
|
||||
```shell Command
|
||||
# Install via pip
|
||||
pip install sglang
|
||||
|
||||
# Or install from source
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
|
||||
# Or use Docker
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For the full Docker setup and other installation methods, refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance tuning.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
|
||||
|
||||
<Nemotron3NanoOmniDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Attention backend:**
|
||||
|
||||
**H100/H200:** Use flash attention 3 backend by default.
|
||||
**B200:** Use flashinfer backend by default.
|
||||
|
||||
- **TP support:**
|
||||
|
||||
To set tensor parallelism, use `--tp <1|2|4|8>`. A 4×H100 setup is recommended for the BF16 variant.
|
||||
|
||||
- **FP8 KV cache:**
|
||||
|
||||
To enable FP8 KV cache, append `--kv-cache-dtype fp8_e4m3`. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
|
||||
|
||||
- **Reasoning parser:**
|
||||
|
||||
Append `--reasoning-parser deepseek-r1` to enable structured reasoning traces (`reasoning_content` field in the response).
|
||||
|
||||
- **Tool calling:**
|
||||
|
||||
Append `--tool-call-parser qwen3_coder` to enable tool calling support.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
The command below launches the server for a 4×H100 setup with reasoning and tool calling enabled. See [Section 4.8](#4-8-fp8-and-nvfp4-deployment) for FP8 and NVFP4 variants.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser deepseek-r1
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage (Text)
|
||||
|
||||
SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "Give me 3 bullet points about SGLang."},
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.reasoning_content, resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning: SGLang is a serving framework I know from my training data. Let me recall the key features...
|
||||
|
||||
Content:
|
||||
- **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn and few-shot workloads.
|
||||
- **OpenAI-compatible API** — Drop-in replacement for the OpenAI Python client; no application code changes required to serve a locally-hosted model.
|
||||
- **High-throughput serving** — Continuous batching, chunked prefill, and optimized CUDA kernels deliver state-of-the-art throughput on NVIDIA GPUs across A100, H100, and B200.
|
||||
```
|
||||
|
||||
Streaming chat completion:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "What are the first 5 prime numbers?"},
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=512,
|
||||
stream=True,
|
||||
)
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta and delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
### 4.2 Image Understanding
|
||||
|
||||
Pass image inputs using the OpenAI vision format. Supports both URLs and base64-encoded images:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
# From URL
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"},
|
||||
},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
For local images, encode as base64:
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
with open("screenshot.png", "rb") as f:
|
||||
image_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{image_b64}"},
|
||||
},
|
||||
{"type": "text", "text": "What UI elements are visible on this screen? What action would you take next?"},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 4.3 Video Understanding
|
||||
|
||||
Nemotron 3 Nano Omni uses Conv3D layers and Efficient Video Sampling (EVS) for temporal-spatial video reasoning, processing longer videos at the same compute budget:
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
with open("video.mp4", "rb") as f:
|
||||
video_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {"url": f"data:video/mp4;base64,{video_b64}"},
|
||||
},
|
||||
{"type": "text", "text": "Summarize what happens in this video step by step."},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 4.4 Audio Understanding
|
||||
|
||||
Pass audio inputs as base64-encoded WAV or MP3 data:
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
with open("audio.wav", "rb") as f:
|
||||
audio_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "input_audio",
|
||||
"input_audio": {"data": audio_b64, "format": "wav"},
|
||||
},
|
||||
{"type": "text", "text": "Transcribe and summarize what was said in this audio."},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 4.5 Mixed Multimodal Input
|
||||
|
||||
Combine modalities in a single request. For example, an image alongside an audio question about it:
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
with open("chart.png", "rb") as f:
|
||||
image_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{image_b64}"},
|
||||
},
|
||||
{"type": "text", "text": "Analyze this chart. What are the key trends and what conclusion does the data support?"},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 4.6 Reasoning
|
||||
|
||||
The model supports two modes — Reasoning ON (default) vs OFF. Toggle per-request by setting `enable_thinking` to `False`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
# Reasoning ON (default)
|
||||
print("Reasoning on")
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is the derivative of x^3 sin(x)?"},
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(f"Reasoning:\n{resp.choices[0].message.reasoning_content[:300]}...\nContent:\n{resp.choices[0].message.content}")
|
||||
print("\n")
|
||||
|
||||
# Reasoning OFF
|
||||
print("Reasoning off")
|
||||
resp = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is 15% of 200?"},
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=256,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
print(f"Content:\n{resp.choices[0].message.content}")
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning on
|
||||
Reasoning:
|
||||
The user wants the derivative of x^3 sin(x). I'll apply the product rule: d/dx[u·v] = u'v + uv'. Here u = x^3, v = sin(x). So u' = 3x^2, v' = cos(x). The result is 3x^2·sin(x) + x^3·cos(x)...
|
||||
Content:
|
||||
Using the product rule: d/dx[x³ sin(x)] = 3x² sin(x) + x³ cos(x)
|
||||
|
||||
|
||||
Reasoning off
|
||||
Content:
|
||||
15% of 200 is **30**.
|
||||
```
|
||||
|
||||
### 4.7 Tool Calling
|
||||
|
||||
Call functions using the OpenAI Tools schema. The server must be launched with `--tool-call-parser qwen3_coder`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and state, e.g. San Francisco, CA",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
completion = client.chat.completions.create(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is the weather like in Santa Clara, CA?"},
|
||||
],
|
||||
tools=TOOLS,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=512,
|
||||
stream=False,
|
||||
)
|
||||
print(completion.choices[0].message.reasoning_content)
|
||||
print(completion.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
The user is asking about weather in Santa Clara, CA. I have a get_weather function that takes a location and optional unit. I should call it with location="Santa Clara, CA".
|
||||
|
||||
[ChatCompletionMessageFunctionToolCall(id='call_abc123', function=Function(arguments='{"location": "Santa Clara, CA", "unit": "fahrenheit"}', name='get_weather'), type='function', index=0)]
|
||||
```
|
||||
|
||||
### 4.8 FP8 and NVFP4 Deployment
|
||||
|
||||
**FP8 variant** (recommended for throughput-critical serving on H100/H200/B200):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser deepseek-r1
|
||||
```
|
||||
|
||||
**NVFP4 variant** (maximum efficiency on Blackwell B200):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser deepseek-r1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Efficiency Benchmark
|
||||
|
||||
Nemotron 3 Nano Omni achieves **9x higher throughput** than other open omni models at the same interactivity level, delivering lower cost and better scalability without sacrificing responsiveness. It also achieves **~20% higher multimodal intelligence** compared to the best open alternative across image, video, and audio reasoning tasks.
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
- Hardware: B200 (8×)
|
||||
- Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: main branch
|
||||
|
||||
Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--max-running-requests 1024 \
|
||||
--host 0.0.0.0 \
|
||||
--attention-backend flashinfer \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 4096 \
|
||||
--max-concurrency 256
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 256
|
||||
Successful requests: 4096
|
||||
Benchmark duration (s): 206.52
|
||||
Total input tokens: 2081726
|
||||
Total input text tokens: 2081726
|
||||
Total generated tokens: 2087288
|
||||
Total generated tokens (retokenized): 1945477
|
||||
Request throughput (req/s): 19.83
|
||||
Input token throughput (tok/s): 10080.25
|
||||
Output token throughput (tok/s): 10107.18
|
||||
Peak output token throughput (tok/s): 20199.00
|
||||
Peak concurrent requests: 291
|
||||
Total token throughput (tok/s): 20187.44
|
||||
Concurrency: 250.83
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 12646.47
|
||||
Median E2E Latency (ms): 12371.84
|
||||
P90 E2E Latency (ms): 22889.81
|
||||
P99 E2E Latency (ms): 26528.70
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 220.66
|
||||
Median TTFT (ms): 97.67
|
||||
P99 TTFT (ms): 2068.63
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.98
|
||||
Median TPOT (ms): 24.36
|
||||
P99 TPOT (ms): 44.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 24.43
|
||||
Median ITL (ms): 10.91
|
||||
P95 ITL (ms): 62.68
|
||||
P99 ITL (ms): 100.60
|
||||
Max ITL (ms): 2171.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.3 Accuracy Benchmark
|
||||
|
||||
**Environment**
|
||||
- Hardware: B200 (8×)
|
||||
- Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: main branch
|
||||
|
||||
**Launch Model**
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--attention-backend flashinfer \
|
||||
--reasoning-parser deepseek-r1
|
||||
```
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark
|
||||
|
||||
**Run Benchmark**
|
||||
```shell Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
Accuracy: 0.830
|
||||
Invalid: 0.000
|
||||
Latency: 13.970 s
|
||||
Output throughput: 1611.623 token/s
|
||||
```
|
||||
|
||||
#### 5.3.2 MMLU Benchmark
|
||||
|
||||
**Run Benchmark**
|
||||
```shell Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.510
|
||||
subject: anatomy, #q:135, acc: 0.711
|
||||
subject: astronomy, #q:152, acc: 0.829
|
||||
subject: business_ethics, #q:100, acc: 0.760
|
||||
subject: clinical_knowledge, #q:265, acc: 0.781
|
||||
subject: college_biology, #q:144, acc: 0.854
|
||||
subject: college_chemistry, #q:100, acc: 0.560
|
||||
subject: college_computer_science, #q:100, acc: 0.700
|
||||
subject: college_mathematics, #q:100, acc: 0.590
|
||||
subject: college_medicine, #q:173, acc: 0.775
|
||||
subject: college_physics, #q:102, acc: 0.559
|
||||
subject: computer_security, #q:100, acc: 0.750
|
||||
subject: conceptual_physics, #q:235, acc: 0.821
|
||||
subject: econometrics, #q:114, acc: 0.605
|
||||
subject: electrical_engineering, #q:145, acc: 0.759
|
||||
subject: elementary_mathematics, #q:378, acc: 0.638
|
||||
subject: formal_logic, #q:126, acc: 0.524
|
||||
subject: global_facts, #q:100, acc: 0.400
|
||||
subject: high_school_biology, #q:310, acc: 0.906
|
||||
subject: high_school_chemistry, #q:203, acc: 0.759
|
||||
subject: high_school_computer_science, #q:100, acc: 0.860
|
||||
subject: high_school_european_history, #q:165, acc: 0.812
|
||||
subject: high_school_geography, #q:198, acc: 0.889
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.933
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.785
|
||||
subject: high_school_mathematics, #q:270, acc: 0.496
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.887
|
||||
subject: high_school_physics, #q:151, acc: 0.675
|
||||
subject: high_school_psychology, #q:545, acc: 0.895
|
||||
subject: high_school_statistics, #q:216, acc: 0.731
|
||||
subject: high_school_us_history, #q:204, acc: 0.858
|
||||
subject: high_school_world_history, #q:237, acc: 0.873
|
||||
subject: human_aging, #q:223, acc: 0.740
|
||||
subject: human_sexuality, #q:131, acc: 0.855
|
||||
subject: international_law, #q:121, acc: 0.851
|
||||
subject: jurisprudence, #q:108, acc: 0.815
|
||||
subject: logical_fallacies, #q:163, acc: 0.847
|
||||
subject: machine_learning, #q:112, acc: 0.598
|
||||
subject: management, #q:103, acc: 0.864
|
||||
subject: marketing, #q:234, acc: 0.910
|
||||
subject: medical_genetics, #q:100, acc: 0.880
|
||||
subject: miscellaneous, #q:783, acc: 0.881
|
||||
subject: moral_disputes, #q:346, acc: 0.780
|
||||
subject: moral_scenarios, #q:895, acc: 0.543
|
||||
subject: nutrition, #q:306, acc: 0.814
|
||||
subject: philosophy, #q:311, acc: 0.733
|
||||
subject: prehistory, #q:324, acc: 0.852
|
||||
subject: professional_accounting, #q:282, acc: 0.553
|
||||
subject: professional_law, #q:1534, acc: 0.565
|
||||
subject: professional_medicine, #q:272, acc: 0.779
|
||||
subject: professional_psychology, #q:612, acc: 0.760
|
||||
subject: public_relations, #q:110, acc: 0.709
|
||||
subject: security_studies, #q:245, acc: 0.759
|
||||
subject: sociology, #q:201, acc: 0.831
|
||||
subject: us_foreign_policy, #q:100, acc: 0.910
|
||||
subject: virology, #q:166, acc: 0.560
|
||||
subject: world_religions, #q:171, acc: 0.807
|
||||
Total latency: 67.512
|
||||
Average accuracy: 0.737
|
||||
```
|
||||
@@ -0,0 +1,375 @@
|
||||
---
|
||||
title: Nemotron3-Nano
|
||||
metatags:
|
||||
description: "Deploy NVIDIA Nemotron3-Nano 30B hybrid LLM with SGLang - MoE, Mamba2, and attention layers with BF16/FP8 precision options."
|
||||
---
|
||||
|
||||
import { Nemotron3NanoDeployment } from '/src/snippets/autoregressive/nemotron3-nano-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
`NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks.
|
||||
|
||||
The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported.
|
||||
|
||||
At a high level:
|
||||
|
||||
- **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only.
|
||||
- **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth.
|
||||
- **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install nightly wheel through:
|
||||
```bash Command
|
||||
uv pip install sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance tuning.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
|
||||
|
||||
<Nemotron3NanoDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Attention backend**:
|
||||
|
||||
**H200**: Use flash attention 3 backend by default.
|
||||
**B200**: Use flashinfer backend by default.
|
||||
|
||||
- **TP support**:
|
||||
|
||||
To set tp size, use `--tp <1|2|4|8>`.
|
||||
|
||||
- **FP8 KV cache**:
|
||||
|
||||
To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage (OpenAI-Compatible API)
|
||||
|
||||
SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Summarize what MoE models are in 5 bullets."},
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=256,
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
|
||||
```
|
||||
|
||||
Streaming chat completion
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "What are the first 5 prime numbers?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=1024,
|
||||
stream=True,
|
||||
)
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta and delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
### 4.2 Reasoning
|
||||
To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable_thinking to False, as shown below.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
# Reasoning on (default)
|
||||
print("Reasoning on")
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Write a haiku about GPUs."}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
|
||||
# Reasoning off
|
||||
print("Reasoning off")
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Write a haiku about GPUs."}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=256,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
|
||||
```
|
||||
|
||||
### 4.3 Tool calling
|
||||
To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool_calls.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
# Tool calling via OpenAI tools schema
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "calculate_tip",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"bill_total": {
|
||||
"type": "integer",
|
||||
"description": "The total amount of the bill"
|
||||
},
|
||||
"tip_percentage": {
|
||||
"type": "integer",
|
||||
"description": "The percentage of tip to be applied"
|
||||
}
|
||||
},
|
||||
"required": ["bill_total", "tip_percentage"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
completion = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "system", "content": ""},
|
||||
{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
|
||||
],
|
||||
tools=TOOLS,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=512,
|
||||
stream=False
|
||||
)
|
||||
|
||||
print(completion.choices[0].message.reasoning_content)
|
||||
print(completion.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU
|
||||
|
||||
**FP8 variant**
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 1024 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 4096 \
|
||||
--max-concurrency 256
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 256
|
||||
Successful requests: 4096
|
||||
Benchmark duration (s): 183.18
|
||||
Total input tokens: 2081726
|
||||
Total input text tokens: 2081726
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2116125
|
||||
Total generated tokens (retokenized): 1076256
|
||||
Request throughput (req/s): 22.36
|
||||
Input token throughput (tok/s): 11364.25
|
||||
Output token throughput (tok/s): 11552.04
|
||||
Peak output token throughput (tok/s): 24692.00
|
||||
Peak concurrent requests: 294
|
||||
Total token throughput (tok/s): 22916.30
|
||||
Concurrency: 251.19
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 11233.74
|
||||
Median E2E Latency (ms): 11142.97
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 172.99
|
||||
Median TTFT (ms): 116.57
|
||||
P99 TTFT (ms): 1193.68
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 21.74
|
||||
Median TPOT (ms): 21.14
|
||||
P99 TPOT (ms): 41.12
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 21.45
|
||||
Median ITL (ms): 9.06
|
||||
P95 ITL (ms): 62.59
|
||||
P99 ITL (ms): 110.83
|
||||
Max ITL (ms): 5368.19
|
||||
==================================================
|
||||
```
|
||||
|
||||
**BF16 variant**
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
|
||||
--trust-remote-code \
|
||||
--max-running-requests 1024 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 4096 \
|
||||
--max-concurrency 256
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 256
|
||||
Successful requests: 4096
|
||||
Benchmark duration (s): 360.22
|
||||
Total input tokens: 2081726
|
||||
Total input text tokens: 2081726
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2087288
|
||||
Total generated tokens (retokenized): 1940652
|
||||
Request throughput (req/s): 11.37
|
||||
Input token throughput (tok/s): 5779.10
|
||||
Output token throughput (tok/s): 5794.55
|
||||
Peak output token throughput (tok/s): 9169.00
|
||||
Peak concurrent requests: 276
|
||||
Total token throughput (tok/s): 11573.65
|
||||
Concurrency: 249.76
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 21965.10
|
||||
Median E2E Latency (ms): 21706.35
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 211.54
|
||||
Median TTFT (ms): 93.06
|
||||
P99 TTFT (ms): 2637.66
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 43.27
|
||||
Median TPOT (ms): 43.04
|
||||
P99 TPOT (ms): 61.15
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 42.77
|
||||
Median ITL (ms): 28.46
|
||||
P95 ITL (ms): 71.85
|
||||
P99 ITL (ms): 113.20
|
||||
Max ITL (ms): 5237.28
|
||||
==================================================
|
||||
|
||||
```
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
**Environment**
|
||||
- Hardware: NVIDIA B200 GPU
|
||||
- Model: BF16 checkpoint
|
||||
|
||||
**Launch Model**
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser nemotron_3
|
||||
```
|
||||
|
||||
**Run Benchmark with lm-eval**
|
||||
```bash Command
|
||||
pip install lm-eval[api]==0.4.9.2
|
||||
|
||||
lm_eval --model local-completions --tasks gsm8k --model_args "model=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=4,max_retries=3,tokenized_requests=False,max_lengths=16384" --gen_kwargs '{"chat_template_kwargs":{"thinking":true}}' --batch_size 256
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||||
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|
||||
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.5603|± |0.0137|
|
||||
| | |strict-match | 5|exact_match|↑ |0.8453|± |0.0100|
|
||||
```
|
||||
@@ -0,0 +1,571 @@
|
||||
---
|
||||
title: NVIDIA Nemotron3-Super
|
||||
metatags:
|
||||
description: "Deploy NVIDIA Nemotron3-Super with SGLang - 120B hybrid MoE model (12B active) with 1M context window optimized for multi-agent systems and tool use."
|
||||
---
|
||||
|
||||
import { Nemotron3SuperDeployment } from '/src/snippets/autoregressive/nemotron3-super-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
`NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step.
|
||||
|
||||
Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows.
|
||||
|
||||
Architecture and key features:
|
||||
|
||||
- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
|
||||
- **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5).
|
||||
- **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text.
|
||||
- **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker.
|
||||
|
||||
```bash Command
|
||||
# Install from source
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
|
||||
# Or use Docker
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance tuning.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command.
|
||||
|
||||
<Nemotron3SuperDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Attention backend**:
|
||||
|
||||
**H200**: Use flash attention 3 backend by default.
|
||||
**B200**: Use flashinfer backend by default.
|
||||
|
||||
- **TP support**:
|
||||
|
||||
To set tp size, use `--tp <2|4|8>`.
|
||||
|
||||
- **FP8 KV cache**:
|
||||
|
||||
To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
|
||||
--host 0.0.0.0 \
|
||||
--port 5000 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser nemotron_3
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage (OpenAI-Compatible API)
|
||||
|
||||
SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:5000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "Give me 3 bullet points about SGLang."},
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
|
||||
print("\n")
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others.
|
||||
|
||||
First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens)
|
||||
|
||||
Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens.
|
||||
- It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput.
|
||||
- Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers.
|
||||
```
|
||||
|
||||
Streaming chat completion:
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:5000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "What are the first 5 prime numbers?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=1024,
|
||||
stream=True,
|
||||
)
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta and delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
The first 5 prime numbers are:
|
||||
**2, 3, 5, 7, 11**.
|
||||
|
||||
### Explanation:
|
||||
- A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself.
|
||||
- **2** is the smallest and only even prime number.
|
||||
- **3** is prime (divisible only by 1 and 3).
|
||||
- **4** is not prime (divisible by 2).
|
||||
- **5** is prime.
|
||||
- **6** is not prime (divisible by 2 and 3).
|
||||
- **7** is prime.
|
||||
- **8, 9, 10** are not prime.
|
||||
- **11** is prime (the fifth in the sequence).
|
||||
|
||||
Note: **1 is not considered a prime number** by definition, as it has only one positive divisor.
|
||||
This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊
|
||||
```
|
||||
|
||||
### 4.2 Reasoning
|
||||
|
||||
The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:5000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
# Reasoning on (default)
|
||||
print("Reasoning on")
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."}
|
||||
],
|
||||
temperature=1,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
|
||||
print("\n")
|
||||
# Reasoning off
|
||||
print("Reasoning off")
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Give me 3 facts about SGLang."}
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=256,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...")
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning on
|
||||
Reasoning:
|
||||
We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs.
|
||||
|
||||
Let's deci...
|
||||
Content:
|
||||
Silicon hearts beat
|
||||
Paint vivid worlds with bright light
|
||||
GPU dreams rise...
|
||||
|
||||
Reasoning off
|
||||
Content:
|
||||
Certainly! Here are three accurate and informative facts about **SGLang**:
|
||||
|
||||
1. **SGLang is a high-performance serving system for large language models (LLMs)**
|
||||
Developed by researchers at UC Berk...
|
||||
```
|
||||
|
||||
### 4.3 Tool Calling
|
||||
|
||||
Call functions using the OpenAI Tools schema and inspect returned `tool_calls`.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:5000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
# Tool calling via OpenAI tools schema
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "calculate_tip",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"bill_total": {
|
||||
"type": "integer",
|
||||
"description": "The total amount of the bill"
|
||||
},
|
||||
"tip_percentage": {
|
||||
"type": "integer",
|
||||
"description": "The percentage of tip to be applied"
|
||||
}
|
||||
},
|
||||
"required": ["bill_total", "tip_percentage"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
completion = client.chat.completions.create(
|
||||
model="nemotron",
|
||||
messages=[
|
||||
{"role": "system", "content": ""},
|
||||
{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
|
||||
],
|
||||
tools=TOOLS,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=512,
|
||||
stream=False
|
||||
)
|
||||
|
||||
print(completion.choices[0].message.reasoning_content)
|
||||
print(completion.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that.
|
||||
|
||||
[ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)]
|
||||
```
|
||||
|
||||
### 4.4 Controlling Reasoning Budget
|
||||
|
||||
The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.
|
||||
|
||||
If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.
|
||||
|
||||
```python Example
|
||||
from typing import Any, Dict, List
|
||||
import openai
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
class ThinkingBudgetClient:
|
||||
def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
|
||||
self.base_url = base_url
|
||||
self.api_key = api_key
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
|
||||
self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)
|
||||
|
||||
def chat_completion(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[Dict[str, Any]],
|
||||
reasoning_budget: int = 512,
|
||||
max_tokens: int = 1024,
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
assert (
|
||||
max_tokens > reasoning_budget
|
||||
), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"
|
||||
|
||||
# 1. first call chat completion to get reasoning content
|
||||
response = self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
max_tokens=reasoning_budget,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
reasoning_content = response.choices[0].message.reasoning_content or ""
|
||||
|
||||
if "</think>" not in reasoning_content:
|
||||
# reasoning content is too long, closed with a period (.)
|
||||
reasoning_content = f"{reasoning_content}.\n</think>\n\n"
|
||||
|
||||
reasoning_tokens_used = len(
|
||||
self.tokenizer.encode(reasoning_content, add_special_tokens=False)
|
||||
)
|
||||
remaining_tokens = max_tokens - reasoning_tokens_used
|
||||
|
||||
assert (
|
||||
remaining_tokens > 0
|
||||
), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."
|
||||
|
||||
# 2. append reasoning content to messages and call completion
|
||||
messages.append({"role": "assistant", "content": reasoning_content})
|
||||
prompt = self.tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
continue_final_message=True,
|
||||
)
|
||||
|
||||
response = self.client.completions.create(
|
||||
model=model,
|
||||
prompt=prompt,
|
||||
max_tokens=remaining_tokens,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
response_data = {
|
||||
"reasoning_content": reasoning_content.strip().strip("</think>").strip(),
|
||||
"content": response.choices[0].text,
|
||||
"finish_reason": response.choices[0].finish_reason,
|
||||
}
|
||||
return response_data
|
||||
```
|
||||
|
||||
Usage example with `reasoning_budget=128`:
|
||||
|
||||
```python Example
|
||||
SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"
|
||||
|
||||
# Client
|
||||
client = ThinkingBudgetClient(
|
||||
base_url="http://127.0.0.1:5000/v1",
|
||||
api_key="null",
|
||||
tokenizer_name_or_path=SERVED_MODEL_NAME
|
||||
)
|
||||
|
||||
resp = client.chat_completion(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Write a haiku about GPUs."}
|
||||
],
|
||||
temperature=1,
|
||||
max_tokens=512,
|
||||
reasoning_budget=128
|
||||
)
|
||||
print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact.
|
||||
|
||||
First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once.
|
||||
|
||||
Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor.
|
||||
Content:
|
||||
|
||||
Silicon minds awaken,
|
||||
Thousands of cores hum in unison—
|
||||
Lightning paints the void.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
- Hardware: H200 (4x)
|
||||
- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: main branch
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--max-running-requests 1024 \
|
||||
--host 0.0.0.0 \
|
||||
--port 5000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 5000 \
|
||||
--model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 4096 \
|
||||
--max-concurrency 256
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 256
|
||||
Successful requests: 4096
|
||||
Benchmark duration (s): 623.49
|
||||
Total input tokens: 2081726
|
||||
Total input text tokens: 2081726
|
||||
Total generated tokens: 2087288
|
||||
Total generated tokens (retokenized): 2044666
|
||||
Request throughput (req/s): 6.57
|
||||
Input token throughput (tok/s): 3338.85
|
||||
Output token throughput (tok/s): 3347.77
|
||||
Peak output token throughput (tok/s): 6349.00
|
||||
Peak concurrent requests: 270
|
||||
Total token throughput (tok/s): 6686.62
|
||||
Concurrency: 250.35
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 38108.46
|
||||
Median E2E Latency (ms): 37186.80
|
||||
P90 E2E Latency (ms): 69325.24
|
||||
P99 E2E Latency (ms): 77776.90
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 436.49
|
||||
Median TTFT (ms): 114.90
|
||||
P99 TTFT (ms): 6938.11
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 75.02
|
||||
Median TPOT (ms): 76.02
|
||||
P99 TPOT (ms): 92.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 74.07
|
||||
Median ITL (ms): 38.45
|
||||
P95 ITL (ms): 230.42
|
||||
P99 ITL (ms): 242.70
|
||||
Max ITL (ms): 7181.72
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
**Environment**
|
||||
- Hardware: H200 (4x)
|
||||
- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: main branch
|
||||
|
||||
**Launch Model**
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--reasoning-parser nemotron_3
|
||||
```
|
||||
|
||||
**Run Benchmark**
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 5000
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
Accuracy: 0.950
|
||||
Invalid: 0.000
|
||||
Latency: 21.442 s
|
||||
Output throughput: 996.815 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
**Run Benchmark**
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 5000
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.730
|
||||
subject: anatomy, #q:135, acc: 0.830
|
||||
subject: astronomy, #q:152, acc: 0.934
|
||||
subject: business_ethics, #q:100, acc: 0.830
|
||||
subject: clinical_knowledge, #q:265, acc: 0.879
|
||||
subject: college_biology, #q:144, acc: 0.931
|
||||
subject: college_chemistry, #q:100, acc: 0.620
|
||||
subject: college_computer_science, #q:100, acc: 0.840
|
||||
subject: college_mathematics, #q:100, acc: 0.820
|
||||
subject: college_medicine, #q:173, acc: 0.821
|
||||
subject: college_physics, #q:102, acc: 0.794
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.919
|
||||
subject: econometrics, #q:114, acc: 0.746
|
||||
subject: electrical_engineering, #q:145, acc: 0.828
|
||||
subject: elementary_mathematics, #q:378, acc: 0.926
|
||||
subject: formal_logic, #q:126, acc: 0.857
|
||||
subject: global_facts, #q:100, acc: 0.570
|
||||
subject: high_school_biology, #q:310, acc: 0.952
|
||||
subject: high_school_chemistry, #q:203, acc: 0.828
|
||||
subject: high_school_computer_science, #q:100, acc: 0.940
|
||||
subject: high_school_european_history, #q:165, acc: 0.861
|
||||
subject: high_school_geography, #q:198, acc: 0.939
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.990
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.928
|
||||
subject: high_school_mathematics, #q:270, acc: 0.700
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.966
|
||||
subject: high_school_physics, #q:151, acc: 0.834
|
||||
subject: high_school_psychology, #q:545, acc: 0.960
|
||||
subject: high_school_statistics, #q:216, acc: 0.852
|
||||
subject: high_school_us_history, #q:204, acc: 0.926
|
||||
subject: high_school_world_history, #q:237, acc: 0.937
|
||||
subject: human_aging, #q:223, acc: 0.879
|
||||
subject: human_sexuality, #q:131, acc: 0.939
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.898
|
||||
subject: logical_fallacies, #q:163, acc: 0.914
|
||||
subject: machine_learning, #q:112, acc: 0.821
|
||||
subject: management, #q:103, acc: 0.903
|
||||
subject: marketing, #q:234, acc: 0.944
|
||||
subject: medical_genetics, #q:100, acc: 0.980
|
||||
subject: miscellaneous, #q:783, acc: 0.945
|
||||
subject: moral_disputes, #q:346, acc: 0.861
|
||||
subject: moral_scenarios, #q:895, acc: 0.542
|
||||
subject: nutrition, #q:306, acc: 0.902
|
||||
subject: philosophy, #q:311, acc: 0.884
|
||||
subject: prehistory, #q:324, acc: 0.920
|
||||
subject: professional_accounting, #q:282, acc: 0.805
|
||||
subject: professional_law, #q:1534, acc: 0.681
|
||||
subject: professional_medicine, #q:272, acc: 0.923
|
||||
subject: professional_psychology, #q:612, acc: 0.889
|
||||
subject: public_relations, #q:110, acc: 0.800
|
||||
subject: security_studies, #q:245, acc: 0.837
|
||||
subject: sociology, #q:201, acc: 0.960
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.590
|
||||
subject: world_religions, #q:171, acc: 0.906
|
||||
Total latency: 150.267
|
||||
Average accuracy: 0.841
|
||||
```
|
||||
@@ -0,0 +1,554 @@
|
||||
---
|
||||
title: NVIDIA Nemotron3-Ultra
|
||||
description: "Deploy NVIDIA Nemotron3-Ultra with SGLang - 550B hybrid MoE model (55B active) with 1M context window, BF16/NVFP4 support, built for long-running autonomous agents."
|
||||
tag:
|
||||
NEW
|
||||
---
|
||||
|
||||
import { Nemotron3UltraDeployment } from '/src/snippets/autoregressive/nemotron3-ultra-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
`NVIDIA Nemotron3-Ultra` is an open frontier reasoning model in the Nemotron 3 family, built for long-running autonomous agents. It is optimized for complex orchestration across coding, deep research, enterprise workflows, and EDA use cases where agents must sustain reasoning across many steps and large context windows.
|
||||
|
||||
Nemotron 3 Ultra is a 550B parameter hybrid MoE model that activates only 55B parameters per forward pass, delivering frontier reasoning accuracy with high-throughput inference. It supports a 1M token context window so agents can keep conversation history, tool outputs, and plan state in view across persistent workflows.
|
||||
|
||||
Architecture and key features:
|
||||
|
||||
- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
|
||||
- **Long-horizon agentic reasoning:** Tuned for agents that plan, call tools, inspect results, recover from failures, and continue working across long task horizons — coding, deep research, enterprise automation, and EDA.
|
||||
- **1M token context window:** Sustains coherent agent state across extended workflows without re-ingestion.
|
||||
- **BF16 and NVFP4 quantization:** Deployable from multi-node H100 down to a single Blackwell node with NVFP4.
|
||||
- **Multi-environment RL post-training:** Post-trained with reinforcement learning across multiple environments for robust reasoning and reliable agentic behavior.
|
||||
- **Open weights, open data, open recipes:** Customizable for domain-specific agents and deployable across your own infrastructure.
|
||||
|
||||
**Modalities:** Input: text — Output: text
|
||||
|
||||
**Supported GPUs:**
|
||||
- **BF16:** 16×H100, 16×H200, 8×B200/B300
|
||||
- **NVFP4:** 4/8×B200/B300, 4×GB200/GB300
|
||||
|
||||
Available model variants on HuggingFace:
|
||||
- [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16)
|
||||
- [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Nemotron3-Ultra support is included in the latest stable release.
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance tuning.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: select model precision, hardware, tensor parallelism, and common knobs to generate a launch command.
|
||||
|
||||
The generator only emits a runnable command for combinations that NVIDIA / SGLang have validated. Selecting an unverified tuple (e.g. NVFP4 on H100/H200, BF16 with TP=4 on H100, …) is **blocked** — the command pane shows an explicit error and the verified support matrix instead of a launch line, so unvalidated commands can't be copied by accident.
|
||||
|
||||
<Nemotron3UltraDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Attention backend**:
|
||||
|
||||
**H100/H200**: Use flash attention 3 backend by default.
|
||||
**B200/GB200/B300/GB300**: Set `--attention-backend trtllm_mha`. The flashinfer default breaks the overlap scheduler on Blackwell, so `trtllm_mha` is required there.
|
||||
|
||||
- **Mamba scheduler strategy**:
|
||||
|
||||
Always launch with `--mamba-radix-cache-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state.
|
||||
|
||||
- **Mamba backend**:
|
||||
|
||||
The Mamba layers use the Triton SSM kernels by default. For better performance, set `--mamba-backend flashinfer` to use the FlashInfer Mamba kernels instead.
|
||||
|
||||
- **Mamba SSM precision**:
|
||||
|
||||
The SSM state dtype defaults to the model config value. Set `--mamba-ssm-dtype float16` to store the Mamba states in FP16, which reduces mamba cache memory without significant accuracy loss.
|
||||
|
||||
- **Mamba SSM stochastic rounding**:
|
||||
|
||||
When storing the Mamba states in FP16, add `--enable-mamba-cache-stochastic-rounding` to round SSM cache writes stochastically and reduce accumulation bias. It requires `--mamba-ssm-dtype float16` and CUDA; with the default `--mamba-backend triton` it additionally requires SM100. Use `--mamba-cache-philox-rounds` to control the number of Philox rounds (`0` uses the backend default).
|
||||
|
||||
- **TP support**:
|
||||
|
||||
To set tp size, use `--tp <4|8|16>`. Recommended pairings:
|
||||
- BF16: `--tp 16` on H100/H200, `--tp 8` on B200/B300
|
||||
- NVFP4: `--tp 4` or `--tp 8` on B200/B300, `--tp 4` on GB200/GB300
|
||||
|
||||
- **Multi-node BF16 on H100**:
|
||||
|
||||
The 16×H100 BF16 setup spans two nodes. Use `--dist-init-addr <head-node-ip>:5000 --nnodes 2 --node-rank <0|1>` on each node and keep `--tp 16`.
|
||||
|
||||
- **DP attention**:
|
||||
|
||||
By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--dp <N> --enable-dp-attention`) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--dp` must divide `--tp`.
|
||||
|
||||
- **Expert parallel (EP)**:
|
||||
|
||||
This MoE only supports `ep_size == 1` (off) or `ep_size == tp_size`. To enable expert parallelism, set `--ep <tp>` with the same value as `--tp`.
|
||||
|
||||
- **Multi-token prediction (MTP)**:
|
||||
|
||||
Enable MTP speculative decoding (the toggle above) for low latency.
|
||||
|
||||
- **FP8 KV cache**:
|
||||
|
||||
To enable fp8 kv cache, set `--kv-cache-dtype fp8_e4m3`. This is enabled by default on the NVFP4 checkpoint.
|
||||
|
||||
- **Reasoning parser**:
|
||||
|
||||
Set `--reasoning-parser nemotron_3` to enable structured reasoning traces (`reasoning_content` field in the response).
|
||||
|
||||
- **Tool calling**:
|
||||
|
||||
Set `--tool-call-parser qwen3_coder` to enable tool calling support.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--attention-backend trtllm_mha \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser nemotron_3
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage (OpenAI-Compatible API)
|
||||
|
||||
SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "Give me 3 bullet points about SGLang."},
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
|
||||
print("\n")
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning: The user wants 3 bullet points about SGLang. Let me recall what I know about SGLang — it's a high-performance serving framework for large language models with a focus on structured generation and efficient KV cache reuse...(more tokens)
|
||||
|
||||
Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn agent loops and few-shot workloads.
|
||||
- **OpenAI-compatible API and structured generation** — Drop-in replacement for the OpenAI client, with first-class support for constrained decoding (JSON schema, regex) and OpenAI-style tool calling.
|
||||
- **High-throughput serving on NVIDIA GPUs** — Continuous batching, chunked prefill, FP8/NVFP4 quantization, and optimized CUDA kernels deliver state-of-the-art throughput across H100, H200, B200, and GB200.
|
||||
```
|
||||
|
||||
Streaming chat completion:
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful AI assistant."},
|
||||
{"role": "user", "content": "What are the first 5 prime numbers?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=1024,
|
||||
stream=True,
|
||||
)
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta and delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
The first 5 prime numbers are:
|
||||
**2, 3, 5, 7, 11**.
|
||||
|
||||
### Explanation:
|
||||
- A **prime number** is a natural number greater than 1 whose only positive divisors are 1 and itself.
|
||||
- **2** is the smallest prime and the only even prime.
|
||||
- **3, 5, 7, 11** are each divisible only by 1 and themselves.
|
||||
- **1** is not prime by definition (it has only one positive divisor).
|
||||
- **4, 6, 8, 9, 10** are composite.
|
||||
```
|
||||
|
||||
### 4.2 Reasoning
|
||||
|
||||
The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
# Reasoning on (default)
|
||||
print("Reasoning on")
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Plan a 3-step approach to debug a flaky integration test. Keep the thinking process short."}
|
||||
],
|
||||
temperature=1,
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
|
||||
print("\n")
|
||||
# Reasoning off
|
||||
print("Reasoning off")
|
||||
resp = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Give me 3 facts about SGLang."}
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=256,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
print(f"Content: \n{resp.choices[0].message.content[:200]}...")
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning on
|
||||
Reasoning:
|
||||
The user wants a short reasoning chain plus a 3-step debug plan for a flaky integration test. I'll think briefly about common causes (timing/race, shared state, external service variance) and pick a t...
|
||||
Content:
|
||||
1. **Reproduce deterministically** — run the test in a loop (e.g. 50–100x) with logging at the suspected race points to confirm the failure rate and surface ordering.
|
||||
2. **Isolate state** — re-run with...
|
||||
|
||||
Reasoning off
|
||||
Content:
|
||||
Here are 3 facts about SGLang:
|
||||
|
||||
1. **High-performance LLM serving system** developed at UC Berkeley with contributions from a broad open-source community, focused on throughput and latency at scale.
|
||||
...
|
||||
```
|
||||
|
||||
### 4.3 Tool Calling
|
||||
|
||||
Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. The server must be launched with `--tool-call-parser qwen3_coder`.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
# Tool calling via OpenAI tools schema
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search_codebase",
|
||||
"description": "Search the project codebase for a symbol or pattern.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The symbol, function name, or regex to search for"
|
||||
},
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": "Optional sub-path to restrict the search to"
|
||||
}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
completion = client.chat.completions.create(
|
||||
model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a coding agent. Use tools to inspect the repo before answering."},
|
||||
{"role": "user", "content": "Where is the `RadixCache` class defined?"}
|
||||
],
|
||||
tools=TOOLS,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=512,
|
||||
stream=False
|
||||
)
|
||||
|
||||
print(completion.choices[0].message.reasoning_content)
|
||||
print(completion.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
The user is asking where the RadixCache class is defined. I should search the codebase for the symbol "RadixCache" to find the file and line. I'll call search_codebase with that query.
|
||||
|
||||
[ChatCompletionMessageFunctionToolCall(id='call_8a7f2c4e1b9d4a3e8c2f1d6b', function=Function(arguments='{"query": "class RadixCache"}', name='search_codebase'), type='function', index=0)]
|
||||
```
|
||||
|
||||
### 4.4 Controlling Reasoning Budget
|
||||
|
||||
The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.
|
||||
|
||||
If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.
|
||||
|
||||
```python Example
|
||||
from typing import Any, Dict, List
|
||||
import openai
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
class ThinkingBudgetClient:
|
||||
def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
|
||||
self.base_url = base_url
|
||||
self.api_key = api_key
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
|
||||
self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)
|
||||
|
||||
def chat_completion(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[Dict[str, Any]],
|
||||
reasoning_budget: int = 512,
|
||||
max_tokens: int = 1024,
|
||||
**kwargs,
|
||||
) -> Dict[str, Any]:
|
||||
assert (
|
||||
max_tokens > reasoning_budget
|
||||
), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"
|
||||
|
||||
# 1. first call chat completion to get reasoning content
|
||||
response = self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
max_tokens=reasoning_budget,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
reasoning_content = response.choices[0].message.reasoning_content or ""
|
||||
|
||||
if "</think>" not in reasoning_content:
|
||||
# reasoning content is too long, closed with a period (.)
|
||||
reasoning_content = f"{reasoning_content}.\n</think>\n\n"
|
||||
|
||||
reasoning_tokens_used = len(
|
||||
self.tokenizer.encode(reasoning_content, add_special_tokens=False)
|
||||
)
|
||||
remaining_tokens = max_tokens - reasoning_tokens_used
|
||||
|
||||
assert (
|
||||
remaining_tokens > 0
|
||||
), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."
|
||||
|
||||
# 2. append reasoning content to messages and call completion
|
||||
messages.append({"role": "assistant", "content": reasoning_content})
|
||||
prompt = self.tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
continue_final_message=True,
|
||||
)
|
||||
|
||||
response = self.client.completions.create(
|
||||
model=model,
|
||||
prompt=prompt,
|
||||
max_tokens=remaining_tokens,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
response_data = {
|
||||
"reasoning_content": reasoning_content.strip().strip("</think>").strip(),
|
||||
"content": response.choices[0].text,
|
||||
"finish_reason": response.choices[0].finish_reason,
|
||||
}
|
||||
return response_data
|
||||
```
|
||||
|
||||
Usage example with `reasoning_budget=256`:
|
||||
|
||||
```python Example
|
||||
SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"
|
||||
|
||||
# Client
|
||||
client = ThinkingBudgetClient(
|
||||
base_url="http://127.0.0.1:30000/v1",
|
||||
api_key="null",
|
||||
tokenizer_name_or_path=SERVED_MODEL_NAME
|
||||
)
|
||||
|
||||
resp = client.chat_completion(
|
||||
model=SERVED_MODEL_NAME,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Outline a research plan to evaluate the throughput of two MoE serving strategies."}
|
||||
],
|
||||
temperature=1,
|
||||
max_tokens=1024,
|
||||
reasoning_budget=256
|
||||
)
|
||||
print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
|
||||
```
|
||||
|
||||
Output:
|
||||
```text Output
|
||||
Reasoning: The user wants a research plan to compare throughput of two MoE serving strategies. I should outline goals, baselines, datasets, metrics (tokens/s, TTFT, ITL, MFU), variables to sweep (TP, batch size, sequence length, concurrency), and statistical handling. Keep it concise since reasoning_budget is 256...
|
||||
Content:
|
||||
|
||||
**Research plan**
|
||||
|
||||
1. **Define goal & metrics** — peak token throughput (input+output), TTFT, P99 ITL, MFU; measured at fixed accuracy.
|
||||
2. **Choose baselines** — Strategy A (TP-only) vs Strategy B (TP + expert-parallel). Hold model checkpoint, precision, and KV-cache dtype constant.
|
||||
3. **Sweep** — `{batch ∈ 1,4,16,64, concurrency ∈ 16,64,256, seq_len ∈ 1k,8k,32k}` per strategy.
|
||||
4. **Workload** — `sglang.bench_serving --dataset-name random` with matched input/output budgets.
|
||||
5. **Analysis** — per-config throughput table + roofline overlay; bootstrap CIs over 3 reruns to bound noise.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
- Hardware: GB200 (4x)
|
||||
- Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: main branch
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--attention-backend trtllm_mha \
|
||||
--max-running-requests 1024
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 4096 \
|
||||
--max-concurrency 256
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 256
|
||||
Successful requests: 4096
|
||||
Benchmark duration (s): 1184.58
|
||||
Total input tokens: 2081726
|
||||
Total input text tokens: 2081726
|
||||
Total generated tokens: 2087288
|
||||
Total generated tokens (retokenized): 1990224
|
||||
Request throughput (req/s): 3.46
|
||||
Input token throughput (tok/s): 1757.35
|
||||
Output token throughput (tok/s): 1762.05
|
||||
Peak output token throughput (tok/s): 3150.00
|
||||
Peak concurrent requests: 266
|
||||
Total token throughput (tok/s): 3519.40
|
||||
Concurrency: 249.55
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 72169.95
|
||||
Median E2E Latency (ms): 71994.47
|
||||
P90 E2E Latency (ms): 99898.56
|
||||
P99 E2E Latency (ms): 107119.61
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 40057.33
|
||||
Median TTFT (ms): 41375.93
|
||||
P99 TTFT (ms): 46377.89
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 63.15
|
||||
Median TPOT (ms): 63.65
|
||||
P99 TPOT (ms): 78.16
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 63.14
|
||||
Median ITL (ms): 35.92
|
||||
P95 ITL (ms): 178.10
|
||||
P99 ITL (ms): 182.10
|
||||
Max ITL (ms): 2466.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
**Environment**
|
||||
- Hardware: GB200 (4x)
|
||||
- Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: main branch
|
||||
|
||||
**Launch Model**
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
|
||||
--trust-remote-code \
|
||||
--tp 4 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--attention-backend trtllm_mha \
|
||||
--reasoning-parser nemotron_3
|
||||
```
|
||||
|
||||
**Run Benchmark**
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
Latency: 29.129 s
|
||||
Output throughput: 745.333 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
**Run Benchmark**
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
```text Output
|
||||
TBD
|
||||
```
|
||||
|
||||
---
|
||||
@@ -0,0 +1,664 @@
|
||||
---
|
||||
title: GPT-OSS
|
||||
metatags:
|
||||
description: "Deploy GPT-OSS (20B/120B) with SGLang - configurable reasoning, full chain-of-thought, MXFP4 quantization for single GPU deployment."
|
||||
---
|
||||
|
||||
## 1.Model Introduction
|
||||
|
||||
[GPT-OSS](https://huggingface.co/openai/gpt-oss-20b) is an advanced large language model developed by OpenAI designed for power reasoning, agentic tasks, and versatile developer use cases. It has versions with two model sizes.
|
||||
|
||||
- **gpt-oss-120b** — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) (117B parameters with 5.1B active parameters)
|
||||
- **gpt-oss-20b** — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters)
|
||||
|
||||
GPT-OSS introduces several groundbreaking innovations:
|
||||
|
||||
- **Configurable reasoning effort**: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs.
|
||||
- **Full chain-of-thought**: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users.
|
||||
- **Fine-tunable**: Fully customize models to your specific use case through parameter fine-tuning.
|
||||
- **Agentic capabilities**: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.
|
||||
- **MXFP4 quantization**: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization.
|
||||
|
||||
## 2.SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3.Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The GPT-OSS series comes in two sizes. Recommended starting configurations vary depending on hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
|
||||
|
||||
import { GPTOSSDeployment } from "/src/snippets/autoregressive/gpt-oss-deployment.jsx";
|
||||
|
||||
<GPTOSSDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Native web search:** Set `EXA_API_KEY` in the SGLang server environment to enable built-in web search (Exa). No `--tool-server` is required, and requests are tagged with `x-exa-integration: sglang`.
|
||||
- **Web search defaults:** `numResults=10`, search `type="auto"`, and `contents.highlights=true`. Override with `SGLANG_EXA_NUM_RESULTS`, `SGLANG_EXA_SEARCH_TYPE`, and `SGLANG_EXA_INCLUDE_HIGHLIGHTS`.
|
||||
- **Python tool:** Add `--tool-server demo` to enable the Python interpreter. Runs in a Docker sandbox by default; set `PYTHON_EXECUTION_BACKEND=UV` to run on the host (model-generated code executes locally — use with care).
|
||||
- **MCP tool servers:** For production, point SGLang at external MCP SSE servers with `--tool-server ip-1:port-1,ip-2:port-2`.
|
||||
- **Responses API:** GPT-OSS supports OpenAI's Responses API (`client.responses.create`) in addition to the standard Chat Completions API (see section 4.2.4).
|
||||
- **Use Python 3.12** when running the demo Python tool.
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4.Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GPT-OSS supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model openai/gpt-oss-120b \
|
||||
--reasoning-parser gpt-oss \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer.
|
||||
=============== Content =================
|
||||
**Step‑by‑step solution**
|
||||
|
||||
1. **Understand what “percent” means**
|
||||
“15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\).
|
||||
|
||||
2. **Convert the percent to a decimal (or fraction)**
|
||||
\[
|
||||
\frac{15}{100}=0.15
|
||||
\]
|
||||
|
||||
3. **Set up the multiplication**
|
||||
To find 15 % of 240 we multiply 240 by the decimal 0.15:
|
||||
\[
|
||||
240 \times 0.15
|
||||
\]
|
||||
|
||||
4. **Do the multiplication**
|
||||
One convenient way is to break it into two easier parts:
|
||||
\[
|
||||
240 \times 0.15 = 240 \times \left(\frac{15}{100}\right)
|
||||
= \frac{240 \times 15}{100}
|
||||
\]
|
||||
|
||||
- First compute \(240 \times 15\):
|
||||
\[
|
||||
240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600
|
||||
\]
|
||||
|
||||
- Then divide by 100:
|
||||
\[
|
||||
\frac{3600}{100} = 36
|
||||
\]
|
||||
|
||||
5. **Write the result**
|
||||
\[
|
||||
15\% \text{ of } 240 = 36
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
**Answer:** \(36\)
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
GPT-OSS supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Python Example (without Thinking Process):**
|
||||
|
||||
Start sglang server:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model openai/gpt-oss-120b \
|
||||
--tool-call-parser gpt-oss \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
Start sglang server:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model openai/gpt-oss-120b \
|
||||
--reasoning-parser gpt-oss \
|
||||
--tool-call-parser gpt-oss \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function.
|
||||
=============== Content =================
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you’d like a forecast for the next few days or any other details!"
|
||||
```
|
||||
|
||||
#### 4.2.3 EAGLE3 Speculative Decoding
|
||||
|
||||
SGLang supports speculative decoding for GPT-OSS models using the EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
<Tip>
|
||||
The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable.
|
||||
</Tip>
|
||||
|
||||
#### 4.2.4 Responses API and Built-in Tools
|
||||
|
||||
GPT-OSS supports the OpenAI Responses API with built-in tool use (web search and Python interpreter). Set `EXA_API_KEY` to enable native web search; add `--tool-server demo` only when you also want the Python tool:
|
||||
|
||||
```shell Command
|
||||
export EXA_API_KEY=YOUR_EXA_KEY
|
||||
# Optional: server-side Exa tuning (defaults shown)
|
||||
export SGLANG_EXA_NUM_RESULTS=10
|
||||
export SGLANG_EXA_SEARCH_TYPE=auto
|
||||
export SGLANG_EXA_INCLUDE_HIGHLIGHTS=true
|
||||
# Optional: run Python tool on host instead of Docker (model code executes locally)
|
||||
export PYTHON_EXECUTION_BACKEND=UV
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
For production, use external MCP SSE servers instead of `demo`:
|
||||
```shell Command
|
||||
mcp run -t sse browser_server.py:mcp
|
||||
mcp run -t sse python_server.py:mcp
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--tool-server ip-1:port-1,ip-2:port-2 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
**Example using Responses API:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="sk-123456")
|
||||
|
||||
search_tools = [{"type": "web_search"}]
|
||||
python_tools = [{"type": "code_interpreter"}]
|
||||
|
||||
# Configurable reasoning effort: "high", "medium", or "low"
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helpful assistant.",
|
||||
reasoning_effort="high",
|
||||
input="In one sentence, explain the transformer architecture.",
|
||||
)
|
||||
print(response.output_text)
|
||||
|
||||
# Web search (requires EXA_API_KEY on the SGLang server)
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helpful assistant, you can search the web when needed.",
|
||||
input="Search the web for the latest news about Nvidia stock price",
|
||||
tools=search_tools,
|
||||
)
|
||||
print(response.output_text)
|
||||
|
||||
# Python tool (requires launching SGLang with --tool-server demo)
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helpful assistant, you could use python tool to execute code.",
|
||||
input="Use python tool to calculate the sum of 29138749187 and 29138749187",
|
||||
tools=python_tools,
|
||||
)
|
||||
print(response.output_text)
|
||||
# Output: The sum is 58,277,498,374.
|
||||
```
|
||||
|
||||
## 5.Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x)
|
||||
- Tensor Parallelism: 8
|
||||
- Model: openai/gpt-oss-120b
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Server Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model openai/gpt-oss-120b \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Test Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompt 100 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 52.35
|
||||
Total input tokens: 33178
|
||||
Total input text tokens: 33178
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 21251
|
||||
Total generated tokens (retokenized): 20868
|
||||
Request throughput (req/s): 1.91
|
||||
Input token throughput (tok/s): 633.76
|
||||
Output token throughput (tok/s): 405.93
|
||||
Peak output token throughput (tok/s): 433.00
|
||||
Peak concurrent requests: 8
|
||||
Total token throughput (tok/s): 1039.69
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 523.30
|
||||
Median E2E Latency (ms): 389.91
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 33.71
|
||||
Median TTFT (ms): 31.79
|
||||
P99 TTFT (ms): 108.98
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.31
|
||||
Median TPOT (ms): 2.31
|
||||
P99 TPOT (ms): 2.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.31
|
||||
Median ITL (ms): 2.31
|
||||
P95 ITL (ms): 2.35
|
||||
P99 ITL (ms): 2.38
|
||||
Max ITL (ms): 3.54
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Server Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model openai/gpt-oss-120b \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Test Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompt 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 24.76
|
||||
Total input tokens: 297156
|
||||
Total input text tokens: 297156
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 192432
|
||||
Total generated tokens (retokenized): 187145
|
||||
Request throughput (req/s): 40.39
|
||||
Input token throughput (tok/s): 12003.57
|
||||
Output token throughput (tok/s): 7773.26
|
||||
Peak output token throughput (tok/s): 13780.00
|
||||
Peak concurrent requests: 156
|
||||
Total token throughput (tok/s): 19776.83
|
||||
Concurrency: 89.23
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2208.97
|
||||
Median E2E Latency (ms): 1591.11
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 102.94
|
||||
Median TTFT (ms): 31.53
|
||||
P99 TTFT (ms): 674.32
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.31
|
||||
Median TPOT (ms): 11.00
|
||||
P99 TPOT (ms): 91.28
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.00
|
||||
Median ITL (ms): 5.75
|
||||
P95 ITL (ms): 25.35
|
||||
P99 ITL (ms): 43.18
|
||||
Max ITL (ms): 621.42
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
- GPT-OSS-120b
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.880
|
||||
Invalid: 0.005
|
||||
Latency: 5.262 s
|
||||
Output throughput: 12143.675 token/s
|
||||
```
|
||||
|
||||
- GPT-OSS-20b
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.535
|
||||
Invalid: 0.165
|
||||
Latency: 4.157 s
|
||||
Output throughput: 19589.165 token/s
|
||||
```
|
||||
@@ -0,0 +1,603 @@
|
||||
---
|
||||
title: MiniCPM-V 4.6
|
||||
metatags:
|
||||
description: "Deploy OpenBMB MiniCPM-V 4.6 (Qwen3.5-style hybrid GDN backbone + NaViT vision encoder) on NVIDIA GPUs with SGLang — multimodal text + image + video, slicing for high-resolution images."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
MiniCPM-V 4.6 is the next-generation multimodal model from [OpenBMB](https://huggingface.co/openbmb), the team behind the MiniCPM-V series. The model combines a **Qwen3.5-style hybrid LLM backbone** (Gated Delta Net + full attention) with a **NaViT-packed vision encoder** that handles arbitrary aspect ratios and high-resolution slicing natively, plus end-to-end video support.
|
||||
|
||||
OpenBMB ships two variants on HuggingFace:
|
||||
|
||||
- [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — base instruct model. Use this for general multimodal serving; thinking mode is still available per-request via `chat_template_kwargs.enable_thinking=true`.
|
||||
- [`openbmb/MiniCPM-V-4.6-Thinking`](https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking) — thinking-tuned variant with stronger chain-of-thought behavior. Pair with the same `--reasoning-parser qwen3` flag.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid LLM backbone**: Qwen3.5-style mix of Gated Delta Net (linear-attention) layers and full-attention layers, providing long-context efficiency without giving up modeling power.
|
||||
- **Native variable-resolution vision**: NaViT-packed vision encoder with mid-ViT merger and per-image window attention. Images of any aspect ratio are processed without forced letterboxing.
|
||||
- **High-resolution slicing**: Source image plus a configurable grid of slice tiles (up to 9 tiles in the open test variant) lets the model reason over fine detail in 1280×720+ images.
|
||||
- **Video**: Frame-by-frame multi-modal data items routed through the same vision encoder; any number of frames per request.
|
||||
- **Reasoning Parser**: switchable thinking mode (Qwen3.5 lineage), exposed via `chat_template_kwargs.enable_thinking` per request and SGLang's `--reasoning-parser qwen3` on the server side.
|
||||
- **Tool Calling**: Qwen3.5-style `<tool_call><function=…><parameter=…>…</parameter></function></tool_call>` XML format, surfaced as OpenAI-compatible `message.tool_calls` via SGLang's `--tool-call-parser qwen3_coder`. Composes with thinking mode and with image / video inputs.
|
||||
|
||||
**License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Pull the nightly Docker image (rolling tag, tracks `main`):
|
||||
|
||||
```bash
|
||||
# CUDA 13 (Hopper / Blackwell, default)
|
||||
docker pull lmsysorg/sglang:dev
|
||||
|
||||
# CUDA 12 (Ampere or older drivers)
|
||||
docker pull lmsysorg/sglang:dev-cu12
|
||||
```
|
||||
|
||||
For the general SGLang installation guide (PyPI, source, Docker) see the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate the appropriate deployment command. The `Variant` toggle switches between `openbmb/MiniCPM-V-4.6` (base) and `openbmb/MiniCPM-V-4.6-Thinking`. The `Reasoning Parser` and `Tool Call Parser` toggles add `--reasoning-parser qwen3` and `--tool-call-parser qwen3_coder` respectively; see §4.4 for usage details.
|
||||
|
||||
import { MiniCPMV46Deployment } from '/src/snippets/autoregressive/minicpm-v-4_6-deployment.jsx'
|
||||
|
||||
<MiniCPMV46Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
|
||||
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs.
|
||||
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
- Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
|
||||
- To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command.
|
||||
- **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
|
||||
- **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300.
|
||||
- For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
|
||||
- Multi-image and high-resolution images: the image processor produces one source patch plus per-slice tile patches; each is its own `MultimodalDataItem`. No special server-side flag needed.
|
||||
- Video: decoded frame-by-frame through the same image-style slicer. No extra flag needed; pass `video_url` in the OpenAI chat completion request.
|
||||
- **Chunked Prefill**: For high-concurrency vision benchmarking with many large/sliced images, pass `--chunked-prefill-size -1` to disable prefill chunking. The default chunked-prefill path can mis-split a request across an image boundary in `mm_utils.embed_mm_inputs` and crash the server; disabling chunking sidesteps this at the cost of higher TTFT under concurrency. For interactive serving leave the default on.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy the model on an H200:
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path openbmb/MiniCPM-V-4.6 \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--mem-fraction-static 0.15 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--page-size 64 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage (Image)
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="openbmb/MiniCPM-V-4.6",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.ilankelman.org/stopsigns/australia.jpg",
|
||||
},
|
||||
},
|
||||
{"type": "text", "text": "Describe this image in one sentence."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=200,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
A black SUV drives past a Chinese-style gate with a red stop sign and traditional architecture, while storefronts and street signs line the sidewalk.
|
||||
```
|
||||
|
||||
### 4.2 High-Resolution / Sliced Images
|
||||
|
||||
The image processor automatically picks a slice grid (up to 9 tiles) for high-resolution inputs. A 1280×720 source produces grid `[2, 3]`
|
||||
+ 7 patches with `tgt_sizes=[(24, 44), 6×(28, 36)]`, byte-for-byte matching the HF reference implementation.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="openbmb/MiniCPM-V-4.6",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg",
|
||||
},
|
||||
},
|
||||
{"type": "text", "text": "Describe this image in one sentence."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=200,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
The Statue of Liberty stands tall against a cloudy sky, holding a torch aloft and a document in her left hand, symbolizing freedom and enlightenment.
|
||||
```
|
||||
|
||||
### 4.3 Video Input
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="openbmb/MiniCPM-V-4.6",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {"url": "<your-video-url-or-file-path>"},
|
||||
},
|
||||
{"type": "text", "text": "Describe what happens in this video in one sentence."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=200,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example** (run against an 8-frame synthetic test mp4 of shifting colored squares):
|
||||
|
||||
```text Output
|
||||
The video shows a grid of colored squares moving in a random pattern.
|
||||
```
|
||||
|
||||
### 4.4 Advanced Usage
|
||||
|
||||
#### 4.4.1 Reasoning Parser
|
||||
|
||||
Pass `--reasoning-parser qwen3` to the server (toggle "Reasoning Parser" on in §3.1, default) so SGLang splits each response on the `<think>` / `</think>` boundaries: the pre-`</think>` block goes to `reasoning_content`, the post-`</think>` text to `content`. Per-request, the chat template's `enable_thinking` flag toggles whether the model actually emits reasoning.
|
||||
|
||||
- **Thinking mode** (default, `enable_thinking=true`): assistant prompt ends with `<think>\n`; the model writes reasoning, closes with `</think>`, then the answer. `reasoning_content` and `content` are both populated.
|
||||
- **Instruct mode** (`enable_thinking=false`): the chat template injects an empty `<think></think>` placeholder so the model emits no thinking tokens; `reasoning_content` ends up empty.
|
||||
|
||||
```python Example (thinking mode)
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="openbmb/MiniCPM-V-4.6",
|
||||
messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}],
|
||||
max_tokens=200,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("reasoning_content:", msg.reasoning_content)
|
||||
print("content :", msg.content)
|
||||
```
|
||||
|
||||
```text Output
|
||||
reasoning_content: Got it, let's see. The user wants a reply with "hi" and no explanation. So I need to just say "hi" as the response. ...
|
||||
content : hi
|
||||
```
|
||||
|
||||
```python Example (instruct mode)
|
||||
response = client.chat.completions.create(
|
||||
model="openbmb/MiniCPM-V-4.6",
|
||||
messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}],
|
||||
max_tokens=200,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("reasoning_content:", msg.reasoning_content)
|
||||
print("content :", msg.content)
|
||||
```
|
||||
|
||||
```text Output
|
||||
reasoning_content:
|
||||
content : hi
|
||||
```
|
||||
|
||||
#### 4.4.2 Tool Calling
|
||||
|
||||
Pass `--tool-call-parser qwen3_coder` to the server (toggle "Tool Call Parser" on in §3.1) so SGLang extracts `<tool_call>` blocks from the model output into the OpenAI-style `message.tool_calls` field (with `finish_reason="tool_calls"`). The model speaks the Qwen3.5 XML tool-call format (`<tool_call><function=name><parameter=k>v</parameter></function></tool_call>`); the `qwen3_coder` parser is the right one. Tool calls compose with both reasoning modes and with image / video inputs.
|
||||
|
||||
<Warning>
|
||||
Do **not** use `--tool-call-parser qwen` for MiniCPM-V 4.6 — that parser expects the older Qwen2.5 JSON format `<tool_call>{"name":..., "arguments":...}</tool_call>`, but both public 4.6 variants emit the Qwen3.5-style XML format with nested `<function=…>` and `<parameter=…>` tags. With `qwen` the outer `<tool_call>` markers match but the inner JSON parse fails, so `tool_calls` returns empty and the raw markup is left in `content`.
|
||||
</Warning>
|
||||
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="openbmb/MiniCPM-V-4.6",
|
||||
messages=[{"role": "user", "content": "What is the weather in San Francisco? Use the tool."}],
|
||||
tools=tools,
|
||||
max_tokens=200,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
choice = response.choices[0]
|
||||
print("finish_reason:", choice.finish_reason)
|
||||
for tc in choice.message.tool_calls or []:
|
||||
print(f" {tc.function.name}({tc.function.arguments})")
|
||||
```
|
||||
|
||||
```text Output
|
||||
finish_reason: tool_calls
|
||||
get_weather({"location": "San Francisco", "unit": "celsius"})
|
||||
```
|
||||
|
||||
To get the final natural-language answer, feed the tool's result back as a `tool` role message and call the API again with the same `tools` list — the model emits `finish_reason="stop"` with the answer in `content`.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
**Common Test Environment (all benchmarks below):**
|
||||
|
||||
- Hardware: 1× NVIDIA H200 (141 GB), single GPU (no TP / DP)
|
||||
- Docker Image: `lmsysorg/sglang:dev` (transformers 5.6.0, sgl-kernel 0.4.2.post1)
|
||||
- Precision: BF16
|
||||
|
||||
**Common Server Launch Command:**
|
||||
|
||||
```bash Command
|
||||
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
|
||||
--model-path openbmb/MiniCPM-V-4.6 \
|
||||
--trust-remote-code \
|
||||
--dtype bfloat16 \
|
||||
--mem-fraction-static 0.5 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--chunked-prefill-size -1 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
(`--chunked-prefill-size -1` is required for the vision throughput run; see §3.2.)
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
#### 5.1.1 MMMU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 48 --max-new-tokens 2048
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```
|
||||
{'Accounting': {'acc': 0.767, 'num': 30},
|
||||
'Agriculture': {'acc': 0.533, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.4, 'num': 30},
|
||||
'Art': {'acc': 0.6, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.667, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.533, 'num': 30},
|
||||
'Biology': {'acc': 0.333, 'num': 30},
|
||||
'Chemistry': {'acc': 0.333, 'num': 30},
|
||||
'Clinical_Medicine': {'acc': 0.467, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.333, 'num': 30},
|
||||
'Design': {'acc': 0.533, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.333, 'num': 30},
|
||||
'Economics': {'acc': 0.633, 'num': 30},
|
||||
'Electronics': {'acc': 0.5, 'num': 30},
|
||||
'Energy_and_Power': {'acc': 0.633, 'num': 30},
|
||||
'Finance': {'acc': 0.533, 'num': 30},
|
||||
'Geography': {'acc': 0.367, 'num': 30},
|
||||
'History': {'acc': 0.533, 'num': 30},
|
||||
'Literature': {'acc': 0.7, 'num': 30},
|
||||
'Manage': {'acc': 0.367, 'num': 30},
|
||||
'Marketing': {'acc': 0.733, 'num': 30},
|
||||
'Materials': {'acc': 0.367, 'num': 30},
|
||||
'Math': {'acc': 0.567, 'num': 30},
|
||||
'Mechanical_Engineering': {'acc': 0.333, 'num': 30},
|
||||
'Music': {'acc': 0.267, 'num': 30},
|
||||
'Overall': {'acc': 0.527, 'num': 900},
|
||||
'Overall-Art and Design': {'acc': 0.517, 'num': 120},
|
||||
'Overall-Business': {'acc': 0.607, 'num': 150},
|
||||
'Overall-Health and Medicine': {'acc': 0.553, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.617, 'num': 120},
|
||||
'Overall-Science': {'acc': 0.473, 'num': 150},
|
||||
'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
|
||||
'Pharmacy': {'acc': 0.667, 'num': 30},
|
||||
'Physics': {'acc': 0.767, 'num': 30},
|
||||
'Psychology': {'acc': 0.567, 'num': 30},
|
||||
'Public_Health': {'acc': 0.767, 'num': 30},
|
||||
'Sociology': {'acc': 0.667, 'num': 30}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.527
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
We use SGLang's built-in `bench_serving` tool with random text prompts (1000 input / 1000 output tokens) to characterize text-only serving performance.
|
||||
|
||||
#### 5.2.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model openbmb/MiniCPM-V-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 7.47
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3554
|
||||
Request throughput (req/s): 1.34
|
||||
Input token throughput (tok/s): 816.44
|
||||
Output token throughput (tok/s): 564.73
|
||||
Peak output token throughput (tok/s): 690.00
|
||||
Peak concurrent requests: 4
|
||||
Total token throughput (tok/s): 1381.17
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 746.20
|
||||
Median E2E Latency (ms): 590.05
|
||||
P90 E2E Latency (ms): 1446.13
|
||||
P99 E2E Latency (ms): 1709.38
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 138.12
|
||||
Median TTFT (ms): 103.70
|
||||
P99 TTFT (ms): 330.79
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.44
|
||||
Median TPOT (ms): 1.44
|
||||
P99 TPOT (ms): 1.45
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.44
|
||||
Median ITL (ms): 1.45
|
||||
P95 ITL (ms): 1.49
|
||||
P99 ITL (ms): 1.57
|
||||
Max ITL (ms): 5.79
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model openbmb/MiniCPM-V-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 47.07
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 469844
|
||||
Request throughput (req/s): 21.24
|
||||
Input token throughput (tok/s): 10675.32
|
||||
Output token throughput (tok/s): 10627.69
|
||||
Peak output token throughput (tok/s): 25911.00
|
||||
Peak concurrent requests: 130
|
||||
Total token throughput (tok/s): 21303.01
|
||||
Concurrency: 97.24
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4576.94
|
||||
Median E2E Latency (ms): 4331.97
|
||||
P90 E2E Latency (ms): 8634.07
|
||||
P99 E2E Latency (ms): 9636.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 206.50
|
||||
Median TTFT (ms): 184.72
|
||||
P99 TTFT (ms): 624.23
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.73
|
||||
Median TPOT (ms): 9.16
|
||||
P99 TPOT (ms): 13.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.75
|
||||
Median ITL (ms): 0.05
|
||||
P95 ITL (ms): 29.95
|
||||
P99 ITL (ms): 108.91
|
||||
Max ITL (ms): 448.40
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.3 Vision Speed Benchmark
|
||||
|
||||
We use SGLang's built-in `bench_serving` tool with random images. Each request has 128 input text tokens, one 720p image, and 1024 output tokens.
|
||||
|
||||
#### 5.3.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model openbmb/MiniCPM-V-4.6 \
|
||||
--dataset-name image \
|
||||
--image-count 1 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 10.26
|
||||
Total input tokens: 767
|
||||
Total input text tokens: 750
|
||||
Total input vision tokens: 17
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.97
|
||||
Input token throughput (tok/s): 74.77
|
||||
Output token throughput (tok/s): 411.39
|
||||
Peak output token throughput (tok/s): 654.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 486.16
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1024.04
|
||||
Median E2E Latency (ms): 897.99
|
||||
P90 E2E Latency (ms): 1584.25
|
||||
P99 E2E Latency (ms): 1781.78
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 416.94
|
||||
Median TTFT (ms): 403.18
|
||||
P99 TTFT (ms): 477.49
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.44
|
||||
Median TPOT (ms): 1.44
|
||||
P99 TPOT (ms): 1.45
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.44
|
||||
Median ITL (ms): 1.44
|
||||
P95 ITL (ms): 1.48
|
||||
P99 ITL (ms): 1.56
|
||||
Max ITL (ms): 2.89
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.3.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model openbmb/MiniCPM-V-4.6 \
|
||||
--dataset-name image \
|
||||
--image-count 1 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 360.01
|
||||
Total input tokens: 79925
|
||||
Total input text tokens: 78283
|
||||
Total input vision tokens: 1642
|
||||
Total generated tokens: 510855
|
||||
Total generated tokens (retokenized): 430289
|
||||
Request throughput (req/s): 2.78
|
||||
Input token throughput (tok/s): 222.01
|
||||
Output token throughput (tok/s): 1419.01
|
||||
Peak output token throughput (tok/s): 19620.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 1641.02
|
||||
Concurrency: 99.69
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 35888.57
|
||||
Median E2E Latency (ms): 35321.48
|
||||
P90 E2E Latency (ms): 41017.37
|
||||
P99 E2E Latency (ms): 60343.22
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 35096.32
|
||||
Median TTFT (ms): 34301.37
|
||||
P99 TTFT (ms): 59966.25
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.63
|
||||
Median TPOT (ms): 1.45
|
||||
P99 TPOT (ms): 10.15
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.58
|
||||
Median ITL (ms): 0.12
|
||||
P95 ITL (ms): 0.23
|
||||
P99 ITL (ms): 0.77
|
||||
Max ITL (ms): 2086.12
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,322 @@
|
||||
---
|
||||
title: Laguna-M.1
|
||||
description: "Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4."
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-M.1 support is already on SGLang `main` — **softplus per-element attention-output gating** ([PR #28400](https://github.com/sgl-project/sglang/pull/28400)) and a **global-attention fix** ([PR #28604](https://github.com/sgl-project/sglang/pull/28604), since M.1 is full-attention `sliding_window: 0`) — but not yet in a tagged release. The two paths below match the **Python / Docker** toggle in the command panel: install from `main` (Python tab), or use the **Docker** image, which bundles the same build (CUDA 13, covers H200 + all Blackwell). The model ships custom config code on the Hub, so `--trust-remote-code` is required (it is included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
# Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a
|
||||
# tagged release — install from main. The serving runtime is in the base dependencies, no extra needed:
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — `lmsysorg/sglang:latest` bundles the CUDA-13 runtime and the M.1 code.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# CUDA 13 — covers H200 + all Blackwell:
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization to generate the launch command. Laguna-M.1 ships a single **Balanced** recipe per cell — poolside's recommended operating point, a good speed/throughput trade-off for typical multi-user serving. The 8-GPU HGX platforms (H200 / B200 / B300) use `--tp 8`; the 4-GPU Grace-Blackwell single nodes (GB200 / GB300) use `--tp 4`.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-m1.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-m1-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs (parsers, DP-Attention, DeepEP / EP) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-M.1](https://huggingface.co/poolside/Laguna-M.1) is an open-weight, **225B-parameter** Mixture-of-Experts model (**23B activated per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software-engineering work. It is released under Apache 2.0.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts, top-16 routing** (+1 shared expert) and auxiliary-loss-free load balancing.
|
||||
- **Global attention with output gating**: global attention across all layers, 64 Q-heads / 8 KV-heads (head dim 128), with **softplus attention output gating** (requires [PR #28400](https://github.com/sgl-project/sglang/pull/28400)).
|
||||
- **Long context**: 262,144 tokens, RoPE with YaRN.
|
||||
- **Agentic coding**: competitive on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0.
|
||||
- **Native reasoning**: interleaved thinking between tool calls, toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.
|
||||
|
||||
**Available Quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "80%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Quantization</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Hugging Face path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1`](https://huggingface.co/poolside/Laguna-M.1)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1-FP8`](https://huggingface.co/poolside/Laguna-M.1-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1-NVFP4`](https://huggingface.co/poolside/Laguna-M.1-NVFP4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-M.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Trust remote code** (`--trust-remote-code`): Laguna-M.1 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model.
|
||||
- **Long-context memory**: M.1 is global-attention (no sliding-window), so the 262,144-token KV cache is large. If you hit OOM at full context, lower `--mem-fraction-static` or cap `--context-length`.
|
||||
- **FP8**: On **Blackwell** the recipe adds `--fp8-gemm-backend triton` — the compressed-tensors block-FP8 weight scales aren't UE8M0-packed, so the default DeepGEMM path emits garbage on Blackwell (sm_100); the Triton backend is correct (~19% slower). Temporary workaround pending [PR #28662](https://github.com/sgl-project/sglang/pull/28662) (which fixes the scales and restores the DeepGEMM fast path). On **Hopper (H200)** FP8 uses DeepGEMM with no extra flag — pre-warm its multi-session JIT with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-M.1-FP8` to avoid paying it on each restart.
|
||||
- **Parsers** (`poolside_v1`): for agentic / tool-using deployments enable the **Reasoning Parser** and **Tool Call Parser** in the Playground above — they emit `--reasoning-parser poolside_v1` (thinking → `reasoning_content`) and `--tool-call-parser poolside_v1` (structured `tool_calls`).
|
||||
- **Thinking default**: thinking is **off by default**; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
- **Served model id**: the server registers the model under whatever you pass to `--model-path`, so a client's `model` field must match it — `poolside/Laguna-M.1` (BF16) or `poolside/Laguna-M.1-FP8` / `-NVFP4` for the quantized cells. The §3 examples use the BF16 id; swap in the id you launched.
|
||||
- **Recommended sampling**: poolside benchmarks M.1 at `temperature=1.0`, `top_k=20` with thinking enabled. These are per-request sampling params (not launch flags) — e.g. `temperature=1.0, extra_body={"top_k": 20}` on the OpenAI client.
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (or toggle **Reasoning Parser** in the **Parsers** card of the Playground above). Reasoning is **opt-in**: the Laguna chat template gates it on `enable_thinking=True` (passed via `chat_template_kwargs`) — the generic `thinking` key is ignored. The `<think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-M.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
=============== Reasoning ===============
|
||||
Okay, so I need to find out what 15% of 240 is. Hmm, percentages can sometimes be
|
||||
tricky, but let me think. I remember that "percent" means per hundred, right? So 15%
|
||||
is the same as 15 per 100 or 15/100. Maybe I can convert that percentage into a decimal
|
||||
first? ... 15 divided by 100 is 0.15. ... Now, to find 15% of 240, I just need to
|
||||
multiply 240 by 0.15. ... 240 times 0.1 is 24 (10% of 240), and 240 times 0.05 is 12
|
||||
(half of that), so 24 + 12 = 36.
|
||||
[… verifies the same result several more ways: 15/100 × 240, 240 × 15 ÷ 100,
|
||||
1% × 15, and the fraction 3/20 × 240 — all give 36 …]
|
||||
So ... all methods are pointing to 36. I'm pretty confident that 15% of 240 is 36.
|
||||
=============== Answer ==================
|
||||
To find 15% of 240, convert the percentage to a decimal (0.15) and multiply by 240:
|
||||
**240 × 0.15 = 36**.
|
||||
|
||||
**Step-by-Step Explanation:**
|
||||
1. **Convert 15% to a decimal:** 15% = 15/100 = 0.15.
|
||||
2. **Multiply by 240:**
|
||||
- Break it down:
|
||||
- 10% of 240 = 24 (since 240 × 0.1 = 24).
|
||||
- 5% of 240 = 12 (half of 24).
|
||||
- Add them: 24 + 12 = **36**.
|
||||
|
||||
**Answer:** 15% of 240 is **36**.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
Laguna-M.1's reasoning traces are long — the model explores and re-verifies an answer
|
||||
multiple ways. Give it a generous `max_tokens` for harder problems (reasoning regularly
|
||||
exceeds 3k tokens). The trace above is abbreviated; the model emits it in full.
|
||||
</Note>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (or toggle **Tool Call Parser** in the **Parsers** card of the Playground above). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (`enable_thinking=False`, the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-M.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 Prefill-Decode (PD) Disaggregation
|
||||
|
||||
[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. Laguna-M.1 is **global-attention with a standard KV cache** (no sliding window, no sparse "index" side-buffer), so its KV pages transfer with **no model-specific flags** — just the `--disaggregation-*` knobs. Both roles auto-select the same attention backend (FlashAttention-3) and page size because they share the model and flags, so the KV layout lines up for transfer.
|
||||
|
||||
**Supported / validated topology:**
|
||||
|
||||
- **Equal tensor parallelism** — prefill and decode run the same `--tp`.
|
||||
- **Single pipeline stage** — PP = 1 (the default).
|
||||
- **mooncake or NIXL** transfer backend over RDMA / InfiniBand.
|
||||
- Validated on **2 × 8×H200** (TP8 prefill + TP8 decode, BF16), one node each, over an 8× 400 Gb/s NDR InfiniBand fabric.
|
||||
|
||||
Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Point `--disaggregation-ib-device` at your RDMA NIC(s).
|
||||
|
||||
```bash Prefill server (node A)
|
||||
sglang serve \
|
||||
--model-path poolside/Laguna-M.1 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--tp 8 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--disaggregation-bootstrap-port 8998
|
||||
```
|
||||
|
||||
```bash Decode server (node B)
|
||||
sglang serve \
|
||||
--model-path poolside/Laguna-M.1 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--tp 8 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:
|
||||
|
||||
```bash PD router
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://<prefill-host>:30000 8998 \
|
||||
--decode http://<decode-host>:30001 \
|
||||
--policy round_robin \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Clients hit the router exactly like a single server — it splits each request across the two stages transparently:
|
||||
|
||||
<Accordion title="PD Client Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-M.1",
|
||||
messages=[{"role": "user", "content": "What is 2 + 2?"}],
|
||||
max_tokens=64,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
2 + 2 = 4
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
**Transfer backend — mooncake (recommended).** mooncake honors `--disaggregation-ib-device` and establishes its RDMA connection at registration, so the **first request is already fast** (no cold start). It works with a single NIC or all eight; using **all 8 NICs lowers TTFT** (more aggregate bandwidth for the KV payload — the gap widens at longer context). On 8×H200 (random isl=512 / osl=256, 16 concurrent) it served ≈ **717 tok/s** output (≈ 2.2k tok/s total), mean **TTFT 244 ms**, mean **TPOT 17.7 ms**; with a single `mlx5_0` NIC, ≈ 697 tok/s and TTFT 287 ms (TPOT unchanged — decode is compute-bound).
|
||||
|
||||
**Transfer backend — NIXL (works, with two caveats).**
|
||||
|
||||
<Warning>
|
||||
The NIXL path **ignores `--disaggregation-ib-device`** — that flag is mooncake-only. NIXL uses its UCX backend, whose NIC is selected by the **`UCX_NET_DEVICES`** environment variable. **Set it** (e.g. `export UCX_NET_DEVICES=mlx5_0:1`) on both servers; without it UCX cannot establish a working cross-node path and every KV transfer hangs until it hits the 300 s timeout (`Request … timed out … in KVPoll.WaitingForInput`) and returns a 500.
|
||||
</Warning>
|
||||
|
||||
With `UCX_NET_DEVICES` pinned, NIXL matches mooncake on quality and steady-state speed (≈ 720 tok/s, TTFT 230 ms, TPOT 17.7 ms). One difference: the **first request after launch pays a ~38 s one-time UCX connection cold-start** (a single port or all eight behave the same). Warm the path with one throwaway request after startup, or raise `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` (default 300 s) so the first real request isn't dropped while UCX connects.
|
||||
|
||||
**Validation.** PD disaggregation preserves output quality — disaggregated output matches non-disaggregated serving, and GSM8K (no-thinking, 200-question subset via the router) scored **0.945** (mooncake, 8 NICs) / **0.940** (NIXL) / **0.950** (mooncake, 1 NIC), all with 100% stop-rate and 0% errors — in line with single-node BF16 (≈ 0.93 on the full split). Logs confirm the split: the prefill node logs `Prefill batch` (CUDA graph off), the decode node logs `Decode batch` (CUDA graph on).
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
title: Laguna-S-2.1
|
||||
description: "Deploy poolside's Laguna-S-2.1 — a 118B hybrid-SWA Mixture-of-Experts model (8B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-S-2.1 uses the same `laguna` model architecture as [Laguna-XS-2.1](./Laguna-XS-2.1), which is fully supported in SGLang `main`. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:
|
||||
|
||||
- **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
|
||||
- **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.
|
||||
|
||||
On the 8-GPU HGX platforms (H200 / B300) all quantizations run `--tp 8`. The 4-GPU GB300 node runs `--tp 4` throughout. NVFP4 is Blackwell-only (B300 / GB300 only).
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-s21.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-s21-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is an open-weight **118B-parameter** hybrid sliding-window-attention MoE model (**~8B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering. It sits between [Laguna XS 2.1](./Laguna-XS-2.1) (33B/3B active) and Laguna M.1 (222B/23B active) in the Laguna family.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Sparse MoE**: 48 layers, 256 routed experts, top-10 routing, plus 1 shared expert.
|
||||
- **Hybrid attention**: 36 sliding-window layers (window 512) interleaved with 12 full-attention layers (1:3 global-to-SWA ratio); 8 KV heads, head dim 128; per-head sigmoid output gating with per-layer-type rotary scales.
|
||||
- **Long context**: 1,048,576 tokens.
|
||||
- **DFlash drafts**: matched draft models ship per quantization for low-latency serving.
|
||||
- **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.
|
||||
|
||||
**Available quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Target model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Draft model</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1`](https://huggingface.co/poolside/Laguna-S-2.1)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-FP8)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>INT4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-INT4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The drafts are small BF16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).
|
||||
|
||||
**License:** [OpenMDW-1.1](https://openmdw.ai/)
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-S-2.1) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Attention backend**
|
||||
|
||||
Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (reproduced on Laguna-XS-2.1, greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Other attention backend choices have not been fully validated on Laguna; keep the default.
|
||||
|
||||
**BF16 memory on H200**
|
||||
|
||||
BF16 on H200 leaves less headroom for CUDA-graph capture and NCCL allocations than FP8/INT4. The High-throughput BF16 command carries `--mem-fraction-static 0.80`. FP8, INT4, and all B300/GB300 cells use the default heuristic.
|
||||
|
||||
**FP8 shared expert**
|
||||
|
||||
`SGLANG_SHARED_EXPERT_TP1=1` is required for FP8 cells on **all hardware** — confirmed on both H200 (TP=8) and GB300 (TP=4). The FP8 checkpoint block-quantizes the shared expert (128×128 scales), which cannot TP-shard cleanly at either TP degree on S-2.1. This env var replicates the shared expert instead of sharding it. INT4 keeps the shared expert in BF16 (no flag needed); BF16 is unquantized. Note: this differs from Laguna-XS-2.1 where TP=4 does not require the flag — the constraint is architecture-specific.
|
||||
|
||||
**FP8 and NVFP4 DFlash drafts**
|
||||
|
||||
Fixed upstream on 2026-07-21: all DFlash draft configs now use a flat top-level `rope_theta` (the `rope_parameters` block was removed). If a server crashes at draft-model load with `KeyError: 'rope_theta'`, you are serving a draft checkpoint cached before 2026-07-21 — re-download it (e.g. `hf download poolside/Laguna-S-2.1-DFlash-FP8`) to pick up the corrected config.
|
||||
|
||||
**DFlash memory**
|
||||
|
||||
Low-latency cells carry `--mem-fraction-static 0.7` (sufficient even for BF16 on H200). Dense cells use the default heuristic (except BF16 on H200 — see above).
|
||||
|
||||
**BF16 reasoning length**
|
||||
|
||||
BF16 reasons approximately 2× longer than FP8/INT4 on AIME25 (median 34.8 k vs 16.9 k tokens), consistently truncating at `max_tokens=64000`. FP8/INT4 truncate at ≈ 2%. For a valid BF16 AIME25 score, serve with `max_tokens ≥ 131072` (the model supports a 1 M context window).
|
||||
|
||||
**Chat template**
|
||||
|
||||
On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ ~5.8) pass `--chat-template <model-dir>/chat_template.jinja` explicitly.
|
||||
|
||||
**Thinking**
|
||||
|
||||
Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.
|
||||
|
||||
**Served model id**
|
||||
|
||||
The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-S-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 DFlash Speculative Decoding
|
||||
|
||||
DFlash is a block-wise speculative decoder: the draft proposes a block of tokens and the target verifies the whole block in one forward pass — output quality is the target's by construction. The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step.
|
||||
|
||||
Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. The generated commands always pair the draft calibrated for the selected target precision.
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-S-2.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=4096,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
Give generous `max_tokens` when thinking is enabled — hard problems regularly reason
|
||||
for thousands of tokens. Keep thinking off for short-form tasks.
|
||||
</Note>
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-S-2.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
title: Laguna-XS-2.1
|
||||
description: "Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-XS-2.1 support is fully merged to SGLang `main` ([PR #29446](https://github.com/sgl-project/sglang/pull/29446): DFlash speculative decoding + shared-expert fix; [PR #29761](https://github.com/sgl-project/sglang/pull/29761): INT4 loader fix). Any build at or past their merge covers every cell below.
|
||||
|
||||
The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:
|
||||
|
||||
- **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
|
||||
- **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.
|
||||
|
||||
On the 8-GPU HGX platforms (H200 / B300), BF16 and NVFP4 run plain `--tp 8`; FP8 and INT4 run `--tp 8 --ep-size 8` because their quantization scales cannot shard the MoE 8-way (see [Configuration Tips](#2-configuration-tips)). The 4-GPU GB300 node runs plain `--tp 4` throughout.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-xs21.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-xs21-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is an open-weight **33B-parameter** hybrid sliding-window-attention MoE model (**~3B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering — the extra-small sibling of [Laguna-M.1](./Laguna-M.1).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Sparse MoE**: 40 layers, 256 routed experts, top-8 routing.
|
||||
- **Hybrid attention**: 30 sliding-window layers (window 512) interleaved with 10 full-attention layers; 48 Q / 8 KV heads.
|
||||
- **Long context**: 262,144 tokens (RoPE + YaRN on the full-attention layers).
|
||||
- **DFlash drafts**: matched draft models (5-layer, ~0.9 GB) ship per quantization for low-latency serving.
|
||||
- **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.
|
||||
|
||||
**Available quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Target model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Draft model</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1`](https://huggingface.co/poolside/Laguna-XS-2.1)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-FP8)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>INT4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-INT4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The drafts themselves are small bf16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-XS-2.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [API platform](https://platform.poolside.ai).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Attention backend**
|
||||
|
||||
Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Never use `triton` attention with Laguna (GSM8K 13%).
|
||||
|
||||
**Quantized checkpoints cap plain TP at 4**
|
||||
|
||||
`moe_intermediate_size=512` with FP8 block `[128,128]` / INT4 `group_size=128` scales cannot shard 8-way (512/8 = 64 < 128 granularity): FP8 fails at weight creation, INT4 crashes in the Marlin kernel, on any hardware. The generated 8-GPU FP8/INT4 commands therefore use `--tp 8 --ep-size 8` — expert parallelism keeps whole experts per rank, using all 8 GPUs on one instance. FP8 additionally needs `SGLANG_SHARED_EXPERT_TP1=1` (its shared expert is also block-quantized; INT4's stays bf16). Alternatives: plain `--tp 4`, or `--tp 4 --dp-size 2`. Accuracy is parallelism-independent within eval noise (verified tp1 ≡ tp4 on GB300 and tp4 ≡ tp8+ep8 on H200).
|
||||
|
||||
**DFlash memory**
|
||||
|
||||
Low-latency cells carry `--mem-fraction-static 0.7`: the default fraction OOMs in the draft vocab all-gather at `tp 4` on GB300. Dense cells use the default heuristic.
|
||||
|
||||
**INT4 is mixed-precision**
|
||||
|
||||
The INT4 checkpoint quantizes MoE layers in mixed 4-bit / 8-bit config groups. Builds older than [PR #29761](https://github.com/sgl-project/sglang/pull/29761) crash at load with `KeyError: 'Linear'`.
|
||||
|
||||
**Chat template**
|
||||
|
||||
On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ ~5.8) the `{% include %}` stub in `tokenizer_config.json` cannot resolve and the server silently falls back to a generic template — pass `--chat-template <model-dir>/chat_template.jinja` explicitly there.
|
||||
|
||||
**Thinking**
|
||||
|
||||
Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.
|
||||
|
||||
**Served model id**
|
||||
|
||||
The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-XS-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 DFlash Speculative Decoding
|
||||
|
||||
DFlash is a block-wise speculative decoder: the 5-layer draft proposes a block of tokens and the target verifies the whole block in one forward pass, so only target-approved tokens are emitted — output quality is the target's by construction (GSM8K matches dense within noise on every quantization). The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step:
|
||||
|
||||
- Measured ~6 tokens/step at `tp 1`, ~4 at `tp 4` (greedy GSM8K, matched-precision pairs; ~3 under mixed reasoning-heavy traffic; FP8 reached 6.75 at `tp 8 + ep 8` on H200) — versus 1 token/step dense.
|
||||
- Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput.
|
||||
- The generated commands always pair the draft calibrated for the selected target precision.
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
XS-2.1 is an extra-small model — give it generous `max_tokens` when thinking is enabled
|
||||
(hard problems regularly reason for thousands of tokens), and keep thinking off for
|
||||
short-form tasks.
|
||||
</Note>
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,323 @@
|
||||
---
|
||||
title: Laguna-XS.2
|
||||
metatags:
|
||||
description: "Deploy Poolside's Laguna-XS.2 hybrid SWA + MoE model with SGLang on NVIDIA H200 / B200 — agentic coding with hybrid reasoning and tool calling."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) is an open-source hybrid sliding-window-attention MoE model from [Poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering work.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **MoE**: 33.4B total parameters, 3.0B active per token, 256 routed experts (top-8) plus 1 shared.
|
||||
- **Long context**: 131,072 tokens.
|
||||
- **Agentic coding**: Tuned for tool-using software engineering agents and long-horizon execution.
|
||||
- **Hybrid reasoning**: `<think>...</think>` segments toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.
|
||||
|
||||
**Available Quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "80%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Hugging Face path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[`poolside/Laguna-XS.2`](https://huggingface.co/poolside/Laguna-XS.2)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[`poolside/Laguna-XS.2-FP8`](https://huggingface.co/poolside/Laguna-XS.2-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[`poolside/Laguna-XS.2-NVFP4`](https://huggingface.co/poolside/Laguna-XS.2-NVFP4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
For details, see the [Hugging Face model card](https://huggingface.co/poolside/Laguna-XS.2) and the [Laguna deeper-dive blog post](https://poolside.ai/blog/laguna-a-deeper-dive).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Laguna-XS.2 support is on `main` but not yet in a tagged release; install from the SGLang nightly wheel index, or pull a pre-built Docker image:
|
||||
|
||||
```bash Command
|
||||
# Install SGLang via pip (CUDA 13) — requires Python 3.10 (nightly wheels are cp310 only)
|
||||
python3 -m pip install --upgrade pip
|
||||
python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu130 \
|
||||
"sglang[all]==0.5.12.dev20260509+g096ad02b0"
|
||||
|
||||
# CUDA 12: swap to the cu129 index
|
||||
python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu129 \
|
||||
"sglang[all]==0.5.12.dev20260509+g096ad02b0"
|
||||
|
||||
# Or use Docker (multi-arch amd64/arm64; CUDA 13, H200 / B200)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for your hardware.
|
||||
|
||||
import { LagunaXS2Deployment } from '/src/snippets/autoregressive/laguna-xs2-deployment.jsx';
|
||||
|
||||
<LagunaXS2Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Trust remote code** (`--trust-remote-code`): Laguna-XS.2 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model.
|
||||
- **Quantization**: NVFP4 requires Blackwell (B200 / B300); BF16 and FP8 run on either H200 or B200. FP8's first launch triggers a multi-session DeepGEMM JIT pre-compile (~10-20 min); pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-XS.2-FP8` to avoid that cost on every restart.
|
||||
- **Reasoning parser** (`--reasoning-parser poolside_v1`): Splits `<think>...</think>` segments into `reasoning_content` so `content` holds only the final answer. Disable only if you want the raw `<think>` tags in `content`.
|
||||
- **Tool call parser** (`--tool-call-parser poolside_v1`): Required for OpenAI-compatible tool-call streaming. Disable only for chat-only deployments.
|
||||
- **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--dp <N> --enable-dp-attention` with `--dp` matching `--tp` (tune independently if needed).
|
||||
- **Thinking default**: Thinking is **off by default** at the model level. Opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
The samples below assume the server is reachable at `http://localhost:30000/v1`.
|
||||
|
||||
### 4.1 Basic Chat
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is the difference between TCP and UDP?"}
|
||||
],
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are two core protocols of the Internet Protocol (IP) suite, both used for network communication but with key differences:
|
||||
|
||||
## Connection Handling
|
||||
- **TCP**: Connection-oriented protocol that establishes a connection before data transfer (like a phone call)
|
||||
- **UDP**: Connectionless protocol that sends data without establishing a connection (like sending a letter)
|
||||
|
||||
## Reliability
|
||||
- **TCP**: Guaranteed delivery with error checking, retransmission of lost packets, and flow control
|
||||
- **UDP**: No guarantee of delivery; packets may be lost, duplicated, or arrive out of order
|
||||
|
||||
## Speed & Overhead
|
||||
- **TCP**: Slower due to connection setup, acknowledgment overhead, and error correction mechanisms
|
||||
- **UDP**: Faster with minimal overhead since it doesn't wait for acknowledgments or retransmit lost data
|
||||
|
||||
## Use Cases
|
||||
- **TCP**: Web browsing (HTTP/HTTPS), email (SMTP), file transfers (FTP), database connections
|
||||
- **UDP**: Video streaming, online gaming, VoIP calls, DNS queries, live broadcasts
|
||||
|
||||
In essence, TCP prioritizes reliability over speed, while UDP prioritizes speed over reliability.
|
||||
```
|
||||
|
||||
### 4.2 Reasoning (Thinking Mode)
|
||||
|
||||
Laguna-XS.2 emits reasoning between `<think>...</think>` tags. The `--reasoning-parser poolside_v1` flag separates the thinking text into `reasoning_content` so `content` holds only the final answer. Thinking is opt-in per request:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[
|
||||
{"role": "user", "content": "If a train travels at 60 km/h for 2.5 hours, how far does it go?"}
|
||||
],
|
||||
max_tokens=4096,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
print("====== Reasoning Content ======")
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
print("====== Answer ======")
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
====== Reasoning Content ======
|
||||
The user is asking a straightforward math problem about distance, speed, and time. I need to calculate the distance using the formula:
|
||||
|
||||
Distance = Speed × Time
|
||||
|
||||
Given:
|
||||
- Speed = 60 km/h
|
||||
- Time = 2.5 hours
|
||||
|
||||
So the calculation would be:
|
||||
Distance = 60 × 2.5 = 150 km
|
||||
|
||||
This is a simple multiplication problem. I should provide a clear, direct answer and maybe explain the calculation briefly.
|
||||
|
||||
====== Answer ======
|
||||
To find the distance, use the formula:
|
||||
|
||||
Distance = Speed × Time
|
||||
Distance = 60 km/h × 2.5 h = 150 km
|
||||
|
||||
The train travels **150 kilometers**.
|
||||
```
|
||||
|
||||
To disable thinking, omit `extra_body` (off by default) or pass `chat_template_kwargs={"enable_thinking": False}` explicitly.
|
||||
|
||||
### 4.3 Tool Calling
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
msg = resp.choices[0].message
|
||||
print("====== Reasoning Content ======")
|
||||
print(msg.reasoning_content)
|
||||
print("====== Content ======")
|
||||
print(msg.content)
|
||||
print("====== Tool Calls ======")
|
||||
for tc in msg.tool_calls or []:
|
||||
print(f" Function: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
====== Reasoning Content ======
|
||||
None
|
||||
====== Content ======
|
||||
|
||||
I'll check the current weather in Tokyo for you.
|
||||
|
||||
====== Tool Calls ======
|
||||
Function: get_weather
|
||||
Arguments: {"location": "Tokyo"}
|
||||
```
|
||||
|
||||
`reasoning_content` is `None` because thinking is off by default; `content` carries the brief assistant message that precedes the tool call. Add `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` if you want interleaved reasoning before the tool call.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 (4×H200)
|
||||
- Model: `poolside/Laguna-XS.2` (BF16)
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs)
|
||||
- Reasoning Parser: `poolside_v1`
|
||||
- Tool Call Parser: `poolside_v1`
|
||||
- Sampling: `temperature=0.6`, `max_tokens=16384`, `chat_template_kwargs={"enable_thinking": true}`, `n_repeats=1`
|
||||
- Grader: NeMo-Skills `math_verify` (math) and `eval_mcq` (multichoice)
|
||||
|
||||
**Results (from [PR #24204](https://github.com/sgl-project/sglang/pull/24204)):**
|
||||
|
||||
| Eval | Accuracy |
|
||||
| --- | ---: |
|
||||
| GPQA Diamond | 0.5556 |
|
||||
| AIME 25 | 0.5667 |
|
||||
| MMLU | 0.836 |
|
||||
| SWE-Bench Verified | 0.6540 |
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 (1×H200 for TP=1, 4×H200 for TP=4)
|
||||
- Model: `poolside/Laguna-XS.2` (BF16)
|
||||
- SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs)
|
||||
- Workload: `sglang.bench_serving --backend sglang --dataset-name random` (defaults: `--random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.0`)
|
||||
- Server flags identical to the accuracy runs above.
|
||||
|
||||
#### 5.2.1 Latency Benchmark (10 prompts, concurrency = 1)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving --backend sglang \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--dataset-name random --num-prompts 10 --max-concurrency 1
|
||||
```
|
||||
|
||||
| Metric | TP=1 | TP=4 |
|
||||
| --- | ---: | ---: |
|
||||
| Successful requests | 10 | 10 |
|
||||
| Output token throughput (tok/s) | 193.10 | 238.88 |
|
||||
| Total token throughput (tok/s) | 471.82 | 583.68 |
|
||||
| Mean TTFT (ms) | 35.32 | 24.17 |
|
||||
| Mean TPOT (ms) | 5.10 | 4.13 |
|
||||
| Median ITL (ms) | 5.14 | 4.14 |
|
||||
|
||||
#### 5.2.2 Throughput Benchmark (1000 prompts, concurrency = 100)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving --backend sglang \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--dataset-name random --num-prompts 1000 --max-concurrency 100
|
||||
```
|
||||
|
||||
| Metric | TP=1 | TP=4 |
|
||||
| --- | ---: | ---: |
|
||||
| Successful requests | 1000 | 1000 |
|
||||
| Request throughput (req/s) | 7.32 | 14.61 |
|
||||
| Output token throughput (tok/s) | 3739.30 | 7465.18 |
|
||||
| Peak output token throughput (tok/s) | 4718.00 | 10133.00 |
|
||||
| Total token throughput (tok/s) | 7485.82 | 14944.81 |
|
||||
| Mean TTFT (ms) | 115.17 | 68.36 |
|
||||
| Mean TPOT (ms) | 25.51 | 12.71 |
|
||||
| Median ITL (ms) | 21.31 | 10.64 |
|
||||
|
||||
TP=4 delivers roughly 2.0× total-token throughput and ~1.7× lower mean TTFT compared to TP=1 on the `cc=100` random workload.
|
||||
@@ -0,0 +1,395 @@
|
||||
---
|
||||
title: Qwen2.5-VL
|
||||
metatags:
|
||||
description: "Deploy Qwen2.5-VL vision-language models with SGLang on AMD MI300X - available in 3B to 72B sizes with enhanced visual understanding."
|
||||
---
|
||||
|
||||
import { Qwen25VLDeployment } from '/src/snippets/autoregressive/qwen25-vl-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images.
|
||||
- **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use.
|
||||
- **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments.
|
||||
- **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes.
|
||||
- **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc.
|
||||
- **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments.
|
||||
- **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs.
|
||||
- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
|
||||
|
||||
For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X as well as Intel Xeon CPU hardware platforms and different use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs as well as Intel Xeon CPUs.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size.
|
||||
|
||||
<Qwen25VLDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
* **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed.
|
||||
* **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X.
|
||||
* **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
Qwen2.5-VL supports image inputs. Here's a basic example with single image input:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Read all the text in the image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen2.5-VL-7B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 2.31s
|
||||
Generated text: Auntie Anne's
|
||||
|
||||
CINNAMON SUGAR
|
||||
1 x 17,000
|
||||
SUB TOTAL
|
||||
17,000
|
||||
|
||||
GRAND TOTAL
|
||||
17,000
|
||||
|
||||
CASH IDR
|
||||
20,000
|
||||
|
||||
CHANGE DUE
|
||||
3,000
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
Qwen2.5-VL can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen2.5-VL-7B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 13.79s
|
||||
Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area.
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- You can also provide local file paths using `file://` protocol.
|
||||
- For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (8x)
|
||||
- Model: Qwen2.5-VL-72B-Instruct
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 37.99
|
||||
Total input tokens: 24781
|
||||
Total input text tokens: 821
|
||||
Total input vision tokens: 23960
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 2365
|
||||
Request throughput (req/s): 0.26
|
||||
Input token throughput (tok/s): 652.26
|
||||
Output token throughput (tok/s): 111.07
|
||||
Peak output token throughput (tok/s): 128.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 763.34
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3797.61
|
||||
Median E2E Latency (ms): 3140.90
|
||||
P90 E2E Latency (ms): 6545.54
|
||||
P99 E2E Latency (ms): 7939.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 504.45
|
||||
Median TTFT (ms): 510.93
|
||||
P99 TTFT (ms): 521.78
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.82
|
||||
Median TPOT (ms): 7.82
|
||||
P99 TPOT (ms): 7.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.07
|
||||
Median ITL (ms): 7.90
|
||||
P95 ITL (ms): 15.79
|
||||
P99 ITL (ms): 15.93
|
||||
Max ITL (ms): 23.60
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 454.68
|
||||
Total input tokens: 2481865
|
||||
Total input text tokens: 85865
|
||||
Total input vision tokens: 2396000
|
||||
Total generated tokens: 510855
|
||||
Total generated tokens (retokenized): 296466
|
||||
Request throughput (req/s): 2.20
|
||||
Input token throughput (tok/s): 5458.50
|
||||
Output token throughput (tok/s): 1123.55
|
||||
Peak output token throughput (tok/s): 5004.00
|
||||
Peak concurrent requests: 106
|
||||
Total token throughput (tok/s): 6582.05
|
||||
Concurrency: 98.63
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 44844.92
|
||||
Median E2E Latency (ms): 42866.15
|
||||
P90 E2E Latency (ms): 82798.20
|
||||
P99 E2E Latency (ms): 106306.30
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4507.79
|
||||
Median TTFT (ms): 1180.83
|
||||
P99 TTFT (ms): 39975.22
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 80.26
|
||||
Median TPOT (ms): 82.38
|
||||
P99 TPOT (ms): 152.89
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 100.66
|
||||
Median ITL (ms): 13.26
|
||||
P95 ITL (ms): 428.45
|
||||
P99 ITL (ms): 1393.35
|
||||
Max ITL (ms): 31943.26
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 MMMU Benchmark
|
||||
|
||||
You can evaluate the model's accuracy using the MMMU dataset:
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 benchmark/mmmu/bench_sglang.py \
|
||||
--port 30000 \
|
||||
--concurrency 64
|
||||
```
|
||||
```text Output
|
||||
Benchmark time: 97.75084622902796
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.633, 'num': 30},
|
||||
'Agriculture': {'acc': 0.5, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.367, 'num': 30},
|
||||
'Art': {'acc': 0.767, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.7, 'num': 30},
|
||||
'Biology': {'acc': 0.467, 'num': 30},
|
||||
'Chemistry': {'acc': 0.433, 'num': 30},
|
||||
'Clinical_Medicine': {'acc': 0.733, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.567, 'num': 30},
|
||||
'Design': {'acc': 0.833, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30},
|
||||
'Economics': {'acc': 0.767, 'num': 30},
|
||||
'Electronics': {'acc': 0.433, 'num': 30},
|
||||
'Energy_and_Power': {'acc': 0.467, 'num': 30},
|
||||
'Finance': {'acc': 0.533, 'num': 30},
|
||||
'Geography': {'acc': 0.633, 'num': 30},
|
||||
'History': {'acc': 0.7, 'num': 30},
|
||||
'Literature': {'acc': 0.867, 'num': 30},
|
||||
'Manage': {'acc': 0.633, 'num': 30},
|
||||
'Marketing': {'acc': 0.733, 'num': 30},
|
||||
'Materials': {'acc': 0.333, 'num': 30},
|
||||
'Math': {'acc': 0.533, 'num': 30},
|
||||
'Mechanical_Engineering': {'acc': 0.433, 'num': 30},
|
||||
'Music': {'acc': 0.367, 'num': 30},
|
||||
'Overall': {'acc': 0.62, 'num': 900},
|
||||
'Overall-Art and Design': {'acc': 0.717, 'num': 120},
|
||||
'Overall-Business': {'acc': 0.66, 'num': 150},
|
||||
'Overall-Health and Medicine': {'acc': 0.693, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120},
|
||||
'Overall-Science': {'acc': 0.553, 'num': 150},
|
||||
'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
|
||||
'Pharmacy': {'acc': 0.833, 'num': 30},
|
||||
'Physics': {'acc': 0.7, 'num': 30},
|
||||
'Psychology': {'acc': 0.767, 'num': 30},
|
||||
'Public_Health': {'acc': 0.733, 'num': 30},
|
||||
'Sociology': {'acc': 0.767, 'num': 30}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.62
|
||||
```
|
||||
@@ -0,0 +1,905 @@
|
||||
---
|
||||
title: Qwen3-Coder-Next
|
||||
metatags:
|
||||
description: "Deploy Qwen3-Coder-Next code-focused models with SGLang on AMD MI300X - available in 3B to 80B sizes with enhanced code understanding."
|
||||
---
|
||||
|
||||
import { Qwen3CoderNextDeployment } from '/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency.
|
||||
- **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser.
|
||||
- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
|
||||
- **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale.
|
||||
- **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms.
|
||||
|
||||
For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options.
|
||||
|
||||
<Qwen3CoderNextDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`.
|
||||
- **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag.
|
||||
- **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed.
|
||||
- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
|
||||
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
|
||||
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
|
||||
- **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--tp 2 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Code Generation Example
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Coder-Next",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."}
|
||||
],
|
||||
max_tokens=2048
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
````text Output
|
||||
Here's a Python function implementing binary search on a sorted list, with comprehensive type hints:
|
||||
|
||||
```python
|
||||
from typing import Sequence, TypeVar, Optional
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]:
|
||||
"""
|
||||
Perform binary search on a sorted list to find the index of a target element.
|
||||
|
||||
Args:
|
||||
sorted_list: A sequence (e.g., list, tuple) sorted in ascending order.
|
||||
target: The element to search for in the list.
|
||||
|
||||
Returns:
|
||||
The index of the target element if found, or None if not found.
|
||||
|
||||
Time Complexity: O(log n)
|
||||
Space Complexity: O(1)
|
||||
|
||||
Note:
|
||||
The function assumes the list is sorted in ascending order.
|
||||
If the list contains duplicate elements, it returns the index of one of them.
|
||||
"""
|
||||
left = 0
|
||||
right = len(sorted_list) - 1
|
||||
|
||||
while left <= right:
|
||||
mid = (left + right) // 2
|
||||
mid_val = sorted_list[mid]
|
||||
|
||||
if mid_val == target:
|
||||
return mid
|
||||
elif mid_val < target:
|
||||
left = mid + 1
|
||||
else:
|
||||
right = mid - 1
|
||||
|
||||
return None
|
||||
```
|
||||
|
||||
### Example usage:
|
||||
|
||||
```python
|
||||
# Example 1: Finding an existing element
|
||||
numbers = [1, 3, 5, 7, 9, 11]
|
||||
print(binary_search(numbers, 7)) # Output: 3
|
||||
|
||||
# Example 2: Element not in the list
|
||||
print(binary_search(numbers, 4)) # Output: None
|
||||
|
||||
# Example 3: Empty list
|
||||
print(binary_search([], 5)) # Output: None
|
||||
|
||||
# Example 4: Single element
|
||||
print(binary_search([1], 1)) # Output: 0
|
||||
print(binary_search([1], 2)) # Output: None
|
||||
```
|
||||
|
||||
### Key features:
|
||||
- Uses `TypeVar` to support generic types (as long as comparison operations are defined)
|
||||
- Returns `Optional[int]` to indicate either the index or no match found
|
||||
- Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.)
|
||||
- Includes comprehensive docstring with time/space complexity
|
||||
- Implements standard iterative binary search for O(1) space complexity
|
||||
````
|
||||
|
||||
#### 4.2.2 Streaming Example
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Coder-Next",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."}
|
||||
],
|
||||
max_tokens=512,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and chunk.choices[0].delta.content:
|
||||
print(chunk.choices[0].delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal.
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling Example
|
||||
|
||||
Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above.
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "execute_code",
|
||||
"description": "Execute Python code and return the result",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "The Python code to execute"
|
||||
}
|
||||
},
|
||||
"required": ["code"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Coder-Next",
|
||||
messages=[
|
||||
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
# Check if the model wants to call a tool
|
||||
if response.choices[0].message.tool_calls:
|
||||
tool_call = response.choices[0].message.tool_calls[0]
|
||||
print(f"Tool: {tool_call.function.name}")
|
||||
print(f"Arguments: {tool_call.function.arguments}")
|
||||
else:
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Tool: execute_code
|
||||
Arguments: {"code": "import math\nmath.factorial(10)"}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (2x)
|
||||
- Model: Qwen/Qwen3-Coder-Next
|
||||
- Tensor Parallelism: 2
|
||||
- sglang version: 0.5.8+
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--tp 2 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 27.86
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4218
|
||||
Request throughput (req/s): 0.36
|
||||
Input token throughput (tok/s): 219.00
|
||||
Output token throughput (tok/s): 151.48
|
||||
Peak output token throughput (tok/s): 166.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 370.48
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2784.14
|
||||
Median E2E Latency (ms): 2258.08
|
||||
P90 E2E Latency (ms): 5044.43
|
||||
P99 E2E Latency (ms): 6130.52
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 161.68
|
||||
Median TTFT (ms): 168.09
|
||||
P99 TTFT (ms): 183.26
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.19
|
||||
Median TPOT (ms): 6.23
|
||||
P99 TPOT (ms): 6.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.23
|
||||
Median ITL (ms): 6.23
|
||||
P95 ITL (ms): 6.51
|
||||
P99 ITL (ms): 6.64
|
||||
Max ITL (ms): 13.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 39.06
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40789
|
||||
Request throughput (req/s): 2.05
|
||||
Input token throughput (tok/s): 1015.62
|
||||
Output token throughput (tok/s): 1044.73
|
||||
Peak output token throughput (tok/s): 1664.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2060.34
|
||||
Concurrency: 14.16
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6910.97
|
||||
Median E2E Latency (ms): 7248.27
|
||||
P90 E2E Latency (ms): 11612.63
|
||||
P99 E2E Latency (ms): 13933.91
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 183.48
|
||||
Median TTFT (ms): 156.50
|
||||
P99 TTFT (ms): 311.46
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 13.61
|
||||
Median TPOT (ms): 13.59
|
||||
P99 TPOT (ms): 21.11
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.22
|
||||
Median ITL (ms): 9.76
|
||||
P95 ITL (ms): 10.43
|
||||
P99 ITL (ms): 158.04
|
||||
Max ITL (ms): 394.39
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 102.81
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252536
|
||||
Request throughput (req/s): 4.86
|
||||
Input token throughput (tok/s): 2429.99
|
||||
Output token throughput (tok/s): 2457.53
|
||||
Peak output token throughput (tok/s): 5299.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 4887.52
|
||||
Concurrency: 94.28
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 19385.20
|
||||
Median E2E Latency (ms): 17584.09
|
||||
P90 E2E Latency (ms): 36762.15
|
||||
P99 E2E Latency (ms): 42518.35
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 270.62
|
||||
Median TTFT (ms): 159.65
|
||||
P99 TTFT (ms): 938.90
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.57
|
||||
Median TPOT (ms): 41.78
|
||||
P99 TPOT (ms): 53.28
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 37.90
|
||||
Median ITL (ms): 18.26
|
||||
P95 ITL (ms): 167.82
|
||||
P99 ITL (ms): 311.45
|
||||
Max ITL (ms): 993.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--tp 2 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Result:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 285.02
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44432
|
||||
Request throughput (req/s): 0.04
|
||||
Input token throughput (tok/s): 21.41
|
||||
Output token throughput (tok/s): 156.00
|
||||
Peak output token throughput (tok/s): 173.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 177.40
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 28499.54
|
||||
Median E2E Latency (ms): 30424.65
|
||||
P90 E2E Latency (ms): 49132.26
|
||||
P99 E2E Latency (ms): 51075.28
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 95.51
|
||||
Median TTFT (ms): 93.86
|
||||
P99 TTFT (ms): 112.56
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.24
|
||||
Median TPOT (ms): 6.30
|
||||
P99 TPOT (ms): 6.60
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.39
|
||||
Median ITL (ms): 6.34
|
||||
P95 ITL (ms): 7.16
|
||||
P99 ITL (ms): 7.42
|
||||
Max ITL (ms): 12.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Result:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 237.77
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 315646
|
||||
Request throughput (req/s): 0.34
|
||||
Input token throughput (tok/s): 166.83
|
||||
Output token throughput (tok/s): 1338.72
|
||||
Peak output token throughput (tok/s): 1727.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 1505.55
|
||||
Concurrency: 13.88
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 41266.21
|
||||
Median E2E Latency (ms): 41010.10
|
||||
P90 E2E Latency (ms): 77574.22
|
||||
P99 E2E Latency (ms): 82688.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 140.73
|
||||
Median TTFT (ms): 84.52
|
||||
P99 TTFT (ms): 365.86
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.32
|
||||
Median TPOT (ms): 10.38
|
||||
P99 TPOT (ms): 10.87
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.34
|
||||
Median ITL (ms): 10.19
|
||||
P95 ITL (ms): 10.75
|
||||
P99 ITL (ms): 11.18
|
||||
Max ITL (ms): 206.79
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 384.82
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1299908
|
||||
Request throughput (req/s): 0.83
|
||||
Input token throughput (tok/s): 413.02
|
||||
Output token throughput (tok/s): 3380.83
|
||||
Peak output token throughput (tok/s): 4317.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 3793.85
|
||||
Concurrency: 56.42
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 67847.54
|
||||
Median E2E Latency (ms): 70724.38
|
||||
P90 E2E Latency (ms): 120888.83
|
||||
P99 E2E Latency (ms): 133234.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 212.24
|
||||
Median TTFT (ms): 115.96
|
||||
P99 TTFT (ms): 652.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 16.76
|
||||
Median TPOT (ms): 16.99
|
||||
P99 TPOT (ms): 18.18
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 16.64
|
||||
Median ITL (ms): 15.83
|
||||
P95 ITL (ms): 31.64
|
||||
P99 ITL (ms): 90.85
|
||||
Max ITL (ms): 576.60
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.3.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 29.42
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.34
|
||||
Input token throughput (tok/s): 1425.35
|
||||
Output token throughput (tok/s): 143.42
|
||||
Peak output token throughput (tok/s): 169.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 1568.77
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2941.19
|
||||
Median E2E Latency (ms): 2411.84
|
||||
P90 E2E Latency (ms): 5661.26
|
||||
P99 E2E Latency (ms): 6497.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 139.46
|
||||
Median TTFT (ms): 160.33
|
||||
P99 TTFT (ms): 184.30
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.56
|
||||
Median TPOT (ms): 6.65
|
||||
P99 TPOT (ms): 7.29
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.65
|
||||
Median ITL (ms): 6.68
|
||||
P95 ITL (ms): 7.39
|
||||
P99 ITL (ms): 7.51
|
||||
Max ITL (ms): 16.34
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 41.62
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41664
|
||||
Request throughput (req/s): 1.92
|
||||
Input token throughput (tok/s): 7208.67
|
||||
Output token throughput (tok/s): 1001.19
|
||||
Peak output token throughput (tok/s): 1536.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 8209.86
|
||||
Concurrency: 14.27
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7421.29
|
||||
Median E2E Latency (ms): 7985.77
|
||||
P90 E2E Latency (ms): 12122.09
|
||||
P99 E2E Latency (ms): 14595.05
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 248.49
|
||||
Median TTFT (ms): 179.25
|
||||
P99 TTFT (ms): 915.90
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.13
|
||||
Median TPOT (ms): 14.28
|
||||
P99 TPOT (ms): 24.02
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.80
|
||||
Median ITL (ms): 10.46
|
||||
P95 ITL (ms): 11.00
|
||||
P99 ITL (ms): 173.14
|
||||
Max ITL (ms): 823.32
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-Coder-Next \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
- Result:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 85.74
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169983
|
||||
Request throughput (req/s): 3.73
|
||||
Input token throughput (tok/s): 14858.12
|
||||
Output token throughput (tok/s): 1982.80
|
||||
Peak output token throughput (tok/s): 3734.00
|
||||
Peak concurrent requests: 70
|
||||
Total token throughput (tok/s): 16840.92
|
||||
Concurrency: 59.75
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16008.12
|
||||
Median E2E Latency (ms): 15460.65
|
||||
P90 E2E Latency (ms): 27705.81
|
||||
P99 E2E Latency (ms): 32874.74
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 476.99
|
||||
Median TTFT (ms): 177.50
|
||||
P99 TTFT (ms): 3014.39
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 29.81
|
||||
Median TPOT (ms): 31.19
|
||||
P99 TPOT (ms): 45.53
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 29.29
|
||||
Median ITL (ms): 15.75
|
||||
P95 ITL (ms): 173.94
|
||||
P99 ITL (ms): 202.00
|
||||
Max ITL (ms): 2783.23
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.965
|
||||
Invalid: 0.000
|
||||
Latency: 26.407 s
|
||||
Output throughput: 929.132 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
cd benchmark/mmlu
|
||||
bash download_data.sh
|
||||
python3 bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.780
|
||||
subject: anatomy, #q:135, acc: 0.807
|
||||
subject: astronomy, #q:152, acc: 0.921
|
||||
subject: business_ethics, #q:100, acc: 0.820
|
||||
subject: clinical_knowledge, #q:265, acc: 0.860
|
||||
subject: college_biology, #q:144, acc: 0.944
|
||||
subject: college_chemistry, #q:100, acc: 0.590
|
||||
subject: college_computer_science, #q:100, acc: 0.820
|
||||
subject: college_mathematics, #q:100, acc: 0.800
|
||||
subject: college_medicine, #q:173, acc: 0.803
|
||||
subject: college_physics, #q:102, acc: 0.775
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.936
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.834
|
||||
subject: elementary_mathematics, #q:378, acc: 0.854
|
||||
subject: formal_logic, #q:126, acc: 0.802
|
||||
subject: global_facts, #q:100, acc: 0.610
|
||||
subject: high_school_biology, #q:310, acc: 0.971
|
||||
subject: high_school_chemistry, #q:203, acc: 0.803
|
||||
subject: high_school_computer_science, #q:100, acc: 0.920
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.929
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.969
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.903
|
||||
subject: high_school_mathematics, #q:270, acc: 0.689
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.854
|
||||
subject: high_school_psychology, #q:545, acc: 0.947
|
||||
subject: high_school_statistics, #q:216, acc: 0.815
|
||||
subject: high_school_us_history, #q:204, acc: 0.907
|
||||
subject: high_school_world_history, #q:237, acc: 0.937
|
||||
subject: human_aging, #q:223, acc: 0.821
|
||||
subject: human_sexuality, #q:131, acc: 0.840
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.870
|
||||
subject: logical_fallacies, #q:163, acc: 0.847
|
||||
subject: machine_learning, #q:112, acc: 0.812
|
||||
subject: management, #q:103, acc: 0.922
|
||||
subject: marketing, #q:234, acc: 0.923
|
||||
subject: medical_genetics, #q:100, acc: 0.970
|
||||
subject: miscellaneous, #q:783, acc: 0.941
|
||||
subject: moral_disputes, #q:346, acc: 0.850
|
||||
subject: moral_scenarios, #q:895, acc: 0.726
|
||||
subject: nutrition, #q:306, acc: 0.915
|
||||
subject: philosophy, #q:311, acc: 0.859
|
||||
subject: prehistory, #q:324, acc: 0.889
|
||||
subject: professional_accounting, #q:282, acc: 0.723
|
||||
subject: professional_law, #q:1534, acc: 0.648
|
||||
subject: professional_medicine, #q:272, acc: 0.923
|
||||
subject: professional_psychology, #q:612, acc: 0.845
|
||||
subject: public_relations, #q:110, acc: 0.782
|
||||
subject: security_studies, #q:245, acc: 0.796
|
||||
subject: sociology, #q:201, acc: 0.925
|
||||
subject: us_foreign_policy, #q:100, acc: 0.950
|
||||
subject: virology, #q:166, acc: 0.572
|
||||
subject: world_religions, #q:171, acc: 0.883
|
||||
Total latency: 208.985
|
||||
Average accuracy: 0.834
|
||||
```
|
||||
@@ -0,0 +1,788 @@
|
||||
---
|
||||
title: Qwen3-Coder
|
||||
metatags:
|
||||
description: "Deploy Qwen3-Coder(480B, 30B) MoE coding model with SGLang on AMD MI300X (MI325X, MI355X)"
|
||||
---
|
||||
|
||||
import { Qwen3CoderDeployment } from '/src/snippets/autoregressive/qwen3-coder-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks.
|
||||
- **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services.
|
||||
- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
|
||||
- **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages.
|
||||
- **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio.
|
||||
- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
|
||||
- **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified).
|
||||
|
||||
For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms.
|
||||
|
||||
### 3.1 Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method.
|
||||
|
||||
<Qwen3CoderDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**AMD (MI300X/MI325X/MI355X):**
|
||||
* **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory.
|
||||
* **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement.
|
||||
* **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage.
|
||||
* **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`.
|
||||
|
||||
**NVIDIA (B200/GB200):**
|
||||
* **GB200 Parallelism**: Use `--tp 4 --ep 4` on GB200. B200 uses the default NVIDIA settings generated above.
|
||||
* **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`).
|
||||
* **DP Attention**: NVFP4 configuration supports `--enable-dp-attention` for improved throughput.
|
||||
|
||||
**Intel Xeon CPU:**
|
||||
* Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
**General:**
|
||||
* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Code Generation Example
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints."
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
````text Output
|
||||
```python
|
||||
from typing import List, Optional, TypeVar
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
def binary_search(arr: List[T], target: T) -> Optional[int]:
|
||||
"""
|
||||
Perform binary search on a sorted list to find the index of a target element.
|
||||
|
||||
This function implements the binary search algorithm, which efficiently finds
|
||||
a target value in a sorted array by repeatedly dividing the search interval
|
||||
in half.
|
||||
|
||||
Args:
|
||||
arr (List[T]): A sorted list of elements to search through.
|
||||
target (T): The element to search for in the list.
|
||||
|
||||
Returns:
|
||||
Optional[int]: The index of the target element if found, None otherwise.
|
||||
|
||||
Time Complexity:
|
||||
O(log n) where n is the number of elements in the array.
|
||||
|
||||
Space Complexity:
|
||||
O(1) - iterative implementation uses constant extra space.
|
||||
|
||||
Examples:
|
||||
>>> binary_search([1, 2, 3, 4, 5], 3)
|
||||
2
|
||||
>>> binary_search([1, 2, 3, 4, 5], 6)
|
||||
None
|
||||
>>> binary_search(['a', 'b', 'c', 'd'], 'b')
|
||||
1
|
||||
>>> binary_search([], 1)
|
||||
None
|
||||
"""
|
||||
if not arr:
|
||||
return None
|
||||
|
||||
left: int = 0
|
||||
right: int = len(arr) - 1
|
||||
|
||||
while left <= right:
|
||||
mid: int = (left + right) // 2
|
||||
|
||||
if arr[mid] == target:
|
||||
return mid
|
||||
elif arr[mid] < target:
|
||||
left = mid + 1
|
||||
else:
|
||||
right = mid - 1
|
||||
|
||||
return None
|
||||
|
||||
# Alternative recursive implementation
|
||||
def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]:
|
||||
"""
|
||||
Perform binary search recursively on a sorted list to find the index of a target element.
|
||||
|
||||
Args:
|
||||
arr (List[T]): A sorted list of elements to search through.
|
||||
target (T): The element to search for in the list.
|
||||
left (int): Left boundary of the search range (inclusive).
|
||||
right (Optional[int]): Right boundary of the search range (inclusive).
|
||||
|
||||
Returns:
|
||||
Optional[int]: The index of the target element if found, None otherwise.
|
||||
|
||||
Time Complexity:
|
||||
O(log n) where n is the number of elements in the array.
|
||||
|
||||
Space Complexity:
|
||||
O(log n) due to recursive call stack.
|
||||
|
||||
Examples:
|
||||
>>> binary_search_recursive([1, 2, 3, 4, 5], 3)
|
||||
2
|
||||
>>> binary_search_recursive([1, 2, 3, 4, 5], 6)
|
||||
None
|
||||
"""
|
||||
if not arr:
|
||||
return None
|
||||
|
||||
if right is None:
|
||||
right = len(arr) - 1
|
||||
|
||||
if left > right:
|
||||
return None
|
||||
|
||||
mid: int = (left + right) // 2
|
||||
|
||||
if arr[mid] == target:
|
||||
return mid
|
||||
elif arr[mid] < target:
|
||||
return binary_search_recursive(arr, target, mid + 1, right)
|
||||
else:
|
||||
return binary_search_recursive(arr, target, left, mid - 1)
|
||||
```
|
||||
|
||||
This implementation provides:
|
||||
|
||||
1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient
|
||||
2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes
|
||||
3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type
|
||||
4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples
|
||||
5. **Edge case handling**: Empty lists, elements not found, etc.
|
||||
6. **Clear variable names**: Self-documenting code
|
||||
7. **Examples**: Doctest-style examples in the docstring
|
||||
|
||||
The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found.
|
||||
````
|
||||
|
||||
#### 4.2.2 Tool Calling Example
|
||||
|
||||
Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model:
|
||||
|
||||
```shell Command
|
||||
SGLANG_USE_AITER=0 python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Coder-30B-A3B-Instruct \
|
||||
--tp 1 \
|
||||
--context-length 8192 \
|
||||
--page-size 32 \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "execute_code",
|
||||
"description": "Execute Python code and return the result",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "The Python code to execute"
|
||||
}
|
||||
},
|
||||
"required": ["code"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Coder-30B-A3B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
# Check if the model wants to call a tool
|
||||
if response.choices[0].message.tool_calls:
|
||||
tool_call = response.choices[0].message.tool_calls[0]
|
||||
print(f"Tool: {tool_call.function.name}")
|
||||
print(f"Arguments: {tool_call.function.arguments}")
|
||||
else:
|
||||
# Model may return tool call in content format
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Tool: execute_code
|
||||
Arguments: {"code": "def factorial(n):\n if n == 0 or n == 1:\n return 1\n else:\n return n * factorial(n-1)\n\nresult = factorial(10)\nresult"}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (8x)
|
||||
- Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- Expert Parallelism: 2
|
||||
- sglang version: 0.5.7
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset.
|
||||
|
||||
#### 5.1.1 AMD Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
SGLANG_USE_AITER=0 python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--tp 8 \
|
||||
--ep 2 \
|
||||
--context-length 8192 \
|
||||
--page-size 32 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 73.79
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4104
|
||||
Request throughput (req/s): 0.14
|
||||
Input token throughput (tok/s): 82.68
|
||||
Output token throughput (tok/s): 57.19
|
||||
Peak output token throughput (tok/s): 59.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 139.86
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7376.26
|
||||
Median E2E Latency (ms): 5851.51
|
||||
P90 E2E Latency (ms): 13351.89
|
||||
P99 E2E Latency (ms): 16908.32
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 191.93
|
||||
Median TTFT (ms): 126.06
|
||||
P99 TTFT (ms): 662.15
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.06
|
||||
Median TPOT (ms): 17.07
|
||||
P99 TPOT (ms): 17.08
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 17.06
|
||||
Median ITL (ms): 17.06
|
||||
P95 ITL (ms): 17.14
|
||||
P99 ITL (ms): 17.19
|
||||
Max ITL (ms): 18.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 87.04
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40364
|
||||
Request throughput (req/s): 0.92
|
||||
Input token throughput (tok/s): 455.77
|
||||
Output token throughput (tok/s): 468.83
|
||||
Peak output token throughput (tok/s): 608.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 924.59
|
||||
Concurrency: 13.76
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 14966.88
|
||||
Median E2E Latency (ms): 15871.93
|
||||
P90 E2E Latency (ms): 24983.41
|
||||
P99 E2E Latency (ms): 29504.85
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 388.94
|
||||
Median TTFT (ms): 157.49
|
||||
P99 TTFT (ms): 1318.63
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 29.41
|
||||
Median TPOT (ms): 29.22
|
||||
P99 TPOT (ms): 43.48
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 28.64
|
||||
Median ITL (ms): 26.42
|
||||
P95 ITL (ms): 27.51
|
||||
P99 ITL (ms): 131.63
|
||||
Max ITL (ms): 995.11
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 177.82
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 170134
|
||||
Total generated tokens (retokenized): 168387
|
||||
Request throughput (req/s): 1.80
|
||||
Input token throughput (tok/s): 893.84
|
||||
Output token throughput (tok/s): 956.80
|
||||
Peak output token throughput (tok/s): 1728.00
|
||||
Peak concurrent requests: 70
|
||||
Total token throughput (tok/s): 1850.64
|
||||
Concurrency: 58.88
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 32716.53
|
||||
Median E2E Latency (ms): 30896.37
|
||||
P90 E2E Latency (ms): 65605.24
|
||||
P99 E2E Latency (ms): 80970.63
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 372.97
|
||||
Median TTFT (ms): 181.67
|
||||
P99 TTFT (ms): 529.01
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 62.98
|
||||
Median TPOT (ms): 50.44
|
||||
P99 TPOT (ms): 204.24
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 60.95
|
||||
Median ITL (ms): 37.87
|
||||
P95 ITL (ms): 143.98
|
||||
P99 ITL (ms): 148.02
|
||||
Max ITL (ms): 36863.32
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 NVIDIA (B200/GB200) Standard Scenario Benchmark
|
||||
|
||||
The following runs use the same random dataset benchmark client commands as the AMD section. On B200, launch the server with the following command:
|
||||
|
||||
```bash
|
||||
sglang serve --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --tp 8 --ep 8 --context-length 8192 --page-size 32 --trust-remote-code
|
||||
|
||||
##### 5.1.2.1 FP8 Model
|
||||
|
||||
- Low Concurrency:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 42.68
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4204
|
||||
Request throughput (req/s): 0.23
|
||||
Input token throughput (tok/s): 142.95
|
||||
Output token throughput (tok/s): 98.88
|
||||
Peak output token throughput (tok/s): 102.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 241.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4266.06
|
||||
Median E2E Latency (ms): 3420.24
|
||||
P90 E2E Latency (ms): 7717.19
|
||||
P99 E2E Latency (ms): 9504.50
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 112.03
|
||||
Median TTFT (ms): 112.70
|
||||
P99 TTFT (ms): 115.35
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.87
|
||||
Median TPOT (ms): 9.86
|
||||
P99 TPOT (ms): 9.92
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.87
|
||||
Median ITL (ms): 9.87
|
||||
P95 ITL (ms): 10.06
|
||||
P99 ITL (ms): 10.18
|
||||
Max ITL (ms): 14.80
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.80
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40543
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 652.43
|
||||
Output token throughput (tok/s): 671.13
|
||||
Peak output token throughput (tok/s): 864.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1323.57
|
||||
Concurrency: 13.93
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10587.26
|
||||
Median E2E Latency (ms): 11486.18
|
||||
P90 E2E Latency (ms): 17374.75
|
||||
P99 E2E Latency (ms): 21107.18
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 155.27
|
||||
Median TTFT (ms): 121.57
|
||||
P99 TTFT (ms): 294.31
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.77
|
||||
Median TPOT (ms): 21.13
|
||||
P99 TPOT (ms): 23.62
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.49
|
||||
Median ITL (ms): 18.73
|
||||
P95 ITL (ms): 19.65
|
||||
P99 ITL (ms): 98.85
|
||||
Max ITL (ms): 536.87
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 100.07
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 170134
|
||||
Total generated tokens (retokenized): 169119
|
||||
Request throughput (req/s): 3.20
|
||||
Input token throughput (tok/s): 1588.32
|
||||
Output token throughput (tok/s): 1700.19
|
||||
Peak output token throughput (tok/s): 2303.00
|
||||
Peak concurrent requests: 71
|
||||
Total token throughput (tok/s): 3288.51
|
||||
Concurrency: 57.93
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18114.01
|
||||
Median E2E Latency (ms): 18279.15
|
||||
P90 E2E Latency (ms): 30557.22
|
||||
P99 E2E Latency (ms): 35889.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 346.40
|
||||
Median TTFT (ms): 129.75
|
||||
P99 TTFT (ms): 1370.20
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.76
|
||||
Median TPOT (ms): 34.62
|
||||
P99 TPOT (ms): 39.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.48
|
||||
Median ITL (ms): 25.70
|
||||
P95 ITL (ms): 99.36
|
||||
P99 ITL (ms): 132.30
|
||||
Max ITL (ms): 1132.39
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 NVFP4 Model
|
||||
|
||||
- Low Concurrency:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 34.49
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4218
|
||||
Request throughput (req/s): 0.29
|
||||
Input token throughput (tok/s): 176.87
|
||||
Output token throughput (tok/s): 122.34
|
||||
Peak output token throughput (tok/s): 127.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 299.21
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3448.01
|
||||
Median E2E Latency (ms): 2768.11
|
||||
P90 E2E Latency (ms): 6225.73
|
||||
P99 E2E Latency (ms): 7668.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 104.55
|
||||
Median TTFT (ms): 105.38
|
||||
P99 TTFT (ms): 105.63
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.94
|
||||
Median TPOT (ms): 7.95
|
||||
P99 TPOT (ms): 7.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.94
|
||||
Median ITL (ms): 7.94
|
||||
P95 ITL (ms): 8.05
|
||||
P99 ITL (ms): 8.11
|
||||
Max ITL (ms): 24.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 43.30
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 39975
|
||||
Request throughput (req/s): 1.85
|
||||
Input token throughput (tok/s): 916.16
|
||||
Output token throughput (tok/s): 942.42
|
||||
Peak output token throughput (tok/s): 1264.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1858.57
|
||||
Concurrency: 13.90
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7521.95
|
||||
Median E2E Latency (ms): 8246.89
|
||||
P90 E2E Latency (ms): 12370.93
|
||||
P99 E2E Latency (ms): 15023.96
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 137.27
|
||||
Median TTFT (ms): 109.59
|
||||
P99 TTFT (ms): 208.78
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.69
|
||||
Median TPOT (ms): 14.87
|
||||
P99 TPOT (ms): 17.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.51
|
||||
Median ITL (ms): 12.75
|
||||
P95 ITL (ms): 13.33
|
||||
P99 ITL (ms): 92.85
|
||||
Max ITL (ms): 113.70
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 73.93
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 170134
|
||||
Total generated tokens (retokenized): 168841
|
||||
Request throughput (req/s): 4.33
|
||||
Input token throughput (tok/s): 2149.98
|
||||
Output token throughput (tok/s): 2301.42
|
||||
Peak output token throughput (tok/s): 3497.00
|
||||
Peak concurrent requests: 71
|
||||
Total token throughput (tok/s): 4451.40
|
||||
Concurrency: 58.28
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13463.58
|
||||
Median E2E Latency (ms): 13498.74
|
||||
P90 E2E Latency (ms): 22957.10
|
||||
P99 E2E Latency (ms): 26656.95
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 239.00
|
||||
Median TTFT (ms): 113.42
|
||||
P99 TTFT (ms): 713.87
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.13
|
||||
Median TPOT (ms): 26.02
|
||||
P99 TPOT (ms): 30.90
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 24.92
|
||||
Median ITL (ms): 16.68
|
||||
P95 ITL (ms): 93.33
|
||||
P99 ITL (ms): 119.26
|
||||
Max ITL (ms): 548.82
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
##### AMD (MI300X/MI325X/MI355X)
|
||||
|
||||
- **Results**:
|
||||
|
||||
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
|
||||
```
|
||||
Accuracy: 0.965
|
||||
Invalid: 0.000
|
||||
Latency: 23.084 s
|
||||
Output throughput: 1148.425 token/s
|
||||
```
|
||||
|
||||
##### NVIDIA (B200/GB200)
|
||||
|
||||
For deployment commands, see [Section 3.1](#3-1-configuration).
|
||||
|
||||
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
|
||||
```
|
||||
Accuracy: 0.965
|
||||
Invalid: 0.000
|
||||
Latency: 14.870 s
|
||||
Output throughput: 1777.726 token/s
|
||||
```
|
||||
|
||||
- nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4)
|
||||
```
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 13.948 s
|
||||
Output throughput: 1988.548 token/s
|
||||
```
|
||||
@@ -0,0 +1,794 @@
|
||||
---
|
||||
title: Qwen3-Next
|
||||
metatags:
|
||||
description: "Deploy Qwen3-Next with SGLang - hybrid attention architecture supporting 262K context, 80B MoE with 3B active parameters, and multi-token prediction."
|
||||
---
|
||||
|
||||
import { Qwen3NextDeployment } from '/src/snippets/autoregressive/qwen3-next-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding.
|
||||
|
||||
Qwen3-Next introduces several groundbreaking innovations:
|
||||
|
||||
- **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts.
|
||||
|
||||
- **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity.
|
||||
|
||||
- **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed.
|
||||
|
||||
- **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications.
|
||||
|
||||
- **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use.
|
||||
|
||||
- **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training.
|
||||
|
||||
For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
|
||||
|
||||
<Qwen3NextDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
|
||||
|
||||
- `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.
|
||||
|
||||
- `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`.
|
||||
|
||||
- **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
|
||||
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
|
||||
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
|
||||
|
||||
- **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
1. **Streaming with Thinking Process:**
|
||||
|
||||
Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections.
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Thinking \
|
||||
--reasoning-parser qwen3 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36.
|
||||
|
||||
=============== Content =================
|
||||
|
||||
To find **15% of 240**, follow these steps:
|
||||
|
||||
---
|
||||
|
||||
### **Step 1: Understand what "percent" means**
|
||||
- "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form.
|
||||
|
||||
---
|
||||
|
||||
### **Step 2: Multiply the number by the decimal**
|
||||
- To find 15% of 240, multiply:
|
||||
$$
|
||||
240 \times 0.15
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Step 3: Break it down for clarity (optional but helpful)**
|
||||
- **10% of 240** = $ 240 \times 0.1 = 24 $
|
||||
- **5% of 240** = $ 240 \times 0.05 = 12 $
|
||||
- Add them together:
|
||||
$$
|
||||
24 + 12 = 36
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Step 4: Confirm with direct multiplication**
|
||||
- $ 240 \times 0.15 = 36 $
|
||||
|
||||
---
|
||||
|
||||
### ✅ Final Answer:
|
||||
$$
|
||||
\boxed{36}
|
||||
$$
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
2. **Turn off Thinking:**
|
||||
|
||||
Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode.
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Turn off thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Next-80B-A3B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
To find **15% of 240**, follow these steps:
|
||||
|
||||
---
|
||||
|
||||
### **Step 1: Understand what percentage means**
|
||||
"Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction:
|
||||
|
||||
$$
|
||||
\frac{15}{100}
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Step 2: Multiply the fraction by the number**
|
||||
To find 15% of 240, multiply:
|
||||
|
||||
$$
|
||||
\frac{15}{100} \times 240
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Step 3: Simplify the multiplication**
|
||||
You can simplify this in a couple of ways.
|
||||
|
||||
#### **Option A: Multiply first, then divide**
|
||||
$$
|
||||
15 \times 240 = 3600
|
||||
$$
|
||||
Then divide by 100:
|
||||
$$
|
||||
\frac{3600}{100} = 36
|
||||
$$
|
||||
|
||||
#### **Option B: Simplify the fraction first**
|
||||
$$
|
||||
\frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)}
|
||||
$$
|
||||
Now multiply:
|
||||
$$
|
||||
\frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Step 4: Final Answer**
|
||||
$$
|
||||
\boxed{36}
|
||||
$$
|
||||
|
||||
So, **15% of 240 is 36**.
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Python Example (without Thinking Process):**
|
||||
|
||||
Start sglang server:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tool-call-parser qwen \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Next-80B-A3B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
<tool_call>
|
||||
{"name": "get_weather", "arguments": {"location": "Beijing"}}
|
||||
</tool_call>
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
Start sglang server:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Thinking \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}.
|
||||
|
||||
<tool_call>
|
||||
{"name": "get_weather", "arguments": {"location": "Beijing"}}
|
||||
</tool_call>
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Processing Ultra-Long Texts
|
||||
|
||||
Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method.
|
||||
|
||||
**Qwen3-Next-80B-A3B-Instruct**
|
||||
|
||||
```shell Command
|
||||
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
|
||||
|
||||
```
|
||||
|
||||
**Qwen3-Next-80B-A3B-Thinking**
|
||||
|
||||
```shell Command
|
||||
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking --reasoning-parser qwen3 --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
|
||||
|
||||
```
|
||||
|
||||
#### 4.2.4 Multi-Token Prediction (NEXTN Speculative Decoding)
|
||||
|
||||
Qwen3-Next ships built-in Multi-Token Prediction (MTP) layers and supports [EAGLE-style speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding) through the `NEXTN` algorithm. The MTP weights are bundled in the main checkpoint, so no separate draft model is required.
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
Tune `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens` for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). See [PR #10233](https://github.com/sgl-project/sglang/pull/10233) for implementation details.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x)
|
||||
- Tensor Parallelism: 8
|
||||
- Model: Qwen/Qwen3-Next-80B-A3B-Instruct
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Server Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Test Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompt 100 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 146.52
|
||||
Total input tokens: 33839
|
||||
Total input text tokens: 33839
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 21640
|
||||
Total generated tokens (retokenized): 21619
|
||||
Request throughput (req/s): 0.68
|
||||
Input token throughput (tok/s): 230.95
|
||||
Output token throughput (tok/s): 147.70
|
||||
Peak output token throughput (tok/s): 164.00
|
||||
Peak concurrent requests: 6
|
||||
Total token throughput (tok/s): 378.65
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1464.81
|
||||
Median E2E Latency (ms): 1077.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 127.88
|
||||
Median TTFT (ms): 132.88
|
||||
P99 TTFT (ms): 212.85
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.19
|
||||
Median TPOT (ms): 6.17
|
||||
P99 TPOT (ms): 6.64
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.21
|
||||
Median ITL (ms): 6.16
|
||||
P95 ITL (ms): 6.51
|
||||
P99 ITL (ms): 6.71
|
||||
Max ITL (ms): 10.07
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Server Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tp 8 \
|
||||
```
|
||||
|
||||
- Test Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompt 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 100.32
|
||||
Total input tokens: 302118
|
||||
Total input text tokens: 302118
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 195775
|
||||
Total generated tokens (retokenized): 195016
|
||||
Request throughput (req/s): 9.97
|
||||
Input token throughput (tok/s): 3011.69
|
||||
Output token throughput (tok/s): 1951.60
|
||||
Peak output token throughput (tok/s): 5909.00
|
||||
Peak concurrent requests: 120
|
||||
Total token throughput (tok/s): 4963.29
|
||||
Concurrency: 93.05
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9333.98
|
||||
Median E2E Latency (ms): 6054.12
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 161.77
|
||||
Median TTFT (ms): 137.94
|
||||
P99 TTFT (ms): 503.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 50.87
|
||||
Median TPOT (ms): 50.28
|
||||
P99 TPOT (ms): 122.87
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 47.11
|
||||
Median ITL (ms): 13.84
|
||||
P95 ITL (ms): 195.33
|
||||
P99 ITL (ms): 289.56
|
||||
Max ITL (ms): 486.38
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
- Qwen3-Next-80B-A3B-Instruct
|
||||
|
||||
```
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 12.673 s
|
||||
Output throughput: 2538.255 token/s
|
||||
```
|
||||
|
||||
- Qwen3-Next-80B-A3B-Thinking
|
||||
```
|
||||
Accuracy: 0.935
|
||||
Invalid: 0.000
|
||||
Latency: 9.912 s
|
||||
Output throughput: 3288.737 token/s
|
||||
```
|
||||
|
||||
### 5.2.2 MMLU Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
cd sglang
|
||||
bash benchmark/mmlu/download_data.sh
|
||||
python3 benchmark/mmlu/bench_sglang.py --nsub 10
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
- Qwen3-Next-80B-A3B-Instruct
|
||||
|
||||
```
|
||||
subject: abstract_algebra, #q:100, acc: 0.800
|
||||
subject: anatomy, #q:135, acc: 0.807
|
||||
subject: astronomy, #q:152, acc: 0.947
|
||||
subject: business_ethics, #q:100, acc: 0.810
|
||||
subject: clinical_knowledge, #q:265, acc: 0.894
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.680
|
||||
subject: college_computer_science, #q:100, acc: 0.860
|
||||
subject: college_mathematics, #q:100, acc: 0.780
|
||||
subject: college_medicine, #q:173, acc: 0.861
|
||||
Total latency: 10.098
|
||||
Average accuracy: 0.856
|
||||
```
|
||||
|
||||
- Qwen3-Next-80B-A3B-Thinking
|
||||
```
|
||||
subject: abstract_algebra, #q:100, acc: 0.780
|
||||
subject: anatomy, #q:135, acc: 0.815
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.870
|
||||
subject: clinical_knowledge, #q:265, acc: 0.894
|
||||
subject: college_biology, #q:144, acc: 0.965
|
||||
subject: college_chemistry, #q:100, acc: 0.670
|
||||
subject: college_computer_science, #q:100, acc: 0.840
|
||||
subject: college_mathematics, #q:100, acc: 0.770
|
||||
subject: college_medicine, #q:173, acc: 0.861
|
||||
Total latency: 10.236
|
||||
Average accuracy: 0.855
|
||||
```
|
||||
@@ -0,0 +1,809 @@
|
||||
---
|
||||
title: Qwen3-VL
|
||||
metatags:
|
||||
description: "Deploy Qwen3-VL vision-language models with SGLang - open model for text, 262K context, enhanced visual reasoning and agent capabilities."
|
||||
---
|
||||
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
- **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942)
|
||||
- **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities.
|
||||
- **Extended context length**: Supports up to 262K tokens for processing long documents and videos.
|
||||
- **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics.
|
||||
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
|
||||
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
|
||||
|
||||
For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
|
||||
|
||||
import { Qwen3VLDeployment } from "/src/snippets/autoregressive/qwen3-vl-deployment.jsx";
|
||||
|
||||
<Qwen3VLDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
* **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility.
|
||||
* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
* **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
|
||||
* **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
|
||||
* **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
|
||||
|
||||
**Hardware-specific notes:**
|
||||
- **H100 (FP8):** Use the `Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
|
||||
**Additional multimodal server parameters:**
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid device-to-host memory copies, improving performance for high-frequency inference.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
Qwen3-VL supports both image and video inputs. Here's a basic example with image input:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Read all the text in the image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.37s
|
||||
Generated text: Auntie Anne's
|
||||
|
||||
CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
Qwen3-VL can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 10.18s
|
||||
Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.
|
||||
|
||||
The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy.
|
||||
|
||||
In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout.
|
||||
|
||||
Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance.
|
||||
|
||||
In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose.
|
||||
```
|
||||
|
||||
**Video Input Example:**
|
||||
|
||||
Qwen3-VL supports video understanding by processing video URLs:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe what happens in this video."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- For video processing, ensure you have sufficient context length configured (up to 262K tokens)
|
||||
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
|
||||
- You can also provide local file paths using `file://` protocol
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.89s
|
||||
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
|
||||
```
|
||||
|
||||
#### 4.2.2 Reasoning Parser
|
||||
|
||||
Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
|
||||
--reasoning-parser qwen3 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
Qwen3-VL supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x)
|
||||
- Model: Qwen3-VL-235B-A22B-Instruct
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 45.97
|
||||
Total input tokens: 18348
|
||||
Total input text tokens: 708
|
||||
Total input vision tokens: 17640
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3423
|
||||
Request throughput (req/s): 0.22
|
||||
Input token throughput (tok/s): 399.17
|
||||
Output token throughput (tok/s): 91.81
|
||||
Peak output token throughput (tok/s): 96.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 490.98
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4594.52
|
||||
Median E2E Latency (ms): 3725.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 193.35
|
||||
Median TTFT (ms): 196.32
|
||||
P99 TTFT (ms): 222.75
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.44
|
||||
Median TPOT (ms): 10.44
|
||||
P99 TPOT (ms): 10.47
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.78
|
||||
Median ITL (ms): 10.48
|
||||
P95 ITL (ms): 21.01
|
||||
P99 ITL (ms): 31.40
|
||||
Max ITL (ms): 31.92
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Optimized Results (with CUDA IPC Transport):**
|
||||
|
||||
For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features.
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 100 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 566.84
|
||||
Total input tokens: 183667
|
||||
Total input text tokens: 7267
|
||||
Total input vision tokens: 176400
|
||||
Total generated tokens: 52444
|
||||
Total generated tokens (retokenized): 28702
|
||||
Request throughput (req/s): 0.18
|
||||
Input token throughput (tok/s): 324.02
|
||||
Output token throughput (tok/s): 92.52
|
||||
Peak output token throughput (tok/s): 96.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 416.54
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5667.50
|
||||
Median E2E Latency (ms): 5830.00
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 191.16
|
||||
Median TTFT (ms): 182.58
|
||||
P99 TTFT (ms): 244.58
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.46
|
||||
Median TPOT (ms): 10.46
|
||||
P99 TPOT (ms): 10.48
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.91
|
||||
Median ITL (ms): 10.56
|
||||
P95 ITL (ms): 21.35
|
||||
P99 ITL (ms): 31.55
|
||||
Max ITL (ms): 42.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 584.65
|
||||
Total input tokens: 1839015
|
||||
Total input text tokens: 75015
|
||||
Total input vision tokens: 1764000
|
||||
Total generated tokens: 510855
|
||||
Total generated tokens (retokenized): 284284
|
||||
Request throughput (req/s): 1.71
|
||||
Input token throughput (tok/s): 3145.50
|
||||
Output token throughput (tok/s): 873.78
|
||||
Peak output token throughput (tok/s): 2855.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 4019.29
|
||||
Concurrency: 98.35
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 57502.05
|
||||
Median E2E Latency (ms): 54301.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 5802.23
|
||||
Median TTFT (ms): 1444.75
|
||||
P99 TTFT (ms): 46675.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 100.22
|
||||
Median TPOT (ms): 105.43
|
||||
P99 TPOT (ms): 144.37
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 134.20
|
||||
Median ITL (ms): 25.57
|
||||
P95 ITL (ms): 558.14
|
||||
P99 ITL (ms): 1449.01
|
||||
Max ITL (ms): 33453.23
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 MMMU Benchmark
|
||||
|
||||
You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`:
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
uv pip install lmms_eval
|
||||
|
||||
python3 -m lmms_eval \
|
||||
--model openai_compatible \
|
||||
--model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \
|
||||
--tasks mmmu_val \
|
||||
--batch_size 128 \
|
||||
--log_samples \
|
||||
--log_samples_suffix "openai_compatible" \
|
||||
--output_path ./logs \
|
||||
--gen_kwargs "max_new_tokens=4096"
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
<col style={{width: "11%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Stderr</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_val</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_acc</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6567</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>±</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,518 @@
|
||||
---
|
||||
title: Qwen3.6
|
||||
metatags:
|
||||
description: "Deploy Qwen3.6 with SGLang - open-weight multimodal series with a 35B MoE (3B active) variant and a 27B dense variant, hybrid reasoning, tool calling, MTP, and long-context support."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
import { Qwen36Deployment } from '/src/snippets/autoregressive/qwen36-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
The Qwen3.6 series is developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation. Two size/sparsity variants are released:
|
||||
|
||||
- [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — **Sparse MoE** (35B total, 3B active) on a Gated Delta Networks backbone.
|
||||
- [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — **Dense** hybrid GDN; smaller weights footprint, single-GPU friendly.
|
||||
|
||||
Both variants share the same hybrid reasoning, tool-calling, and multimodal interface and natively handle context lengths of up to 262,144 tokens, extensible to over 1M tokens.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision
|
||||
- **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development
|
||||
- **Efficient Hybrid Architecture**: Gated Delta Networks backbone; sparse MoE (35B / 3B active) or dense 27B variant
|
||||
- **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
|
||||
- **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser
|
||||
- **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency; both MoE and Dense variants ship `mtp.safetensors`
|
||||
- **Multimodal**: Unified vision-language model supporting text, image, and video inputs
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Model</th>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Architecture</th>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Weights</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-35B-A3B (BF16)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE 35B / 3B active</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6-35B-A3B (FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MoE 35B / 3B active</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-35B-A3B (NVFP4)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE 35B / 3B active (Blackwell)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-27B (BF16)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Dense 27B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6-27B (FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Dense 27B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen3.6-27B-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-27B (NVFP4)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Dense 27B (Blackwell)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang `>=0.5.10` is required for Qwen3.6. You can install from PyPI, from source, or use a Docker image:
|
||||
|
||||
```bash Command
|
||||
# Install from PyPI
|
||||
uv pip install sglang
|
||||
|
||||
# Or install from source
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
|
||||
# Or use Docker (NVIDIA GPUs; also serves the NVFP4 variants)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.
|
||||
|
||||
|
||||
<Qwen36Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
|
||||
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
|
||||
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
- Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
|
||||
- **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
|
||||
- **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300.
|
||||
- For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
|
||||
- Hardware requirements:
|
||||
- **35B-A3B BF16**: ~70GB for weights. TP=1 fits on all supported hardware.
|
||||
- **35B-A3B FP8**: ~35GB for weights. TP=1 fits on all supported hardware.
|
||||
- **35B-A3B NVFP4**: ~23GB for weights. TP=1 fits on B200/B300.
|
||||
- **27B BF16**: ~54GB for weights. TP=1 fits on all supported hardware.
|
||||
- **27B FP8**: ~27GB for weights. TP=1 fits on all supported hardware.
|
||||
- **27B NVFP4**: ~22GB for weights. TP=1 fits on B200/B300.
|
||||
|
||||
All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU. NVFP4 is available on B200/B300:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Hardware</th>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Memory</th>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>BF16 TP</th>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>FP8 TP</th>
|
||||
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>NVFP4 TP</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>183GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>B300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>275GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Vision Input
|
||||
|
||||
Qwen3.6 supports image and video inputs as a unified vision-language model.
|
||||
|
||||
**Image Input Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image in detail."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Video Input Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe what happens in this video."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Reasoning Parser
|
||||
|
||||
Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.3.2 Thinking Preservation
|
||||
|
||||
Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Help me plan a web app architecture."}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.3.3 Tool Calling
|
||||
|
||||
Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
@@ -0,0 +1,887 @@
|
||||
---
|
||||
title: Qwen3
|
||||
metatags:
|
||||
description: "Deploy Qwen3 series models with SGLang - featuring advanced reasoning, 256K context, and flexible Dense/MoE architectures for edge to cloud."
|
||||
---
|
||||
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
- **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage.
|
||||
- **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages.
|
||||
- **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation.
|
||||
- **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts.
|
||||
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
|
||||
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
|
||||
|
||||
For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
|
||||
|
||||
import { Qwen3Deployment } from "/src/snippets/autoregressive/qwen3-deployment.jsx";
|
||||
|
||||
<Qwen3Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Memory Management:** Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
|
||||
- **Expert Parallelism:** SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
|
||||
- **Kernel Tuning:** For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
|
||||
- **Speculative Decoding:** Using Speculative Decoding for latency-sensitive scenarios.
|
||||
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
|
||||
- `--speculative-num-steps 3`: Number of speculative verification rounds
|
||||
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
|
||||
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
|
||||
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan).
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-235B-A22B-Thinking-2507 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
|
||||
Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method.
|
||||
...
|
||||
So conclusion: The answer is 36.
|
||||
|
||||
=============== Content =================
|
||||
|
||||
|
||||
To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution:
|
||||
|
||||
....
|
||||
|
||||
### Final Answer:
|
||||
|
||||
$$
|
||||
\boxed{36}
|
||||
$$
|
||||
|
||||
Thus, 15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
Qwen3 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-235B-A22B-Thinking-2507 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen25 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
|
||||
Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them.
|
||||
|
||||
|
||||
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞"
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x)
|
||||
- Model: Qwen3-235B-A22B-Instruct-2507
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 43.56
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4206
|
||||
Request throughput (req/s): 0.23
|
||||
Input token throughput (tok/s): 140.07
|
||||
Output token throughput (tok/s): 96.65
|
||||
Peak output token throughput (tok/s): 100.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 236.72
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4353.63
|
||||
Median E2E Latency (ms): 3475.79
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 99.03
|
||||
Median TTFT (ms): 92.18
|
||||
P99 TTFT (ms): 166.05
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.12
|
||||
Median TPOT (ms): 10.12
|
||||
P99 TPOT (ms): 10.15
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.13
|
||||
Median ITL (ms): 10.12
|
||||
P95 ITL (ms): 10.49
|
||||
P99 ITL (ms): 10.70
|
||||
Max ITL (ms): 13.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 48.95
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40725
|
||||
Total generated tokens (retokenized): 40716
|
||||
Request throughput (req/s): 1.63
|
||||
Input token throughput (tok/s): 810.44
|
||||
Output token throughput (tok/s): 832.04
|
||||
Peak output token throughput (tok/s): 1151.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1642.48
|
||||
Concurrency: 13.61
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8326.72
|
||||
Median E2E Latency (ms): 8827.86
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 215.70
|
||||
Median TTFT (ms): 88.82
|
||||
P99 TTFT (ms): 727.08
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 16.36
|
||||
Median TPOT (ms): 16.12
|
||||
P99 TPOT (ms): 24.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.96
|
||||
Median ITL (ms): 14.52
|
||||
P95 ITL (ms): 16.04
|
||||
P99 ITL (ms): 67.69
|
||||
Max ITL (ms): 457.52
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 92.07
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252162
|
||||
Total generated tokens (retokenized): 251124
|
||||
Request throughput (req/s): 5.43
|
||||
Input token throughput (tok/s): 2713.46
|
||||
Output token throughput (tok/s): 2738.78
|
||||
Peak output token throughput (tok/s): 4400.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5452.24
|
||||
Concurrency: 90.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16665.09
|
||||
Median E2E Latency (ms): 16060.10
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 260.55
|
||||
Median TTFT (ms): 122.68
|
||||
P99 TTFT (ms): 863.11
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.94
|
||||
Median TPOT (ms): 34.04
|
||||
P99 TPOT (ms): 41.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 32.59
|
||||
Median ITL (ms): 23.54
|
||||
P95 ITL (ms): 69.79
|
||||
P99 ITL (ms): 119.09
|
||||
Max ITL (ms): 577.70
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 457.45
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44452
|
||||
Total generated tokens (retokenized): 44059
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 13.34
|
||||
Output token throughput (tok/s): 97.17
|
||||
Peak output token throughput (tok/s): 100.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 110.51
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 45742.42
|
||||
Median E2E Latency (ms): 49266.87
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 110.60
|
||||
Median TTFT (ms): 109.36
|
||||
P99 TTFT (ms): 167.43
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.23
|
||||
Median TPOT (ms): 10.24
|
||||
P99 TPOT (ms): 10.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.27
|
||||
Median ITL (ms): 10.26
|
||||
P95 ITL (ms): 10.71
|
||||
P99 ITL (ms): 10.97
|
||||
Max ITL (ms): 15.79
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 340.17
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318226
|
||||
Total generated tokens (retokenized): 318104
|
||||
Request throughput (req/s): 0.24
|
||||
Input token throughput (tok/s): 116.61
|
||||
Output token throughput (tok/s): 935.49
|
||||
Peak output token throughput (tok/s): 1120.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 1052.10
|
||||
Concurrency: 13.85
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 58885.30
|
||||
Median E2E Latency (ms): 59238.70
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 169.71
|
||||
Median TTFT (ms): 101.61
|
||||
P99 TTFT (ms): 455.71
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.82
|
||||
Median TPOT (ms): 14.91
|
||||
P99 TPOT (ms): 15.20
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.76
|
||||
Median ITL (ms): 14.63
|
||||
P95 ITL (ms): 15.46
|
||||
P99 ITL (ms): 16.62
|
||||
Max ITL (ms): 104.94
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 544.83
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1300705
|
||||
Total generated tokens (retokenized): 1293015
|
||||
Request throughput (req/s): 0.59
|
||||
Input token throughput (tok/s): 291.72
|
||||
Output token throughput (tok/s): 2387.34
|
||||
Peak output token throughput (tok/s): 3008.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 2679.06
|
||||
Concurrency: 56.35
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 95937.70
|
||||
Median E2E Latency (ms): 99362.32
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 265.03
|
||||
Median TTFT (ms): 129.11
|
||||
P99 TTFT (ms): 823.85
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 23.66
|
||||
Median TPOT (ms): 24.07
|
||||
P99 TPOT (ms): 24.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.54
|
||||
Median ITL (ms): 23.07
|
||||
P95 ITL (ms): 25.92
|
||||
P99 ITL (ms): 63.87
|
||||
Max ITL (ms): 408.30
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.3.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 44.82
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4210
|
||||
Request throughput (req/s): 0.22
|
||||
Input token throughput (tok/s): 935.86
|
||||
Output token throughput (tok/s): 93.94
|
||||
Peak output token throughput (tok/s): 99.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 1029.80
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4479.60
|
||||
Median E2E Latency (ms): 3622.99
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 139.90
|
||||
Median TTFT (ms): 114.85
|
||||
P99 TTFT (ms): 225.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.31
|
||||
Median TPOT (ms): 10.33
|
||||
P99 TPOT (ms): 10.51
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.33
|
||||
Median ITL (ms): 10.33
|
||||
P95 ITL (ms): 10.73
|
||||
P99 ITL (ms): 10.93
|
||||
Max ITL (ms): 14.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 50.68
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41589
|
||||
Total generated tokens (retokenized): 41578
|
||||
Request throughput (req/s): 1.58
|
||||
Input token throughput (tok/s): 5920.41
|
||||
Output token throughput (tok/s): 820.69
|
||||
Peak output token throughput (tok/s): 1200.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 6741.10
|
||||
Concurrency: 13.90
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8805.54
|
||||
Median E2E Latency (ms): 9368.79
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 284.29
|
||||
Median TTFT (ms): 168.48
|
||||
P99 TTFT (ms): 1027.21
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 16.81
|
||||
Median TPOT (ms): 16.66
|
||||
P99 TPOT (ms): 27.18
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 16.42
|
||||
Median ITL (ms): 13.68
|
||||
P95 ITL (ms): 17.23
|
||||
P99 ITL (ms): 90.75
|
||||
Max ITL (ms): 574.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 94.77
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 169680
|
||||
Total generated tokens (retokenized): 169640
|
||||
Request throughput (req/s): 3.38
|
||||
Input token throughput (tok/s): 13441.86
|
||||
Output token throughput (tok/s): 1790.43
|
||||
Peak output token throughput (tok/s): 2687.00
|
||||
Peak concurrent requests: 70
|
||||
Total token throughput (tok/s): 15232.28
|
||||
Concurrency: 58.63
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 17364.14
|
||||
Median E2E Latency (ms): 17495.95
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 238.22
|
||||
Median TTFT (ms): 203.27
|
||||
P99 TTFT (ms): 510.48
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.50
|
||||
Median TPOT (ms): 34.27
|
||||
P99 TPOT (ms): 40.59
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 32.36
|
||||
Median ITL (ms): 22.50
|
||||
P95 ITL (ms): 97.81
|
||||
P99 ITL (ms): 151.55
|
||||
Max ITL (ms): 352.79
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
- Qwen/Qwen3-235B-A22B-Instruct-2507
|
||||
```text Output
|
||||
Accuracy: 0.945
|
||||
Invalid: 0.000
|
||||
Latency: 11.980 s
|
||||
Output throughput: 2358.105 token/s
|
||||
```
|
||||
@@ -0,0 +1,324 @@
|
||||
---
|
||||
title: Step-3.7-Flash (new)
|
||||
metatags:
|
||||
description: "Deploy Step-3.7-Flash multimodal reasoning engine with SGLang."
|
||||
---
|
||||
|
||||
import { Step37FlashDeployment } from '/src/snippets/autoregressive/step-37-flash-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) is a 198B-parameter Mixture-of-Experts (MoE) vision-language model that combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding. Engineered for high-frequency production workloads, it activates approximately 11B parameters per token and supports a 256k context window with three selectable reasoning levels (low, medium, and high). The model is available in multiple quantization formats (BF16, FP8, NVFP4).
|
||||
|
||||
Step-3.7-Flash is built for developers who need to scale agentic workflows that combine perception, search, and reasoning — from parsing massive financial reports in one pass, to running multi-step search loops with cross-source verification, to operating concurrent coding agents in high-throughput pipelines.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Step-3.7-Flash is currently available in SGLang via Docker image install.
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash Command
|
||||
# Pull the docker image
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# Launch the container
|
||||
docker run -it --gpus all \
|
||||
--shm-size=32g \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
lmsysorg/sglang:latest bash
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Step-3.7-Flash series comes in one size with multiple quantization options. Recommended starting configurations vary depending on hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities.
|
||||
|
||||
<Step37FlashDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4x, TP=4), B200/B300 (4x, TP=4), GB200/GB300 (4x, TP=4).
|
||||
- **NVFP4 Quantization**: NVFP4 provides the smallest memory footprint. Requires `--quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 --moe-runner-backend flashinfer_trtllm`.
|
||||
- **Trust Remote Code**: All Step-3.7-Flash variants require `--trust-remote-code` due to the custom model architecture.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
Step-3.7-Flash supports image inputs alongside text. Here's a basic example:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Read all the text in the image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.7-Flash",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
Step-3.7-Flash can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.7-Flash",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
#### 4.2.2 Reasoning Parser
|
||||
|
||||
Step-3.7-Flash supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path stepfun-ai/Step-3.7-Flash \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser step3p5
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
Step-3.7-Flash supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Start sglang server:**
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path stepfun-ai/Step-3.7-Flash \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser step3p5 \
|
||||
--tool-call-parser step3p5
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
import json
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# 1. define tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# 2. tool run
|
||||
def get_weather(location, unit="celsius"):
|
||||
return f"The weather in {location} is 22 {unit[0].upper()} and sunny."
|
||||
|
||||
# 3. send first request
|
||||
print("--- Sending first request ---")
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=1.0,
|
||||
stream=False
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
|
||||
# 4. Handle Reasoning Content
|
||||
reasoning = getattr(message, 'reasoning_content', None)
|
||||
if reasoning:
|
||||
print("=============== Thinking =================")
|
||||
print(reasoning)
|
||||
print("==========================================")
|
||||
|
||||
# 5. Handle Tool Calls
|
||||
if message.tool_calls:
|
||||
print("\nTool Calls detected:")
|
||||
history_messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
message
|
||||
]
|
||||
|
||||
for tool_call in message.tool_calls:
|
||||
print(f" Tool: {tool_call.function.name}")
|
||||
print(f" Args: {tool_call.function.arguments}")
|
||||
|
||||
args = json.loads(tool_call.function.arguments)
|
||||
tool_result = get_weather(args.get("location"), args.get("unit", "celsius"))
|
||||
|
||||
history_messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": tool_result
|
||||
})
|
||||
|
||||
print("\n--- Sending tool results ---")
|
||||
final_response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.7-Flash",
|
||||
messages=history_messages,
|
||||
temperature=1.0,
|
||||
stream=False
|
||||
)
|
||||
|
||||
print("=============== Final Content =================")
|
||||
print(final_response.choices[0].message.content)
|
||||
|
||||
else:
|
||||
if message.content:
|
||||
print("=============== Content =================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
*Benchmark results will be added soon.*
|
||||
@@ -0,0 +1,694 @@
|
||||
---
|
||||
title: Step3-VL-10B
|
||||
metatags:
|
||||
description: "Deploy Step3-VL-10B multimodal model with SGLang - compact 10B dense model with frontier-level vision understanding, complex reasoning, and tool calling capabilities."
|
||||
---
|
||||
|
||||
import { Step3VL10BDeployment } from '/src/snippets/autoregressive/step-3vl-10b-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Step3-VL-10B](https://huggingface.co/stepfun-ai/Step3-VL-10B) is a lightweight open-source multimodal model developed by StepFun, designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. Despite its compact 10B parameter footprint, Step3-VL-10B excels in visual perception, complex reasoning, and human-centric alignment.
|
||||
|
||||
Key highlights of Step3-VL-10B include:
|
||||
|
||||
- **STEM Reasoning**: Achieves 94.43% on AIME 2025 and 75.95% on MathVision (with PaCoRe), demonstrating exceptional complex reasoning capabilities that outperform models 10×–20× larger.
|
||||
- **Visual Perception**: Records 92.05% on MMBench and 80.11% on MMMU, establishing strong general visual understanding and multimodal reasoning.
|
||||
- **GUI & OCR**: Delivers state-of-the-art performance on ScreenSpot-V2 (92.61%), ScreenSpot-Pro (51.55%), and OCRBench (86.75%), optimized for agentic and document understanding tasks.
|
||||
- **Spatial Understanding**: Demonstrates emergent spatial awareness with 66.79% on BLINK and 57.21% on All-Angles-Bench, establishing strong potential for embodied intelligence applications.
|
||||
|
||||
For more details, please refer to the [Step3-VL-10B model card on Hugging Face](https://huggingface.co/stepfun-ai/Step3-VL-10B).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Step3-VL-10B is a compact 10B dense model that can run on a single GPU. Recommended starting configurations vary depending on hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and quantization method. SGLang supports serving Step3-VL-10B on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs.
|
||||
|
||||
<Step3VL10BDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Single GPU Deployment**: Step3-VL-10B fits comfortably on a single GPU with BF16 precision, no tensor parallelism required.
|
||||
- **Memory Management**: Set lower `--context-length` to conserve memory if needed. A value of `32768` is sufficient for most scenarios.
|
||||
- **FP8 Quantization**: Use FP8 quantization to further reduce memory usage while maintaining quality.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
Step3-VL-10B supports image inputs. Here's a basic example with image input:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Read all the text in the image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step3-VL-10B",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
extra_body={"top_k": -1}
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 5.89s
|
||||
Generated text: Auntie Anne's
|
||||
|
||||
CINNAMON SUGAR
|
||||
1 × 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
Step3-VL-10B can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step3-VL-10B",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
extra_body={"top_k": -1}
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.24s
|
||||
Generated text: First image: Single red Hong Kong taxi close - up, clear license plate (RX 5004), “4 SEATS” sticker, urban street with shops behind. Second image: Aerial view of many taxis (red, green) on a highway with a viaduct, some hoods open, dense arrangement. Differences: Scale (single vs many), perspective (close - up vs aerial), context (street shops vs highway), and taxi conditions (normal vs some open hoods).
|
||||
```
|
||||
|
||||
#### 4.2.2 Reasoning Parser
|
||||
|
||||
Step3-VL-10B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model stepfun-ai/Step3-VL-10B \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step3-VL-10B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
extra_body={"top_k": -1}
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
Okay, let's see. The problem is asking for 15% of 240. Hmm, I need to remember how to calculate percentages. So, percentage means "per hundred," right? So, 15% is the same as 15 per 100, or 15/100. To find a percentage of a number, I think you convert the percentage to a decimal and then multiply it by the number. Let me check that.
|
||||
|
||||
First, 15% as a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Yeah, that's right. So 15% is 0.15 in decimal form. Then, to find 15% of 24
|
||||
0, I need to multiply 0.15 by 240. Let me do that calculation.
|
||||
|
||||
Let me write it out: 0.15 * 240. Let's compute that. Maybe break it down. 0.1 is 10%, and 0.05 is 5%, so 10% of 240 is 24, and 5% of 240 is 12. Then 10% + 5% is 15%, so 24 + 12 is 36. Oh,
|
||||
that's a good way to check. So 15% is 10% plus 5%, which adds up to 36. Let me verify with the decimal method. 0.15 * 240. Let's multiply 240 by 0.1 first: 24. Then 240 by 0.05: 12. Adding them gives 36. Yep, that matches. Alternatively, 240 * 15 = 3600, and then divide by 100 (since it's per hundred), so 3600 / 100 = 36. That's another way. So 15% of 240 is 36. Let me make sure I didn't make a mistake. Let's check with another method. 10% of 240 is 24, 20% would be 48, so 15% is halfway between 10% and 20%, which is (24 + 48)/2 = 36. Yep, that works too
|
||||
. So all methods point to 36. I think that's the answer.
|
||||
|
||||
=============== Content =================
|
||||
|
||||
To solve the problem "What is 15% of 240?" step by step:
|
||||
|
||||
---
|
||||
|
||||
### **Step 1: Understand the concept of percentage**
|
||||
A percentage represents a portion of a whole. Specifically, "percent" means "per hundred." So, 15% means **15 out of 100**, or **15/100**.
|
||||
|
||||
---
|
||||
|
||||
### **Step 2: Convert the percentage to a decimal**
|
||||
To use percentages in calculations, convert them to decimals by dividing by 100:
|
||||
$$
|
||||
15\% = \frac{15}{100} = 0.15
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Step 3: Multiply the decimal by the given number**
|
||||
Now, multiply 0.15 (the decimal form of 15%) by 240:
|
||||
$$
|
||||
0.15 \times 240 = 36
|
||||
$$
|
||||
|
||||
---
|
||||
|
||||
### **Alternative Verification Methods**
|
||||
|
||||
#### **Method A: Break into parts**
|
||||
- 10% of 240 = $ 0.10 \times 240 = 24 $
|
||||
- 5% of 240 = $ 0.05 \times 240 = 12 $
|
||||
- Add them: $ 24 + 12 = 36 $
|
||||
|
||||
#### **Method B: Use direct multiplication**
|
||||
- $ 15\% \text{ of } 240 = \frac{15}{100} \times 240 = \frac{3600}{100} = 36 $
|
||||
|
||||
#### **Method C: Estimate using known percentages**
|
||||
- 20% of 240 = $ 0.20 \times 240 = 48 $
|
||||
- 10% of 240 = $ 0.10 \times 240 = 24 $
|
||||
- 15% is halfway between 10% and 20%: $ \frac{24 + 48}{2} = 36 $
|
||||
|
||||
---
|
||||
|
||||
### **Final Answer**
|
||||
$$
|
||||
\boxed{36}
|
||||
$$
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
Step3-VL-10B supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model stepfun-ai/Step3-VL-10B \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--tool-call-parser hermes \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step3-VL-10B",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True,
|
||||
extra_body={"top_k": -1}
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I have a function called "get_weather" that can provide weather information for a location. Let me check the parameters:
|
||||
|
||||
- location: required (string) - "Beijing"
|
||||
- unit: optional (string, enum: ["celsius", "fahrenheit"]) - not specified by the user, so I won't include it
|
||||
|
||||
I should call the function with location="Beijing".
|
||||
|
||||
<tool_calls>
|
||||
|
||||
=============== Content =================
|
||||
|
||||
</tool_calls>Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step3-VL-10B",
|
||||
messages=messages,
|
||||
temperature=0.7,
|
||||
extra_body={"top_k": -1}
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (1x)
|
||||
- Model: stepfun-ai/Step3-VL-10B
|
||||
- Tensor Parallelism: 1
|
||||
- sglang version: 0.5.8+
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model stepfun-ai/Step3-VL-10B \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model stepfun-ai/Step3-VL-10B \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Result:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 30.85
|
||||
Total input tokens: 14120
|
||||
Total input text tokens: 720
|
||||
Total input vision tokens: 13400
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4217
|
||||
Request throughput (req/s): 0.32
|
||||
Input token throughput (tok/s): 457.71
|
||||
Output token throughput (tok/s): 136.79
|
||||
Peak output token throughput (tok/s): 240.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 594.50
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3083.40
|
||||
Median E2E Latency (ms): 2747.00
|
||||
P90 E2E Latency (ms): 4574.50
|
||||
P99 E2E Latency (ms): 5462.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1327.69
|
||||
Median TTFT (ms): 1341.01
|
||||
P99 TTFT (ms): 1486.11
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.16
|
||||
Median TPOT (ms): 4.17
|
||||
P99 TPOT (ms): 4.18
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.17
|
||||
Median ITL (ms): 4.18
|
||||
P95 ITL (ms): 4.30
|
||||
P99 ITL (ms): 4.38
|
||||
Max ITL (ms): 8.24
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model stepfun-ai/Step3-VL-10B \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Result:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 976.52
|
||||
Total input tokens: 1416949
|
||||
Total input text tokens: 76949
|
||||
Total input vision tokens: 1340000
|
||||
Total generated tokens: 510855
|
||||
Total generated tokens (retokenized): 510526
|
||||
Request throughput (req/s): 1.02
|
||||
Input token throughput (tok/s): 1451.02
|
||||
Output token throughput (tok/s): 523.14
|
||||
Peak output token throughput (tok/s): 20429.00
|
||||
Peak concurrent requests: 103
|
||||
Total token throughput (tok/s): 1974.16
|
||||
Concurrency: 99.81
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 97463.22
|
||||
Median E2E Latency (ms): 91872.75
|
||||
P90 E2E Latency (ms): 118553.42
|
||||
P99 E2E Latency (ms): 198445.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 94379.07
|
||||
Median TTFT (ms): 87163.09
|
||||
P99 TTFT (ms): 194871.41
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.89
|
||||
Median TPOT (ms): 5.72
|
||||
P99 TPOT (ms): 23.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.05
|
||||
Median ITL (ms): 0.13
|
||||
P95 ITL (ms): 0.56
|
||||
P99 ITL (ms): 3.99
|
||||
Max ITL (ms): 97551.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 MMMU Benchmark
|
||||
|
||||
You can evaluate the model's accuracy using the MMMU dataset:
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model stepfun-ai/Step3-VL-10B \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 benchmark/mmmu/bench_sglang.py \
|
||||
--port 30000 \
|
||||
--concurrency 64
|
||||
```
|
||||
|
||||
- Result:
|
||||
|
||||
```text Output
|
||||
Benchmark time: 934.6179109360091
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.667, 'num': 30},
|
||||
'Agriculture': {'acc': 0.367, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.4, 'num': 30},
|
||||
'Art': {'acc': 0.467, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.5, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.367, 'num': 30},
|
||||
'Biology': {'acc': 0.3, 'num': 30},
|
||||
'Chemistry': {'acc': 0.467, 'num': 30},
|
||||
'Clinical_Medicine': {'acc': 0.567, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.467, 'num': 30},
|
||||
'Design': {'acc': 0.567, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.3, 'num': 30},
|
||||
'Economics': {'acc': 0.6, 'num': 30},
|
||||
'Electronics': {'acc': 0.567, 'num': 30},
|
||||
'Energy_and_Power': {'acc': 0.633, 'num': 30},
|
||||
'Finance': {'acc': 0.733, 'num': 30},
|
||||
'Geography': {'acc': 0.333, 'num': 30},
|
||||
'History': {'acc': 0.533, 'num': 30},
|
||||
'Literature': {'acc': 0.533, 'num': 30},
|
||||
'Manage': {'acc': 0.6, 'num': 30},
|
||||
'Marketing': {'acc': 0.767, 'num': 30},
|
||||
'Materials': {'acc': 0.6, 'num': 30},
|
||||
'Math': {'acc': 0.7, 'num': 30},
|
||||
'Mechanical_Engineering': {'acc': 0.333, 'num': 30},
|
||||
'Music': {'acc': 0.4, 'num': 30},
|
||||
'Overall': {'acc': 0.523, 'num': 900},
|
||||
'Overall-Art and Design': {'acc': 0.483, 'num': 120},
|
||||
'Overall-Business': {'acc': 0.673, 'num': 150},
|
||||
'Overall-Health and Medicine': {'acc': 0.513, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.492, 'num': 120},
|
||||
'Overall-Science': {'acc': 0.5, 'num': 150},
|
||||
'Overall-Tech and Engineering': {'acc': 0.481, 'num': 210},
|
||||
'Pharmacy': {'acc': 0.6, 'num': 30},
|
||||
'Physics': {'acc': 0.7, 'num': 30},
|
||||
'Psychology': {'acc': 0.467, 'num': 30},
|
||||
'Public_Health': {'acc': 0.733, 'num': 30},
|
||||
'Sociology': {'acc': 0.433, 'num': 30}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.523
|
||||
```
|
||||
@@ -0,0 +1,528 @@
|
||||
---
|
||||
title: Step-3.5-Flash
|
||||
metatags:
|
||||
description: "Deploy Step-3.5 reasoning engine with SGLang. "
|
||||
---
|
||||
|
||||
import { Step35Deployment } from '/src/snippets/autoregressive/step-35-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is StepFun's production-grade reasoning engine built to decouple elite intelligence from heavy compute, and cuts attention cost for low-latency, cost-effective long-context inference—purpose-built for autonomous agents in real-world workflows. The model is available in multiple quantization formats optimized for different hardware platforms.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio and an aggressive 128-token window. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models.
|
||||
- **Sparse Mixture-of-Experts**: Only 11B active parameters out of 196B parameters.
|
||||
- **Multi-Layer Multi-Token Prediction (MTP)**: Equipped with a 3-way Multi-Token Prediction (MTP-3). This allows for complex, multi-step reasoning chains with immediate responsiveness.
|
||||
|
||||
## 2.SGLang Installation
|
||||
|
||||
Step-3.5-Flash is currently available in SGLang via Docker image install.
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash Command
|
||||
# Pull the docker image
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# Launch the container
|
||||
docker run -it --gpus all \
|
||||
--shm-size=32g \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
lmsysorg/sglang:latest bash
|
||||
```
|
||||
|
||||
### Docker (AMD ROCm)
|
||||
```bash Command
|
||||
# For MI300X/MI325X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
|
||||
|
||||
# For MI350X/MI355X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
|
||||
|
||||
docker run -it \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
--shm-size=32g \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
lmsysorg/sglang:v0.5.9-rocm700-mi30x bash # or mi35x for MI350X/MI355X
|
||||
```
|
||||
|
||||
## 3.Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Step-3.5-Flash series comes in only one sizes. Recommended starting configurations vary depending on hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
|
||||
|
||||
<Step35Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4×, TP=4), MI300X/MI325X/MI350X/MI355X (4×, TP=4 EP=4).
|
||||
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X and `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X.
|
||||
- **AMD Expert Parallelism Required**: On AMD GPUs, always use `--ep 4` with `--tp 4`. Both BF16 and FP8 models require expert parallelism. Without EP, the MoE intermediate dimension is split across GPUs (N=320), which triggers an AITER CK GEMM incompatibility. With EP=4, each GPU handles 72 full experts (N=1280), which works correctly with cuda graph enabled.
|
||||
- **AITER JIT Compilation**: First inference on AMD may take 30-40 seconds for AITER kernel JIT compilation. Subsequent requests use cached kernels.
|
||||
|
||||
## 4.Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
Step-3.5-Flash only supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path stepfun-ai/Step-3.5-Flash \
|
||||
--tp 4 \
|
||||
--ep 4 \
|
||||
--reasoning-parser step3p5
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.5-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We are asked: "What is 15% of 240?" We need to solve step by step.
|
||||
|
||||
Step 1: Understand that "15% of 240" means we need to calculate 15 percent of 240. In mathematical terms, it is (15/100) * 240.
|
||||
|
||||
Step 2: Simplify the calculation. We can compute 15% of 240 by first finding 10% of 240 and then 5% of 240, and adding them. Alternatively, we can multiply directly.
|
||||
|
||||
Method 1:
|
||||
10% of 240 = 240 * 0.10 = 24.
|
||||
5% is half of 10%, so 5% of 240 = 24 / 2 = 12.
|
||||
Then 15% = 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Method 2: Direct multiplication: 15% = 15/100 = 0.15, so 0.15 * 240 = 36.
|
||||
|
||||
We can also compute fractionally: (15/100)*240 = (15*240)/100. 15*240 = 3600, divided by 100 gives 36.
|
||||
|
||||
Thus, the answer is 36.
|
||||
|
||||
We'll present the solution step by step.
|
||||
|
||||
=============== Content =================
|
||||
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. **Convert the percentage to a decimal**:
|
||||
\( 15\% = \frac{15}{100} = 0.15 \)
|
||||
|
||||
2. **Multiply by the number**:
|
||||
\( 0.15 \times 240 = 36 \)
|
||||
|
||||
Alternatively, break it down:
|
||||
- \( 10\% \text{ of } 240 = 240 \times 0.10 = 24 \)
|
||||
- \( 5\% \text{ of } 240 = \frac{24}{2} = 12 \) (since 5% is half of 10%)
|
||||
- \( 15\% = 10\% + 5\% = 24 + 12 = 36 \)
|
||||
|
||||
**Answer:** 36
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Step-3.5 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Python Example:**
|
||||
|
||||
Start sglang server:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path stepfun-ai/Step-3.5-Flash \
|
||||
--tp 4 \
|
||||
--ep 4 \
|
||||
--reasoning-parser step3p5 \
|
||||
--tool-call-parser step3p5
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
import json
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# 1. define tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# 2. tool run
|
||||
def get_weather(location, unit="celsius"):
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# 3. send first request
|
||||
print("--- Sending first request ---")
|
||||
response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.5-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=1.0,
|
||||
stream=False
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
|
||||
# 4. Handle Reasoning Content
|
||||
reasoning = getattr(message, 'reasoning_content', None)
|
||||
if reasoning:
|
||||
print("=============== Thinking =================")
|
||||
print(reasoning)
|
||||
print("==========================================")
|
||||
|
||||
# 5. Handle Tool Calls
|
||||
if message.tool_calls:
|
||||
print("\n🔧 Tool Calls detected:")
|
||||
history_messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
message
|
||||
]
|
||||
|
||||
for tool_call in message.tool_calls:
|
||||
print(f" Tool: {tool_call.function.name}")
|
||||
print(f" Args: {tool_call.function.arguments}")
|
||||
|
||||
args = json.loads(tool_call.function.arguments)
|
||||
tool_result = get_weather(args.get("location"), args.get("unit", "celsius"))
|
||||
|
||||
history_messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": tool_result
|
||||
})
|
||||
|
||||
print("\n--- Sending tool results ---")
|
||||
final_response = client.chat.completions.create(
|
||||
model="stepfun-ai/Step-3.5-Flash",
|
||||
messages=history_messages,
|
||||
temperature=1.0,
|
||||
stream=False
|
||||
)
|
||||
|
||||
print("=============== Final Content =================")
|
||||
print(final_response.choices[0].message.content)
|
||||
|
||||
else:
|
||||
if message.content:
|
||||
print("=============== Content =================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
--- Sending first request ---
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I should use the get_weather function with location="Beijing". The unit parameter is optional and the user didn't specify a preference, so I'll leave it out (the default should be fine).
|
||||
|
||||
==========================================
|
||||
|
||||
🔧 Tool Calls detected:
|
||||
Tool: get_weather
|
||||
Args: {"location": "Beijing"}
|
||||
|
||||
--- Sending tool results ---
|
||||
=============== Final Content =================
|
||||
The weather in Beijing is 22°C and sunny.
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 GPU (4x)
|
||||
- Model: Step-3.5-Flash
|
||||
- Tensor Parallelism: 4
|
||||
- Expert Parallelism: 4
|
||||
- sglang version: 0.5.8
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path stepfun-ai/Step-3.5-Flash \
|
||||
--tp 4 \
|
||||
--ep 4
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model stepfun-ai/Step-3.5-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.30
|
||||
Total input tokens: 6091
|
||||
Total input text tokens: 6091
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4212
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 172.57
|
||||
Output token throughput (tok/s): 119.56
|
||||
Peak output token throughput (tok/s): 124.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 292.14
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3527.94
|
||||
Median E2E Latency (ms): 2884.72
|
||||
P90 E2E Latency (ms): 6350.38
|
||||
P99 E2E Latency (ms): 7858.53
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 107.53
|
||||
Median TTFT (ms): 80.93
|
||||
P99 TTFT (ms): 269.52
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.12
|
||||
Median TPOT (ms): 8.13
|
||||
P99 TPOT (ms): 8.14
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.12
|
||||
Median ITL (ms): 8.11
|
||||
P95 ITL (ms): 8.61
|
||||
P99 ITL (ms): 8.91
|
||||
Max ITL (ms): 20.77
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model stepfun-ai/Step-3.5-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 54.06
|
||||
Total input tokens: 39588
|
||||
Total input text tokens: 39588
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40479
|
||||
Request throughput (req/s): 1.48
|
||||
Input token throughput (tok/s): 732.33
|
||||
Output token throughput (tok/s): 754.84
|
||||
Peak output token throughput (tok/s): 928.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1487.17
|
||||
Concurrency: 14.06
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9501.23
|
||||
Median E2E Latency (ms): 10010.71
|
||||
P90 E2E Latency (ms): 15655.09
|
||||
P99 E2E Latency (ms): 18803.63
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 198.34
|
||||
Median TTFT (ms): 89.50
|
||||
P99 TTFT (ms): 984.66
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.97
|
||||
Median TPOT (ms): 18.80
|
||||
P99 TPOT (ms): 35.67
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.27
|
||||
Median ITL (ms): 17.48
|
||||
P95 ITL (ms): 18.44
|
||||
P99 ITL (ms): 62.47
|
||||
Max ITL (ms): 460.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model stepfun-ai/Step-3.5-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 125.88
|
||||
Total input tokens: 249331
|
||||
Total input text tokens: 249331
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 251323
|
||||
Request throughput (req/s): 3.97
|
||||
Input token throughput (tok/s): 1980.77
|
||||
Output token throughput (tok/s): 2007.23
|
||||
Peak output token throughput (tok/s): 2500.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 3987.99
|
||||
Concurrency: 92.25
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 23223.31
|
||||
Median E2E Latency (ms): 22631.90
|
||||
P90 E2E Latency (ms): 42269.38
|
||||
P99 E2E Latency (ms): 47637.53
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 372.13
|
||||
Median TTFT (ms): 127.26
|
||||
P99 TTFT (ms): 1880.42
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 46.06
|
||||
Median TPOT (ms): 47.61
|
||||
P99 TPOT (ms): 51.34
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 45.31
|
||||
Median ITL (ms): 39.86
|
||||
P95 ITL (ms): 72.49
|
||||
P99 ITL (ms): 117.05
|
||||
Max ITL (ms): 1359.81
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
- Step-3.5-Flash
|
||||
```
|
||||
Accuracy: 0.885
|
||||
Invalid: 0.005
|
||||
Latency: 9.986 s
|
||||
Output throughput: 1972.911 token/s
|
||||
```
|
||||
@@ -0,0 +1,526 @@
|
||||
---
|
||||
title: Hunyuan 3 Preview
|
||||
metatags:
|
||||
description: "Deploy Tencent Hunyuan 3 Preview BF16 (~276B / ~20B active MoE) on NVIDIA GPUs with SGLang — hybrid thinking, native tool calling, 256K context, and built-in MTP speculative decoding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
Hunyuan 3 Preview (Hy3-preview) is Tencent's preview of its third-generation flagship MoE language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **MoE Architecture**: 192 routed experts + 1 shared expert, 8 experts activated per token. ~276B total parameters with ~20B active, delivering dense-model quality at MoE inference cost.
|
||||
- **Hybrid Thinking**: Reasoning modes (`high`, `medium`, `low`, `none`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning.
|
||||
- **Native Tool Calling**: Trained on structured `<tool_call>` / `<arg_key>` / `<arg_value>` grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output.
|
||||
- **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
|
||||
- **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- [tencent/Hy3-preview](https://huggingface.co/tencent/Hy3-preview) — BF16 instruct
|
||||
- [tencent/Hy3-preview-Base](https://huggingface.co/tencent/Hy3-preview-Base) — BF16 base
|
||||
|
||||
**Recommended Generation Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`temperature`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`top_p`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`reasoning_effort`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`high` / `medium` / `low` (thinking) or `none` (instant)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** TODO — verify on HuggingFace model card.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA H200 / B200 / B300 / GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:latest`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
`lmsysorg/sglang:latest` bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization, and feature capabilities.
|
||||
|
||||
import { Hunyuan3PreviewDeployment } from '/src/snippets/autoregressive/hunyuan3-preview-deployment.jsx'
|
||||
|
||||
<Hunyuan3PreviewDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Key Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tool-call-parser`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tool call parser for function-calling support</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`hunyuan`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning parser for hybrid thinking modes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`hunyuan`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for Hunyuan model loading</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction (KV + activations)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.9`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tp`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor parallelism size</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2` / `4` / `8` depending on hardware</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention backend (Blackwell only)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`trtllm_mha`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-algorithm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Speculative decoding via the bundled MTP draft</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`EAGLE` + `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Hardware Requirements: NVIDIA BF16 (`Hy3-preview`, ~552GB weights)**
|
||||
|
||||
- **H200 (141GB) / B200 (180GB)**: TP=8 (minimum for BF16 to fit single-node).
|
||||
- **B300 (275GB) / GB300**: TP=4.
|
||||
- **A100 / H100 (80GB)**: not supported single-node — BF16 requires multi-node TP=16+ on 80GB-class GPUs.
|
||||
|
||||
**Blackwell (B200 / B300 / GB300):** Auto-selected attention backend can mis-route for HYV3 on Blackwell. Always pass `--attention-backend trtllm_mha` explicitly on Blackwell hardware (the config generator above enforces this).
|
||||
|
||||
**Multi-Token Prediction (MTP):** The `Hy3-preview` release bundles an MTP draft module. SGLang runs it via its EAGLE speculative-decoding path — the draft module auto-loads from the same `--model-path`. Enable with the standard MTP flags:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3-preview \
|
||||
--tp 8 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--reasoning-parser hunyuan \
|
||||
--tool-call-parser hunyuan \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
**Deployment Command (H200 × 8, BF16 default):**
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3-preview \
|
||||
--tp 8 \
|
||||
--reasoning-parser hunyuan \
|
||||
--tool-call-parser hunyuan \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.9
|
||||
```
|
||||
|
||||
**Testing Deployment:**
|
||||
|
||||
After startup, you can test the SGLang OpenAI-compatible API with the following command:
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "tencent/Hy3-preview",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Who won the world series in 2020?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
**Simple Completion Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3-preview",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Who won the world series in 2020?"}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("Content: ", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Reasoning: None
|
||||
Content: The Los Angeles Dodgers won the 2020 World Series. They defeated the Tampa Bay Rays in six games (4-2). This was the Dodgers' first World Series championship since 1988. The series was notable for being played in a neutral-site bubble at Globe Life Field in Arlington, Texas, due to the COVID-19 pandemic.
|
||||
```
|
||||
|
||||
When `reasoning_effort` is not set, the server defaults to instant mode (no thinking, `reasoning_content=None`). To opt into thinking, pass `reasoning_effort="high" / "medium" / "low"` on the request — see the Hybrid Thinking section below.
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser (Hybrid Thinking)
|
||||
|
||||
Hy3-preview is a hybrid-thinking model. Control the thinking budget via the OpenAI-standard `reasoning_effort`:
|
||||
|
||||
- `high` / `medium` / `low` — increasing amounts of chain-of-thought in `reasoning_content`
|
||||
- `none` — skip thinking entirely (instant responses, content-only)
|
||||
|
||||
Enable the reasoning parser during deployment so that the thinking section (`<think>...</think>`) is separated into `reasoning_content`:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3-preview \
|
||||
--tp 8 \
|
||||
--reasoning-parser hunyuan \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.9
|
||||
```
|
||||
|
||||
**Thinking Mode — High Effort:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3-preview",
|
||||
messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
|
||||
reasoning_effort="high",
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("=============== Thinking =================")
|
||||
print(msg.reasoning_content)
|
||||
print("=============== Content =================")
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to solve: "What is 15% of 240?" Step by step. So we need to compute 15% of 240. The process: 15% means 15 per hundred, i.e., 15/100 = 0.15. Multiply 0.15 by 240. Or we can do: 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. Or do multiplication: 15/100 * 240 = (15*240)/100 = (3600)/100 = 36. So answer is 36.
|
||||
|
||||
We need to produce step-by-step explanation. The instruction: "Solve step by step: What is 15% of 240?" So we should provide a clear solution with steps. The final answer: 36. Also maybe include units? No units.
|
||||
|
||||
We'll output the solution in a clear manner.
|
||||
=============== Content =================
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. **Understand that percent means "per hundred."**
|
||||
So, 15% = 15/100 or 0.15.
|
||||
|
||||
2. **Multiply the number (240) by the percentage in decimal form.**
|
||||
0.15 × 240.
|
||||
|
||||
Alternatively, you can use fractions:
|
||||
(15/100) × 240.
|
||||
|
||||
3. **Perform the multiplication.**
|
||||
0.15 × 240 = 36.
|
||||
Or:
|
||||
(15 × 240) / 100 = 3600 / 100 = 36.
|
||||
|
||||
4. **Check using an alternative method:**
|
||||
- 10% of 240 = 24.
|
||||
- 5% of 240 = half of 10% = 12.
|
||||
- 15% = 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Thus, **15% of 240 is 36**.
|
||||
```
|
||||
|
||||
**Instant Mode — No Thinking:**
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3-preview",
|
||||
messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
|
||||
reasoning_effort="none",
|
||||
max_tokens=256,
|
||||
)
|
||||
|
||||
print("Content:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Hy3-preview supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips thinking tokens before the tool-call parser runs:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3-preview \
|
||||
--tp 8 \
|
||||
--reasoning-parser hunyuan \
|
||||
--tool-call-parser hunyuan \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.9
|
||||
```
|
||||
|
||||
**Non-Streaming Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {"type": "string"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3-preview",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("Reasoning:", msg.reasoning_content)
|
||||
print("Content: ", msg.content)
|
||||
for tc in msg.tool_calls or []:
|
||||
print(f"Tool Call: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Reasoning: None
|
||||
Content: I'll get the current weather for Beijing in Fahrenheit for you.
|
||||
Tool Call: get_weather
|
||||
Arguments: {"city": "Beijing", "unit": "fahrenheit"}
|
||||
```
|
||||
|
||||
**Streaming Example (incremental argument deltas):**
|
||||
|
||||
Hy3-preview's `hunyuan` tool-call parser emits tool names first, then argument JSON in incremental fragments — matching the OpenAI streaming contract:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="tencent/Hy3-preview",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
|
||||
tools=tools,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
tool_buffer = {}
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
for tc in delta.tool_calls or []:
|
||||
buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
|
||||
if tc.function and tc.function.name:
|
||||
buf["name"] += tc.function.name
|
||||
if tc.function and tc.function.arguments:
|
||||
buf["args"] += tc.function.arguments
|
||||
|
||||
for idx, buf in tool_buffer.items():
|
||||
print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
I'll check the current weather in Beijing for you using Fahrenheit.
|
||||
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: 8× NVIDIA H200 (141GB)
|
||||
- Docker Image: `lmsysorg/sglang:hy3-preview`
|
||||
- Model: `tencent/Hy3-preview` (BF16)
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang version: latest `main`
|
||||
|
||||
#### 5.1.1 GSM8K
|
||||
|
||||
- Benchmark Method: 5-shot CoT on 200 questions, evaluated via SGLang native backend
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --parallel 64
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
TODO — replace with real GSM8K accuracy after benchmark run on Hy3-preview (BF16).
|
||||
```
|
||||
|
||||
#### 5.1.2 MMLU
|
||||
|
||||
- Benchmark Method: 5-shot, all 57 subjects
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --nsub 60 --parallel 64
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
TODO — replace with real MMLU accuracy after benchmark run on Hy3-preview (BF16).
|
||||
```
|
||||
|
||||
#### 5.1.3 Tool-Call Accuracy (MiniMax-Provider-Verifier)
|
||||
|
||||
- Benchmark Tool: [MiniMax-Provider-Verifier](https://github.com/MiniMax-AI/MiniMax-Provider-Verifier)
|
||||
- Metric: function-call schema validity, argument match, and end-to-end response correctness
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
TODO — replace with real tool-call accuracy after benchmark run on Hy3-preview (BF16).
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
#### 5.2.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model tencent/Hy3-preview \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
TODO — replace with real low-concurrency output on Hy3-preview (BF16).
|
||||
```
|
||||
|
||||
#### 5.2.2 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model tencent/Hy3-preview \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
TODO — replace with real high-concurrency output on Hy3-preview (BF16).
|
||||
```
|
||||
@@ -0,0 +1,370 @@
|
||||
---
|
||||
title: Hy3
|
||||
description: "Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
# Install from source (main carries the suffix-aware `hunyuan` parser + the
|
||||
# HYV3 model code). Once a tagged release picks it up, `uv pip install sglang`
|
||||
# is enough.
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser.
|
||||
docker pull lmsysorg/sglang:dev
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
<Note>
|
||||
The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding.
|
||||
</Note>
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command.
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/tencent/hy3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/tencent/hy3-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
The knobs come in two flavors:
|
||||
|
||||
- **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
|
||||
- **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime).
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost.
|
||||
- **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning.
|
||||
- **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output.
|
||||
- **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
|
||||
- **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box.
|
||||
|
||||
**Available Model:**
|
||||
|
||||
- [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct
|
||||
- [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8
|
||||
|
||||
**Recommended Generation Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #0052d9"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>temperature</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>top_p</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>reasoning_effort</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>high</code> / <code>low</code> (thinking) or <code>no_think</code> (instant)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `<tool_calls:TAG>` instead of the bare `<tool_calls>`). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model.
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Hardware requirements (BF16, ~590GB weights):**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #0052d9"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPU</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>VRAM</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>TP</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>minimum single-node for BF16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 590GB → 148GB/GPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>B300 / GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>288GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 590GB → 148GB/GPU; ample KV headroom</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>GB200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>192GB</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node 4×192GB = 768GB fits BF16 590GB</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason.
|
||||
|
||||
**MTP (Multi-Token Prediction, EAGLE).**
|
||||
|
||||
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
|
||||
- `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient.
|
||||
|
||||
**`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning (Hybrid Thinking)
|
||||
|
||||
Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`:
|
||||
|
||||
- `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content`
|
||||
- `no_think` — skip thinking entirely (instant responses, content-only)
|
||||
|
||||
Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`:
|
||||
|
||||
<Accordion title="Deploy with reasoning parser">
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3 \
|
||||
--tp 8 \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: thinking (reasoning_effort=high)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
|
||||
reasoning_effort="high",
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("=============== Thinking =================")
|
||||
print(msg.reasoning_content)
|
||||
print("=============== Content =================")
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240.
|
||||
10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36.
|
||||
=============== Content =================
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. 15% = 15/100 or 0.15.
|
||||
2. Multiply 240 by 0.15: 0.15 × 240 = 36.
|
||||
3. Check: 10% of 240 = 24, 5% = 12, 15% = 36.
|
||||
|
||||
Thus, 15% of 240 is 36.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: instant mode (reasoning_effort=no_think)">
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
|
||||
reasoning_effort="no_think",
|
||||
max_tokens=256,
|
||||
)
|
||||
|
||||
print("Content:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
```text Output
|
||||
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs:
|
||||
|
||||
<Accordion title="Deploy with tool-call parser">
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path tencent/Hy3 \
|
||||
--tp 8 \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: non-streaming tool call">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {"type": "string"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
print("Reasoning:", msg.reasoning_content)
|
||||
print("Content: ", msg.content)
|
||||
for tc in msg.tool_calls or []:
|
||||
print(f"Tool Call: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
```text Output
|
||||
Reasoning: None
|
||||
Content: I'll get the current weather for Beijing in Fahrenheit for you.
|
||||
Tool Call: get_weather
|
||||
Arguments: {"city": "Beijing", "unit": "fahrenheit"}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example: streaming tool call (incremental argument deltas)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="tencent/Hy3",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
|
||||
tools=tools,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
tool_buffer = {}
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
for tc in delta.tool_calls or []:
|
||||
buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
|
||||
if tc.function and tc.function.name:
|
||||
buf["name"] += tc.function.name
|
||||
if tc.function and tc.function.arguments:
|
||||
buf["args"] += tc.function.arguments
|
||||
|
||||
for idx, buf in tool_buffer.items():
|
||||
print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
|
||||
```
|
||||
|
||||
```text Output
|
||||
I'll check the current weather in Beijing for you using Fahrenheit.
|
||||
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,312 @@
|
||||
---
|
||||
title: Inkling-Small
|
||||
description: "Deploy Inkling-Small with SGLang — launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' Inkling-Small Mixture-of-Experts model."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
Inkling-Small has merged to `main` but isn't in a `pip` release yet — install from source:
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
<Note>The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status.</Note>
|
||||
|
||||
There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU. DGX Spark (GB10) uses a dedicated arm64 CUDA 13 image:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13
|
||||
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12
|
||||
docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121)
|
||||
docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/thinkingmachines/inkling-small.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Inkling-Small** is a Mixture-of-Experts model from Thinking Machines with **open weights** (BF16 and NVFP4 checkpoints below), in the same architecture family as Inkling. It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling-Small on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache).
|
||||
|
||||
**Resources:** HuggingFace — [Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small) (BF16) · [Inkling-Small-NVFP4](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving.
|
||||
|
||||
**DGX Spark (2× GB10).** The verified cell runs NVFP4 with TP=2 across two Sparks over ConnectX-7 (1 GPU per node). Use the `dev-inkling-small-dgx-spark` image, Triton attention + Marlin FP4/MoE, and `--disable-prefill-cuda-graph`. The Docker command already carries the ConnectX-7 flags `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`.
|
||||
|
||||
**Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage.
|
||||
|
||||
**MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage.
|
||||
|
||||
**Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):
|
||||
|
||||
<table style={{width: "60%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "8px 12px", fontWeight: 700}}>reasoning_effort</th>
|
||||
<th style={{textAlign: "left", padding: "8px 12px", fontWeight: 700}}>value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>none</code></td><td style={{padding: "6px 12px"}}>0.0</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>minimal</code></td><td style={{padding: "6px 12px"}}>0.1</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>low</code></td><td style={{padding: "6px 12px"}}>0.2</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>medium</code></td><td style={{padding: "6px 12px"}}>0.7</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>high</code></td><td style={{padding: "6px 12px"}}>0.9</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>xhigh</code></td><td style={{padding: "6px 12px"}}>0.99</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>max</code></td><td style={{padding: "6px 12px"}}>0.99</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-Small-NVFP4",
|
||||
messages=[{"role": "user", "content": "What is 17 times 24?"}],
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user is asking for the product of 17 and 24. Let me calculate that.
|
||||
|
||||
17 × 24
|
||||
|
||||
I can break this down:
|
||||
17 × 20 = 340
|
||||
17 × 4 = 68
|
||||
340 + 68 = 408
|
||||
|
||||
Alternatively:
|
||||
24 × 10 = 240
|
||||
24 × 7 = 168
|
||||
240 + 168 = 408
|
||||
|
||||
So the answer is 408.
|
||||
Answer: 17 times 24 is **408**.
|
||||
|
||||
Here's a quick breakdown:
|
||||
- 17 × 20 = 340
|
||||
- 17 × 4 = 68
|
||||
- 340 + 68 = **408**
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"location": {"type": "string", "description": "The city name"}},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-Small-NVFP4",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Content:", msg.content)
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location.
|
||||
Content:
|
||||
Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 Multimodal Input (Image + Audio)
|
||||
|
||||
Inkling-Small is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. The server must be started with `--enable-multimodal` (already included in every recipe above).
|
||||
|
||||
<Accordion title="Image + Audio Example (Python)">
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
with open("image.png", "rb") as f:
|
||||
image_b64 = base64.b64encode(f.read()).decode()
|
||||
with open("audio.wav", "rb") as f:
|
||||
audio_b64 = base64.b64encode(f.read()).decode()
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-Small-NVFP4",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
|
||||
{"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
|
||||
{"type": "text", "text": "Describe the image, then transcribe the audio."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows.
|
||||
</Note>
|
||||
|
||||
### 3.4 LoRA (Serving Adapters)
|
||||
|
||||
The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name.
|
||||
|
||||
Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`:
|
||||
|
||||
<Accordion title="LoRA Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
# Option A (recommended): "<model>:<adapter-name>" in the model field
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-Small-NVFP4:lora0",
|
||||
messages=[{"role": "user", "content": "Summarize the changelog."}],
|
||||
)
|
||||
|
||||
# Option B: explicit lora_path via extra_body
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-Small-NVFP4",
|
||||
messages=[{"role": "user", "content": "Summarize the changelog."}],
|
||||
extra_body={"lora_path": "lora0"},
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence.
|
||||
</Note>
|
||||
|
||||
### 3.5 HiCache (Hierarchical KV Caching)
|
||||
|
||||
Inkling-Small serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads.
|
||||
|
||||
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`.
|
||||
|
||||
### 3.6 Long Context (MXFP8 KV)
|
||||
|
||||
The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound.
|
||||
|
||||
**Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200).
|
||||
|
||||
The tradeoff is not just a ~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality.
|
||||
|
||||
To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300.
|
||||
|
||||
### 3.7 DSpark (Speculative Decoding)
|
||||
|
||||
The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark-Preview` — served unquantized alongside the NVFP4 target.
|
||||
|
||||
DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).
|
||||
@@ -0,0 +1,309 @@
|
||||
---
|
||||
title: Inkling
|
||||
description: "Deploy Inkling with SGLang — verified launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' 975B Mixture-of-Experts model with 1M-token context."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
Inkling has merged to `main` but isn't in a `pip` release yet — install from source:
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
<Note>The Inkling images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status.</Note>
|
||||
|
||||
There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13
|
||||
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12
|
||||
docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/thinkingmachines/inkling.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/thinkingmachines/inkling-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Inkling** is a Mixture-of-Experts model from Thinking Machines — **975B** total parameters, **41B** active per token, with a **1M-token** context window and **open weights** (BF16 and NVFP4 checkpoints below). It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache).
|
||||
|
||||
**Resources:** HuggingFace — [Inkling](https://huggingface.co/thinkingmachines/Inkling) (BF16) · [Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving.
|
||||
|
||||
**Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage.
|
||||
|
||||
**MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage.
|
||||
|
||||
**Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):
|
||||
|
||||
<table style={{width: "60%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "8px 12px", fontWeight: 700}}>reasoning_effort</th>
|
||||
<th style={{textAlign: "left", padding: "8px 12px", fontWeight: 700}}>value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>none</code></td><td style={{padding: "6px 12px"}}>0.0</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>minimal</code></td><td style={{padding: "6px 12px"}}>0.1</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>low</code></td><td style={{padding: "6px 12px"}}>0.2</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>medium</code></td><td style={{padding: "6px 12px"}}>0.7</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>high</code></td><td style={{padding: "6px 12px"}}>0.9</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>xhigh</code></td><td style={{padding: "6px 12px"}}>0.99</td></tr>
|
||||
<tr><td style={{padding: "6px 12px"}}><code>max</code></td><td style={{padding: "6px 12px"}}>0.99</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-NVFP4",
|
||||
messages=[{"role": "user", "content": "What is 17 times 24?"}],
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user is asking for the product of 17 and 24. Let me calculate that.
|
||||
|
||||
17 × 24
|
||||
|
||||
I can break this down:
|
||||
17 × 20 = 340
|
||||
17 × 4 = 68
|
||||
340 + 68 = 408
|
||||
|
||||
Alternatively:
|
||||
24 × 10 = 240
|
||||
24 × 7 = 168
|
||||
240 + 168 = 408
|
||||
|
||||
So the answer is 408.
|
||||
Answer: 17 times 24 is **408**.
|
||||
|
||||
Here's a quick breakdown:
|
||||
- 17 × 20 = 340
|
||||
- 17 × 4 = 68
|
||||
- 340 + 68 = **408**
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"location": {"type": "string", "description": "The city name"}},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-NVFP4",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Content:", msg.content)
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location.
|
||||
Content:
|
||||
Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 Multimodal Input (Image + Audio)
|
||||
|
||||
Inkling is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. The server must be started with `--enable-multimodal` (already included in every recipe above).
|
||||
|
||||
<Accordion title="Image + Audio Example (Python)">
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
with open("image.png", "rb") as f:
|
||||
image_b64 = base64.b64encode(f.read()).decode()
|
||||
with open("audio.wav", "rb") as f:
|
||||
audio_b64 = base64.b64encode(f.read()).decode()
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-NVFP4",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
|
||||
{"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
|
||||
{"type": "text", "text": "Describe the image, then transcribe the audio."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows.
|
||||
</Note>
|
||||
|
||||
### 3.4 LoRA (Serving Adapters)
|
||||
|
||||
The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name.
|
||||
|
||||
Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`:
|
||||
|
||||
<Accordion title="LoRA Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
# Option A (recommended): "<model>:<adapter-name>" in the model field
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-NVFP4:lora0",
|
||||
messages=[{"role": "user", "content": "Summarize the changelog."}],
|
||||
)
|
||||
|
||||
# Option B: explicit lora_path via extra_body
|
||||
resp = client.chat.completions.create(
|
||||
model="thinkingmachines/Inkling-NVFP4",
|
||||
messages=[{"role": "user", "content": "Summarize the changelog."}],
|
||||
extra_body={"lora_path": "lora0"},
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence.
|
||||
</Note>
|
||||
|
||||
### 3.5 HiCache (Hierarchical KV Caching)
|
||||
|
||||
Inkling serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads.
|
||||
|
||||
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`.
|
||||
|
||||
### 3.6 Long Context (MXFP8 KV)
|
||||
|
||||
The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound.
|
||||
|
||||
**Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200).
|
||||
|
||||
The tradeoff is not just a ~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality.
|
||||
|
||||
To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300.
|
||||
|
||||
### 3.7 DSpark (Speculative Decoding)
|
||||
|
||||
The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target.
|
||||
|
||||
DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: MiMo-V2-Flash
|
||||
metatags:
|
||||
description: "Deploy MiMo-V2-Flash 309B MoE model with SGLang - hybrid attention, multi-token prediction, and 256K context for efficient inference."
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
XiaomiMiMo/MiMo-V2-Flash, with 309B total parameters and 15B activated parameters, is a new inference-centric model designed to maximize decoding efficiency created by XiaomiMiMo Team explicitly co-designed for real-world serving workloads, enabling flexible tradeoffs between throughput and latency on different hardware.
|
||||
|
||||
This model creates a new balance between long-context modeling capability and inference efficiency. Key features include:
|
||||
- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and an aggressive 128-token window. This reduces KV-cache storage by nearly 6x while maintaining long-context performance via learnable attention sink bias.
|
||||
- **Multi-Token Prediction (MTP)**: Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training.
|
||||
- **Efficient Pre-Training**: Trained on 27T tokens using FP8 mixed precision and native 32k seq length. The context window supports up to 256k length.
|
||||
- **Agentic Capabilities**: Post-training utilizes Multi-Teacher On-Policy Distillation (MOPD) and large-scale agentic RL, achieving superior performance on SWE-Bench and complex reasoning tasks.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
MiMo-V2-Flash is currently available in SGLang via Docker image and pip install.
|
||||
|
||||
### Docker
|
||||
|
||||
```bash Command
|
||||
# Pull the docker image
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# Launch the container
|
||||
docker run -it --gpus all \
|
||||
--shm-size=32g \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
lmsysorg/sglang:latest bash
|
||||
```
|
||||
|
||||
### Pip Installation
|
||||
|
||||
```bash Command
|
||||
# On a machine with SGLang dependencies installed or inside a SGLang nightly container
|
||||
# Start an SGLang nightly container
|
||||
docker run -it --gpus all \
|
||||
--shm-size=32g \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
lmsysorg/sglang:latest bash
|
||||
|
||||
# If you already have SGLang installed, uninstall the current SGLang version
|
||||
pip uninstall sglang -y
|
||||
|
||||
# Install the PyPI Package
|
||||
pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \
|
||||
--extra-index-url https://sgl-project.github.io/whl/pr/
|
||||
```
|
||||
|
||||
## Model Deployment
|
||||
|
||||
Use the configuration selector below to automatically generate the appropriate deployment command.
|
||||
|
||||
import { MiMoV2FlashDeployment } from "/src/snippets/autoregressive/mimo-v2-flash-deployment.jsx";
|
||||
|
||||
<MiMoV2FlashDeployment />
|
||||
|
||||
MI355X (ROCm) is validated in the selector above with `--tp-size 4`, Triton attention, and `--disable-custom-all-reduce`. `--tp-size 8` hit a QKV sharding error during validation. EAGLE speculative decoding is still WIP on MI355X.
|
||||
|
||||
## Testing the deployment
|
||||
|
||||
Once the server is running, test it with a chat completion request in another terminal:
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "XiaomiMiMo/MiMo-V2-Flash",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello! What can you help me with?"}
|
||||
],
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected response:**
|
||||
|
||||
```json Config
|
||||
{
|
||||
"id": "...",
|
||||
"object": "chat.completion",
|
||||
"model": "XiaomiMiMo/MiMo-V2-Flash",
|
||||
"choices": [{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "Hello! I can help you with..."
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**DeepGEMM Timeout Error**
|
||||
|
||||
Occasionally DeepGEMM timeout errors occur during first launch. Simply rerun the server command in the same container - the compiled kernels are cached and subsequent launches will be fast.
|
||||
|
||||
**ROCm MI355X Attention Backend**
|
||||
|
||||
If you see an error such as `AiterAttnBackend.forward_decode() got an unexpected keyword argument 'sinks'` on MI355X, use the `MI355X` + `Performance Optimizations` command from the selector above, which switches to Triton attention and keeps `--disable-custom-all-reduce`.
|
||||
@@ -0,0 +1,842 @@
|
||||
---
|
||||
title: MiMo-V2.5
|
||||
metatags:
|
||||
description: "Deploy XiaomiMiMo MiMo-V2.5-Pro (1.02T MoE, text) and MiMo-V2.5 (310B MoE, multimodal) with SGLang — EAGLE speculative decoding, hybrid attention, and 1M-token context."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiMo-V2.5-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro) and [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5) are next-generation Mixture-of-Experts models from the XiaomiMiMo Team.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "45%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Total params</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Active (MoE)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Modalities</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro">MiMo-V2.5-Pro</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.02T</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>42B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Text (multimodal planned)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/XiaomiMiMo/MiMo-V2.5">MiMo-V2.5</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>310B</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>15B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Text, Image, Video, Audio</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) for reduced KV cache while preserving long-context capability.
|
||||
- **Multi-Token Prediction (MTP)**: 3-layer MTP module accelerates decoding. Both variants support EAGLE speculative decoding with MTP weights.
|
||||
- **1M-Token Context**: Both variants support up to 1 million token context windows.
|
||||
- **Agentic Capabilities**: Post-training with large-scale agentic RL achieves strong performance on coding, reasoning, and tool-use benchmarks.
|
||||
- **MiMo-V2.5 Multimodal** (V2.5 only): Native omnimodal architecture with a 729M-param ViT Vision Encoder (28 layers: 24 SWA + 4 Full) and a 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full); supports image, video, and audio understanding via standard OpenAI-compatible multimodal API.
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
**Docker Image:** All variants (MiMo-V2.5 310B and MiMo-V2.5-Pro 1.02T) use `lmsysorg/sglang:latest`, which ships CUDA 13.0 and runs on both Hopper (H100 / H200) and Blackwell (B200 / GB300).
|
||||
|
||||
**TPU (sgl-jax):** MiMo-V2.5-Pro can also be served on TPU via the JAX-based [sgl-jax](https://github.com/sgl-project/sglang-jax) runtime. The container image and `pip install` steps are listed in [§3.3 TPU Deployment](#3-3-tpu-deployment-mimo-v2-5-pro-sgl-jax).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Use the selector below to generate the deployment command for your variant and hardware.
|
||||
|
||||
import { MiMoV25Deployment } from '/src/snippets/autoregressive/mimo-v25-deployment.jsx'
|
||||
|
||||
<MiMoV25Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**MiMo-V2.5-Pro (1.02T):**
|
||||
- **B200**: single node, TP=8 (verified). Uses `--attention-backend fa4` + `--moe-runner-backend flashinfer_trtllm` + `--mem-fraction-static 0.8`. Set `--swa-full-tokens-ratio 0.1` to keep KV-cache footprint within 192 GB HBM.
|
||||
- **GB300**: 2 nodes, TP=8 (verified). Same Blackwell stack as B200; multi-node interconnect requires `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`. Default SWA ratio is fine.
|
||||
- **H100/H200**: 2 nodes × 8 GPUs (TP=16, not yet verified). Uses the Hopper stack (`fa3` + DeepEP + EAGLE multi-layer); fits with `--mem-fraction-static 0.7` and `--swa-full-tokens-ratio 0.3`. DeepEP dispatch tuning: `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` avoids memory spikes during prefill.
|
||||
- EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `--enable-multi-layer-eagle` (both Hopper and Blackwell). See §5.4 for acceptance-rate behavior on natural text vs random prompts.
|
||||
|
||||
**MiMo-V2.5 (310B):**
|
||||
- The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--dp = TP / 4`; for TP > 4 this also requires DP-attention. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--dp 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`.
|
||||
- Single-node deployments: H100/H200 8× GPUs (`--tp 8 --dp 2`), B200 4× GPUs (`--tp 4`, dp=1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization.
|
||||
- On Blackwell, pass `--attention-backend fa4`: MiMoV2's asymmetric KV (`head_dim` 192 / `v_head_dim` 128) fails on the SM100 default `trtllm_mha`, which requires equal K/V widths.
|
||||
- On Blackwell, pass `--mm-attention-backend fa4` for the V2.5 vision encoder. The checkpoint config requests FlashAttention-3 internally, but SGLang rejects FA3 on Blackwell and expects FA4 for multimodal attention.
|
||||
- On Blackwell, pass `--moe-runner-backend flashinfer_trtllm`; the default `auto` falls through to the triton fused-MoE runner, ~12% slower at bs=1 decode.
|
||||
- `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--enable-dp-attention` is on, to keep LM head and encoder sharding consistent.
|
||||
- EAGLE MTP uses the checkpoint's MTP weights. Enable with `--speculative-algorithm EAGLE` and `--enable-multi-layer-eagle` (both Hopper and Blackwell).
|
||||
- **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples.
|
||||
|
||||
**DeepEP (optional toggle, Hopper-only):**
|
||||
- DeepEP replaces the default MoE all-to-all dispatch with a fused [DeepEP](https://github.com/deepseek-ai/DeepEP) backend; it lowers expert dispatch latency and memory traffic, so it pays off under **high concurrency / throughput-bound workloads** on H100/H200. Under concurrency=1 / latency-bound workloads the gain is negligible — leave it off.
|
||||
- Enabling adds `--moe-a2a-backend deepep` + `--moe-dense-tp-size 1` (and `--ep <tp>` for Pro) plus `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` env to cap the dispatch buffer. Requires `pip install deep_ep` (not part of the default sglang install).
|
||||
- On Blackwell (B200, GB300) the verified MoE backend is `flashinfer_trtllm`; the DeepEP toggle is a no-op there.
|
||||
|
||||
### 3.3 TPU Deployment (MiMo-V2.5-Pro, sgl-jax)
|
||||
|
||||
MiMo-V2.5-Pro can also be served on TPU via [sgl-jax](https://github.com/sgl-project/sglang-jax). The runtime is a separate JAX-based stack (`sgl_jax.launch_server`); pick **TPU v7x** or **TPU v6e** in the panel above to generate the launch command. Verified topologies:
|
||||
|
||||
| TPU Type | Topology | Chips/Node | Nodes | Total Chips | JAX Devices/Chip | Total JAX Devices (= `--tp-size`) |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| **v7x** | 2×2×4 | 4 | 4 | 16 | 2 | 32 |
|
||||
| **v6e** | 4×4×4 | 4 | 16 | 64 | 1 | 64 |
|
||||
|
||||
> v7x exposes **2 logical JAX devices per chip**, so `--tp-size = 16 chips × 2 = 32`. v6e exposes 1 device per chip, so `--tp-size = 64`. Always set `--tp-size` to the total JAX device count across all nodes, not the chip count.
|
||||
|
||||
All nodes must sit in the same TPU slice and reach each other on the JAX init port (`20000`) and the TPU process port (`8471`).
|
||||
|
||||
**Step 1 — Launch the JAX TPU container on every node:**
|
||||
|
||||
```shell Command
|
||||
docker run -it --privileged \
|
||||
--shm-size=32g \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
-v /dev:/dev \
|
||||
us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:jax0.8.1-rev1 bash
|
||||
```
|
||||
|
||||
> The image is pinned to `jax0.8.1-rev1` to keep the JAX runtime aligned with sgl-jax's TPU extras.
|
||||
|
||||
**Step 2 — Clone and install sgl-jax (inside the container):**
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/sgl-project/sglang-jax.git
|
||||
cd sglang-jax
|
||||
pip install -e "python[tpu]"
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
See [Basic API Usage](../../../docs/basic_usage/send_request).
|
||||
|
||||
### 4.2 Reasoning Output
|
||||
|
||||
Both variants support hybrid thinking mode. Thinking content is separated via the reasoning parser.
|
||||
|
||||
**Thinking Mode (default):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="XiaomiMiMo/MiMo-V2.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."}
|
||||
]
|
||||
)
|
||||
|
||||
print("====== Reasoning ======")
|
||||
print(response.choices[0].message.reasoning_content)
|
||||
print("====== Answer ======")
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example (MiMo-V2.5):**
|
||||
|
||||
```text
|
||||
====== Reasoning ======
|
||||
Comparing 9.11 and 9.9.
|
||||
|
||||
The integer parts are both 9. Now compare the decimal parts: 0.11 vs 0.9.
|
||||
|
||||
0.9 = 0.90, which is greater than 0.11.
|
||||
|
||||
So 9.9 > 9.11.
|
||||
====== Answer ======
|
||||
**9.9 is larger than 9.11.**
|
||||
|
||||
Here's the reasoning: When comparing decimals, line them up to the same number of decimal places:
|
||||
|
||||
- 9.11
|
||||
- 9.90
|
||||
|
||||
Both have a **9** in the ones place, but in the tenths place, **9 > 1**, so 9.90 > 0.11.
|
||||
|
||||
**9.9 > 9.11**
|
||||
```
|
||||
|
||||
**Thinking Off (instant mode):**
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="XiaomiMiMo/MiMo-V2.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"thinking": False}}
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example (MiMo-V2.5):**
|
||||
|
||||
```text
|
||||
## Comparing 9.11 and 9.9
|
||||
|
||||
**9.9 is larger.**
|
||||
|
||||
The key is to compare them place by place. It helps to write them with the same number of decimal places:
|
||||
|
||||
- **9.11** → 9.11
|
||||
- **9.9** → 9.90
|
||||
|
||||
Both have **9** in the ones place, but in the tenths place: **9** (in 9.90) is greater than **1** (in 9.11).
|
||||
|
||||
So **9.90 > 9.11**.
|
||||
```
|
||||
|
||||
### 4.3 Multimodal Invocation (V2.5 only)
|
||||
|
||||
**Image Understanding:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="XiaomiMiMo/MiMo-V2.5",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/man_ironing_on_back_of_suv.png"}},
|
||||
{"type": "text", "text": "Describe this image in detail."}
|
||||
]
|
||||
}]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text
|
||||
Based on the image provided, here is a detailed description:
|
||||
|
||||
The image captures a whimsical or surreal scene set on a busy city street, likely in New York City given the iconic yellow cabs. In the center foreground, a man is sitting on a folding chair, casually crossing his legs. He is wearing a bright yellow hoodie with a graphic on the front and blue jeans. He is intently focused on ironing a white dress shirt that rests on an ironing board set up directly on the asphalt.
|
||||
|
||||
Behind him, a yellow SUV taxi cab is stopped or moving slowly, angled slightly away from the camera. To his left, another yellow taxi sedan is captured in motion blur, indicating it is driving past him. The background features tall city buildings with glass windows and storefronts. There are banners hanging from streetlights, and some greenery is visible in the distance. The overall impression is one of incongruity—performing a domestic chore like ironing in the middle of a chaotic urban environment.
|
||||
```
|
||||
|
||||
**Video Understanding:**
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="XiaomiMiMo/MiMo-V2.5",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "video_url", "video_url": {"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"}},
|
||||
{"type": "text", "text": "Summarize what happens in this video."}
|
||||
]
|
||||
}]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text
|
||||
A person wearing blue protective gloves is shown operating a microscope in a close-up shot. The individual is adjusting a knob on the side of the microscope, which moves the stage holding a glass slide, likely focusing the lens on the specimen.
|
||||
```
|
||||
|
||||
> Video decoding requires `decord` (`pip install decord`); SGLang's MiMo-V2.5 multimodal processor uses `decord.VideoReader` for frame extraction.
|
||||
|
||||
**Audio Understanding:**
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="XiaomiMiMo/MiMo-V2.5",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "audio_url", "audio_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/audios/Trump_WEF_2018_10s.mp3"}},
|
||||
{"type": "text", "text": "Transcribe and summarize this audio."}
|
||||
]
|
||||
}]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text
|
||||
**Transcript:**
|
||||
"Thank you Klaus very much. It's a privilege to be here at this forum where leaders in business, science, art, diplomacy and world affairs have gathered for..."
|
||||
|
||||
**Summary:**
|
||||
The speaker thanks Klaus for the introduction and expresses their honor at attending a forum. They highlight that the event has brought together high-level leaders from various sectors, including business, science, art, and diplomacy.
|
||||
```
|
||||
|
||||
### 4.4 Tool Calling
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="XiaomiMiMo/MiMo-V2.5",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
if msg.reasoning_content:
|
||||
print("=== Reasoning ===")
|
||||
print(msg.reasoning_content)
|
||||
if msg.tool_calls:
|
||||
print("=== Tool Calls ===")
|
||||
for tc in msg.tool_calls:
|
||||
print(f" Function: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output Example (MiMo-V2.5):**
|
||||
|
||||
```text
|
||||
=== Reasoning ===
|
||||
The user wants to know the weather in Beijing. I have a function available called "get_weather" that can retrieve current weather for a location. Let me call that function with Beijing as the location.
|
||||
=== Tool Calls ===
|
||||
Function: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
Accuracy numbers come from `sglang.test.run_eval` (GSM8K standard 5-shot, MMMU validation split). Speed numbers come from `sglang.bench_serving` with generated random prompts; text runs use 1024 input tokens and 1024 output tokens per request, and the image run uses 2 random 720p images per request.
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
#### 5.1.1 GSM8K
|
||||
|
||||
Standard 5-shot, `temperature=0`, `max_tokens=4096`, model defaults to thinking-on (responses contain `<think>...</think>` and the eval extracts the trailing number via regex). Server launch: see [Section 3](#3-model-deployment).
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.run_eval \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--eval-name gsm8k \
|
||||
--num-examples 200 \
|
||||
--num-threads 8 \
|
||||
--max-tokens 4096 \
|
||||
--temperature 0.0
|
||||
```
|
||||
|
||||
> `run_eval.py` automatically appends `/v1` to `--base-url`; pass the bare `host:port` URL (without trailing `/v1`), otherwise requests resolve to `/v1/v1/chat/completions` and 404.
|
||||
|
||||
- **Test Results:**
|
||||
- MiMo-V2.5-Pro (FP8, 8× B200)
|
||||
```
|
||||
Score: 0.965 (193 / 200)
|
||||
Latency: 253.90 s
|
||||
Output throughput: 461.78 tok/s
|
||||
```
|
||||
- MiMo-V2.5 (FP8, 8× H200)
|
||||
```
|
||||
Score: 0.980 (196 / 200)
|
||||
Latency: 477.52 s
|
||||
Output throughput: 88.9 tok/s
|
||||
```
|
||||
|
||||
#### 5.1.2 MMMU (V2.5 only)
|
||||
|
||||
`MMMU/MMMU` validation split (multi-discipline multimodal), `concurrency=16`, default sampling.
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 benchmark/mmmu/bench_sglang.py \
|
||||
--port 30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--concurrency 16
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
- MiMo-V2.5 (FP8)
|
||||
```
|
||||
Pending update
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark — MiMo-V2.5-Pro
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8×)
|
||||
- Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
|
||||
- Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`)
|
||||
- Recipe: Blackwell verified baseline (EAGLE off for this benchmark — see note below)
|
||||
- sglang version: 0.5.11
|
||||
|
||||
> The numbers in §5.2 are the **no-EAGLE baseline** on `random 1024/1024`. On uniform-random token streams the MiMo-V2.5-Pro 3-layer MTP draft has very low accept-rate (~0.13–0.27 vs ~0.75 on natural-text prompts, see §5.4) — there's no token-co-occurrence signal for the draft to model — so EAGLE here adds verify overhead without recovering enough draft tokens to be a net win on this workload. EAGLE MTP itself works on B200 + `--enable-multi-layer-eagle` (see §3 deployment command and §5.4 for an acceptance profile on natural text).
|
||||
|
||||
#### 5.2.1 Latency-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** see the [command panel above](#3-model-deployment).
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5-Pro \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 27.59
|
||||
Total input tokens: 1997
|
||||
Total input text tokens: 1997
|
||||
Total generated tokens: 2798
|
||||
Total generated tokens (retokenized): 2794
|
||||
Request throughput (req/s): 0.36
|
||||
Input token throughput (tok/s): 72.38
|
||||
Output token throughput (tok/s): 101.41
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 173.79
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2757.26
|
||||
Median E2E Latency (ms): 3319.10
|
||||
P90 E2E Latency (ms): 4157.47
|
||||
P99 E2E Latency (ms): 4869.32
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 162.17
|
||||
Median TTFT (ms): 68.11
|
||||
P99 TTFT (ms): 929.58
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.19
|
||||
Median TPOT (ms): 9.33
|
||||
P99 TPOT (ms): 9.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.31
|
||||
Median ITL (ms): 9.35
|
||||
P95 ITL (ms): 9.44
|
||||
P99 ITL (ms): 9.77
|
||||
Max ITL (ms): 19.80
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** see the [command panel above](#3-model-deployment).
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5-Pro \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 112.78
|
||||
Total input tokens: 302118
|
||||
Total input text tokens: 302118
|
||||
Total generated tokens: 195775
|
||||
Total generated tokens (retokenized): 191069
|
||||
Request throughput (req/s): 8.87
|
||||
Input token throughput (tok/s): 2678.83
|
||||
Output token throughput (tok/s): 1735.90
|
||||
Peak output token throughput (tok/s): 3040.00
|
||||
Peak concurrent requests: 121
|
||||
Total token throughput (tok/s): 4414.73
|
||||
Concurrency: 87.80
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9901.96
|
||||
Median E2E Latency (ms): 6525.54
|
||||
P90 E2E Latency (ms): 23567.98
|
||||
P99 E2E Latency (ms): 42109.22
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 223.69
|
||||
Median TTFT (ms): 139.45
|
||||
P99 TTFT (ms): 1082.02
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 50.63
|
||||
Median TPOT (ms): 51.66
|
||||
P99 TPOT (ms): 91.41
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 49.79
|
||||
Median ITL (ms): 33.69
|
||||
P95 ITL (ms): 103.37
|
||||
P99 ITL (ms): 151.34
|
||||
Max ITL (ms): 1600.00
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.3 Speed Benchmark — MiMo-V2.5
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 GPU (8×)
|
||||
- Model: `XiaomiMiMo/MiMo-V2.5` (FP8)
|
||||
- Tensor Parallelism: 8 (DP-attention with `--dp 2`)
|
||||
- Recipe: Balanced (DP-attn + EAGLE MTP)
|
||||
- sglang version: `0.0.0.dev1+g7d99af439` (`lmsysorg/sglang:dev-mimo-v2.5`)
|
||||
|
||||
#### 5.3.1 Latency-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 14.72
|
||||
Total input tokens: 1997
|
||||
Total input text tokens: 1997
|
||||
Total generated tokens: 2798
|
||||
Total generated tokens (retokenized): 2697
|
||||
Request throughput (req/s): 0.68
|
||||
Input token throughput (tok/s): 135.67
|
||||
Output token throughput (tok/s): 190.09
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 325.77
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.08
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1469.98
|
||||
Median E2E Latency (ms): 1652.84
|
||||
P90 E2E Latency (ms): 2210.80
|
||||
P99 E2E Latency (ms): 2823.86
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 143.89
|
||||
Median TTFT (ms): 99.25
|
||||
P99 TTFT (ms): 481.01
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.87
|
||||
Median TPOT (ms): 4.30
|
||||
P99 TPOT (ms): 6.64
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.76
|
||||
Median ITL (ms): 3.46
|
||||
P95 ITL (ms): 13.52
|
||||
P99 ITL (ms): 13.84
|
||||
Max ITL (ms): 74.37
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.3.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 93.41
|
||||
Total input tokens: 302118
|
||||
Total input text tokens: 302118
|
||||
Total generated tokens: 195775
|
||||
Total generated tokens (retokenized): 188139
|
||||
Request throughput (req/s): 10.71
|
||||
Input token throughput (tok/s): 3234.48
|
||||
Output token throughput (tok/s): 2095.97
|
||||
Peak output token throughput (tok/s): 3019.00
|
||||
Peak concurrent requests: 121
|
||||
Total token throughput (tok/s): 5330.45
|
||||
Concurrency: 91.04
|
||||
Accept length: 2.95
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8503.45
|
||||
Median E2E Latency (ms): 7491.96
|
||||
P90 E2E Latency (ms): 13706.99
|
||||
P99 E2E Latency (ms): 20474.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4399.20
|
||||
Median TTFT (ms): 4333.35
|
||||
P99 TTFT (ms): 8004.81
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 58.23
|
||||
Median TPOT (ms): 21.78
|
||||
P99 TPOT (ms): 747.79
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.06
|
||||
Median ITL (ms): 15.28
|
||||
P95 ITL (ms): 48.36
|
||||
P99 ITL (ms): 96.99
|
||||
Max ITL (ms): 969.61
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.3.3 Multimodal (Image) Benchmark
|
||||
|
||||
- **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 25.73
|
||||
Total input tokens: 661
|
||||
Total input text tokens: 631
|
||||
Total input vision tokens: 30
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 0
|
||||
Request throughput (req/s): 0.39
|
||||
Input token throughput (tok/s): 25.69
|
||||
Output token throughput (tok/s): 164.03
|
||||
Peak output token throughput (tok/s): 1.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 189.73
|
||||
Concurrency: 1.00
|
||||
Accept length: 2.94
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2570.74
|
||||
Median E2E Latency (ms): 2411.92
|
||||
P90 E2E Latency (ms): 3711.62
|
||||
P99 E2E Latency (ms): 4949.74
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 0.00
|
||||
Median TTFT (ms): 0.00
|
||||
P99 TTFT (ms): 0.00
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.31
|
||||
Median TPOT (ms): 6.17
|
||||
P99 TPOT (ms): 17.18
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 0.00
|
||||
Median ITL (ms): 0.00
|
||||
P95 ITL (ms): 0.00
|
||||
P99 ITL (ms): 0.00
|
||||
Max ITL (ms): 0.00
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.4 Multi-Layer EAGLE Acceptance Profile — MiMo-V2.5-Pro
|
||||
|
||||
Pro's 3-layer MTP behaves very differently on natural text vs uniform-random token streams. The §5.2 benchmarks use `random 1024/1024`, which collapses accept-rate; this section measures the same server on GSM8K so the acceptance number is comparable to real workloads.
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8×)
|
||||
- Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
|
||||
- Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`)
|
||||
- Recipe: 3-layer EAGLE — `--enable-multi-layer-eagle --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` (top-1, max accept length 4)
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.run_eval \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--model XiaomiMiMo/MiMo-V2.5-Pro \
|
||||
--eval-name gsm8k \
|
||||
--num-examples 200 \
|
||||
--num-threads 4
|
||||
```
|
||||
|
||||
The `accept_rate` and `accept_length` rows below are not part of `run_eval`'s own output — they were aggregated from the server-side `Decode batch ... accept rate: X accept len: Y` log lines emitted during the GSM8K run (307 batches total).
|
||||
|
||||
| Workload | accept_rate | accept_length (max = 4) |
|
||||
| ------------------------------ | ----------- | ----------------------- |
|
||||
| GSM8K (natural text) | **0.755** | **3.27** |
|
||||
| `random 1024/1024` (reference) | 0.13–0.27 | ~1.x |
|
||||
|
||||
GSM8K Score: **0.97** (194 / 200), output throughput ≈ 635 tok/s end-to-end on this single-server run.
|
||||
|
||||
The accept-rate gap is intrinsic to MTP-style speculative decoding: the draft model is trained on natural-language token distributions and has no useful signal on uniform-random byte sequences. Workloads with structure (chat, code, reasoning traces) should expect the GSM8K-class number; the random-prompt baseline in §5.2 is a worst case for draft acceptance.
|
||||
|
||||
### 5.5 Long-Context Prefill & MTP Decode — MiMo-V2.5-Pro (Reference)
|
||||
|
||||
Reference numbers from the [day0 enablement PR](https://github.com/sgl-project/sglang/pull/23808), collected on a 2-node Hopper deployment with the **EP=16, DP=2, TP=16** recipe (`--moe-a2a-backend deepep`, `--attention-backend fa3`, `--enable-multi-layer-eagle`). The setup, parallelism, and benchmark methodology all differ from §5.2 (Blackwell TP=8 with `random 1024/1024`), so treat these as a separate operating point — long-context prefill scaling and the MTP decode speedup — rather than a comparison against §5.2.
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA Hopper GPU (2 nodes × 8 GPUs, GPU SKU intentionally not disclosed)
|
||||
- Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
|
||||
- Parallelism: `--tp 16 --dp 2 --ep 16 --moe-dense-tp-size 1 --enable-dp-attention`
|
||||
- Recipe: Hopper EP16 (DeepEP + EAGLE multi-layer MTP)
|
||||
|
||||
#### 5.5.1 Long-Context Prefill Throughput
|
||||
|
||||
Test setting: `chunked_prefill_size=32K`, `random_output_len=1`, cache flushed before every run. For input lengths ≥ 512K the workload was split into two requests routed to distinct DP ranks and the per-node throughput was read from `bench_serving` output.
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model XiaomiMiMo/MiMo-V2.5-Pro \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--dataset-name random \
|
||||
--random-input-len <INPUT_LEN> \
|
||||
--random-output-len 1 \
|
||||
--random-range-ratio 1.0 \
|
||||
--flush-cache \
|
||||
--seed 12345 \
|
||||
--num-prompts 10000
|
||||
```
|
||||
|
||||
- **Test Results** — single-node prefill throughput, cache-miss:
|
||||
|
||||
| Input length | Output length | Single-node prefill throughput |
|
||||
| ------------ | ------------- | ------------------------------ |
|
||||
| 4K | 1 | 30.80K tok/s |
|
||||
| 8K | 1 | 30.65K tok/s |
|
||||
| 16K | 1 | 29.85K tok/s |
|
||||
| 32K | 1 | 28.60K tok/s |
|
||||
| 64K | 1 | 26.65K tok/s |
|
||||
| 128K | 1 | 23.00K tok/s |
|
||||
| 256K | 1 | 17.90K tok/s |
|
||||
| 512K | 1 | 11.30K tok/s |
|
||||
| 768K | 1 | 9.40K tok/s |
|
||||
| 1M | 1 | 7.30K tok/s |
|
||||
|
||||
Prefill throughput stays within ~10% of peak from 4K up to 32K and degrades gracefully past 128K, confirming the hybrid SWA+GA attention works correctly at 1M context.
|
||||
|
||||
#### 5.5.2 Decode Throughput — MTP Speedup
|
||||
|
||||
Test setting: fixed **16K input / 1K output**, varying batch size per DP rank, with and without the 3-layer MTP module. `MTP accept length` is the average number of draft tokens accepted per step under EAGLE speculative decoding. **TPS** below is per-request output tokens/sec (i.e. single-user perceived speed); the rightmost column is aggregated single-node decode throughput (= TPS × batch size).
|
||||
|
||||
- **Test Results** — single-node decode throughput:
|
||||
|
||||
| BS per DP rank | MTP | MTP accept length | Per-request TPS | Single-node decode throughput |
|
||||
| -------------- | -------- | ----------------- | --------------- | ----------------------------- |
|
||||
| 64 | disabled | - | 29.3 | 1875 tok/s |
|
||||
| 64 | 3-layer | 3 | 60.5 | 3873 tok/s |
|
||||
| 64 | 3-layer | 4 | 79.7 | 5103 tok/s |
|
||||
| 96 | disabled | - | 26.7 | 2564 tok/s |
|
||||
| 96 | 3-layer | 3 | 50.4 | 4840 tok/s |
|
||||
| 96 | 3-layer | 4 | 64.8 | 6225 tok/s |
|
||||
|
||||
**Summary — MTP on / off:**
|
||||
|
||||
| BS per DP rank | Without MTP | 3-layer MTP, accept=3 | 3-layer MTP, accept=4 |
|
||||
| -------------- | ----------- | --------------------- | --------------------- |
|
||||
| 64 | 1875 tok/s | 3873 tok/s (2.07×) | 5103 tok/s (2.72×) |
|
||||
| 96 | 2564 tok/s | 4840 tok/s (1.89×) | 6225 tok/s (2.43×) |
|
||||
|
||||
The 3-layer MTP module delivers ~2× decode throughput at accept length 3 and ~2.5–2.7× at accept length 4 — the same order of magnitude as the "2–3× decode speedup" guidance in §3.2.
|
||||
@@ -0,0 +1,166 @@
|
||||
---
|
||||
title: Overview
|
||||
mode: wide
|
||||
description: Practical guides for deploying and using large language models and vision language models with SGLang.
|
||||
metatags:
|
||||
description: "Explore SGLang autoregressive model cookbooks for LLM and VLM deployment, invocation, optimization, and benchmarking examples."
|
||||
---
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card
|
||||
title="Kimi (Moonshot AI)"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Moonshotai/Kimi-K3"
|
||||
img="/cards/logos/moonshotai.png"
|
||||
/>
|
||||
<Card
|
||||
title="Thinking Machines"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/ThinkingMachines/Inkling"
|
||||
img="/cards/logos/thinkingmachines.png"
|
||||
/>
|
||||
<Card
|
||||
title="GLM"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/GLM/GLM-5.2"
|
||||
img="/cards/logos/glm.png"
|
||||
/>
|
||||
<Card
|
||||
title="Qwen"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Qwen/Qwen3.6"
|
||||
img="/cards/logos/qwen.png"
|
||||
/>
|
||||
<Card
|
||||
title="DeepSeek"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/DeepSeek/DeepSeek-V4"
|
||||
img="/cards/logos/deepseek.png"
|
||||
/>
|
||||
<Card
|
||||
title="DeepReinforce"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/DeepReinforce/Ornith-1.0"
|
||||
img="/cards/logos/deepreinforce.png"
|
||||
/>
|
||||
<Card
|
||||
title="Llama"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Llama/Llama3.3-70B"
|
||||
img="/cards/logos/llama.png"
|
||||
/>
|
||||
<Card
|
||||
title="Meituan"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Meituan/LongCat-2.0"
|
||||
img="/cards/logos/meituan.png"
|
||||
/>
|
||||
<Card
|
||||
title="Google"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Google/Gemma4"
|
||||
img="/cards/logos/google.png"
|
||||
/>
|
||||
<Card
|
||||
title="LiquidAI"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/LiquidAI/LFM2.5"
|
||||
img="/cards/logos/liquidai.png"
|
||||
/>
|
||||
<Card
|
||||
title="OpenAI"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/OpenAI/GPT-OSS"
|
||||
img="/cards/logos/openai.png"
|
||||
/>
|
||||
<Card
|
||||
title="MiniMax"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/MiniMax/MiniMax-M3"
|
||||
img="/cards/logos/minimax.png"
|
||||
/>
|
||||
<Card
|
||||
title="NVIDIA"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra"
|
||||
img="/cards/logos/nvidia.png"
|
||||
/>
|
||||
<Card
|
||||
title="Baidu"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Baidu/Unlimited-OCR"
|
||||
img="/cards/logos/baidu.svg"
|
||||
/>
|
||||
<Card
|
||||
title="Ernie"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Ernie/Ernie4.5"
|
||||
img="/cards/logos/ernie.png"
|
||||
/>
|
||||
<Card
|
||||
title="StepFun"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/StepFun/Step3.5"
|
||||
img="/cards/logos/stepfun.png"
|
||||
/>
|
||||
<Card
|
||||
title="InclusionAI"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/InclusionAI/Ling-2.5-1T"
|
||||
img="/cards/logos/inclusionai.png"
|
||||
/>
|
||||
<Card
|
||||
title="InternLM"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/InternLM/Intern-S2-Preview"
|
||||
img="/cards/logos/internlm.png"
|
||||
/>
|
||||
<Card
|
||||
title="InternVL"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/InternVL/InternVL3.5"
|
||||
img="/cards/logos/internvl.png"
|
||||
/>
|
||||
<Card
|
||||
title="OpenBMB"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6"
|
||||
img="/cards/logos/openbmb.png"
|
||||
/>
|
||||
<Card
|
||||
title="Jina AI"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Jina/Jina-reranker-m0"
|
||||
img="/cards/logos/jina.png"
|
||||
/>
|
||||
<Card
|
||||
title="Mistral"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Mistral/Ministral-3"
|
||||
img="/cards/logos/mistral.png"
|
||||
/>
|
||||
<Card
|
||||
title="Xiaomi"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Xiaomi/MiMo-V2.5"
|
||||
img="/cards/logos/xiaomi.png"
|
||||
/>
|
||||
<Card
|
||||
title="FlashLabs"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/FlashLabs/Chroma1.0"
|
||||
img="/cards/logos/flashlabs.png"
|
||||
/>
|
||||
<Card
|
||||
title="Tencent"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Tencent/Hy3"
|
||||
img="/cards/logos/tencent.png"
|
||||
/>
|
||||
<Card
|
||||
title="Poolside"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Poolside/Laguna-S-2.1"
|
||||
img="/cards/logos/poolside.png"
|
||||
/>
|
||||
</CardGroup>
|
||||
Reference in New Issue
Block a user