[Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c949e91f18
commit
b819d2fb5b
@@ -0,0 +1,516 @@
|
||||
---
|
||||
title: GLM-4.5
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5 with SGLang on AMD GPUs - advanced reasoning, function calling, BF16/FP8 quantization options."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM45Deployment } from "/src/snippets/autoregressive/glm-45-deployment.jsx";
|
||||
|
||||
<GLM45Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.5 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
|
||||
- Model: GLM-4.5
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
@@ -0,0 +1,582 @@
|
||||
---
|
||||
title: GLM-4.5V
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5V vision-language model with SGLang - SOTA multimodal performance, 64K context, image reasoning and video understanding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
|
||||
|
||||
GLM-4.5V introduces several key features:
|
||||
|
||||
- **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection.
|
||||
- **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences.
|
||||
- **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces.
|
||||
- **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context.
|
||||
- **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
|
||||
|
||||
import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deployment.jsx";
|
||||
|
||||
<GLM45VDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`.
|
||||
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
|
||||
- **Hardware Notes:**
|
||||
- **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
- **Additional Multimodal Parameters:**
|
||||
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
GLM-4.5V supports both image and video inputs. Here's a basic example with image input:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image in detail."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.37s
|
||||
Generated text: Auntie Anne's
|
||||
|
||||
CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
GLM-4.5V can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.86s
|
||||
Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot).
|
||||
```
|
||||
|
||||
**Video Input Example:**
|
||||
|
||||
GLM-4.5V supports video understanding by processing video URLs:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe what happens in this video."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- For video processing, ensure you have sufficient context length configured (up to 64K tokens)
|
||||
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
|
||||
- You can also provide local file paths using `file://` protocol
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.89s
|
||||
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
|
||||
```
|
||||
|
||||
#### 4.2.2 Thinking Mode
|
||||
|
||||
GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
**Disable Thinking Mode:**
|
||||
|
||||
To disable thinking mode for a specific request:
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[{"role": "user", "content": "What is the capital of France?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
GLM-4.5V supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Beyond enabling/disabling the full reasoning mode (section 4.2.2), you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[{"role": "user", "content": "Describe this image briefly."}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.1.1 MMMU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```text Output
|
||||
Benchmark time: 616.6163094160147
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.867, 'num': 30},
|
||||
'Agriculture': {'acc': 0.567, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.667, 'num': 30},
|
||||
'Art': {'acc': 0.667, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.8, 'num': 30},
|
||||
'Biology': {'acc': 0.6, 'num': 30},
|
||||
'Chemistry': {'acc': 0.533, 'num': 30},
|
||||
'Clinical_Medicine': {'acc': 0.667, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.8, 'num': 30},
|
||||
'Design': {'acc': 0.867, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30},
|
||||
'Economics': {'acc': 0.833, 'num': 30},
|
||||
'Electronics': {'acc': 0.433, 'num': 30},
|
||||
'Energy_and_Power': {'acc': 0.733, 'num': 30},
|
||||
'Finance': {'acc': 0.767, 'num': 30},
|
||||
'Geography': {'acc': 0.667, 'num': 30},
|
||||
'History': {'acc': 0.8, 'num': 30},
|
||||
'Literature': {'acc': 0.9, 'num': 30},
|
||||
'Manage': {'acc': 0.733, 'num': 30},
|
||||
'Marketing': {'acc': 0.9, 'num': 30},
|
||||
'Materials': {'acc': 0.567, 'num': 30},
|
||||
'Math': {'acc': 0.8, 'num': 30},
|
||||
'Mechanical_Engineering': {'acc': 0.767, 'num': 30},
|
||||
'Music': {'acc': 0.3, 'num': 30},
|
||||
'Overall': {'acc': 0.732, 'num': 900},
|
||||
'Overall-Art and Design': {'acc': 0.683, 'num': 120},
|
||||
'Overall-Business': {'acc': 0.82, 'num': 150},
|
||||
'Overall-Health and Medicine': {'acc': 0.787, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120},
|
||||
'Overall-Science': {'acc': 0.707, 'num': 150},
|
||||
'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210},
|
||||
'Pharmacy': {'acc': 0.9, 'num': 30},
|
||||
'Physics': {'acc': 0.933, 'num': 30},
|
||||
'Psychology': {'acc': 0.767, 'num': 30},
|
||||
'Public_Health': {'acc': 0.9, 'num': 30},
|
||||
'Sociology': {'acc': 0.667, 'num': 30}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.732
|
||||
```
|
||||
@@ -0,0 +1,914 @@
|
||||
---
|
||||
title: GLM-4.6
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6 with SGLang - 200K context window, superior coding, advanced reasoning, and enhanced agentic capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows:
|
||||
|
||||
- **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks.
|
||||
- **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages.
|
||||
- **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability.
|
||||
- **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks.
|
||||
- **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios.
|
||||
|
||||
For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM46Deployment } from "/src/snippets/autoregressive/glm-46-deployment.jsx";
|
||||
|
||||
<GLM46Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.6 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
|
||||
- Model: GLM-4.6
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 63.82
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4209
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 95.60
|
||||
Output token throughput (tok/s): 65.97
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 161.57
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6379.24
|
||||
Median E2E Latency (ms): 5085.00
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 155.57
|
||||
Median TTFT (ms): 149.79
|
||||
P99 TTFT (ms): 207.69
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.81
|
||||
Median TPOT (ms): 14.80
|
||||
P99 TPOT (ms): 14.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.82
|
||||
Median ITL (ms): 14.82
|
||||
P95 ITL (ms): 15.17
|
||||
P99 ITL (ms): 15.36
|
||||
Max ITL (ms): 25.05
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 72.06
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40725
|
||||
Total generated tokens (retokenized): 40672
|
||||
Request throughput (req/s): 1.11
|
||||
Input token throughput (tok/s): 550.47
|
||||
Output token throughput (tok/s): 565.14
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1115.61
|
||||
Concurrency: 13.71
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 12348.93
|
||||
Median E2E Latency (ms): 13164.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 196.08
|
||||
Median TTFT (ms): 155.22
|
||||
P99 TTFT (ms): 377.98
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.24
|
||||
Median TPOT (ms): 24.55
|
||||
P99 TPOT (ms): 30.42
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.92
|
||||
Median ITL (ms): 21.40
|
||||
P95 ITL (ms): 22.49
|
||||
P99 ITL (ms): 123.83
|
||||
Max ITL (ms): 486.54
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 138.50
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252162
|
||||
Total generated tokens (retokenized): 251841
|
||||
Request throughput (req/s): 3.61
|
||||
Input token throughput (tok/s): 1803.78
|
||||
Output token throughput (tok/s): 1820.61
|
||||
Peak output token throughput (tok/s): 2900.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3624.40
|
||||
Concurrency: 90.91
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 25183.97
|
||||
Median E2E Latency (ms): 23968.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 337.77
|
||||
Median TTFT (ms): 180.65
|
||||
P99 TTFT (ms): 906.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 49.97
|
||||
Median TPOT (ms): 52.20
|
||||
P99 TPOT (ms): 61.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 49.36
|
||||
Median ITL (ms): 35.05
|
||||
P95 ITL (ms): 124.91
|
||||
P99 ITL (ms): 187.69
|
||||
Max ITL (ms): 440.34
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 666.64
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44452
|
||||
Total generated tokens (retokenized): 44387
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 9.15
|
||||
Output token throughput (tok/s): 66.68
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 75.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 66661.35
|
||||
Median E2E Latency (ms): 71902.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 160.21
|
||||
Median TTFT (ms): 140.32
|
||||
P99 TTFT (ms): 295.56
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.92
|
||||
Median TPOT (ms): 14.94
|
||||
P99 TPOT (ms): 15.02
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.96
|
||||
Median ITL (ms): 14.96
|
||||
P95 ITL (ms): 15.36
|
||||
P99 ITL (ms): 15.57
|
||||
Max ITL (ms): 19.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 503.30
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318226
|
||||
Total generated tokens (retokenized): 318025
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 78.82
|
||||
Output token throughput (tok/s): 632.28
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 711.09
|
||||
Concurrency: 13.88
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 87349.22
|
||||
Median E2E Latency (ms): 88248.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 228.54
|
||||
Median TTFT (ms): 142.78
|
||||
P99 TTFT (ms): 569.84
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 21.97
|
||||
Median TPOT (ms): 22.14
|
||||
P99 TPOT (ms): 22.47
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 21.91
|
||||
Median ITL (ms): 21.80
|
||||
P95 ITL (ms): 22.30
|
||||
P99 ITL (ms): 22.78
|
||||
Max ITL (ms): 137.19
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 772.28
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1300705
|
||||
Total generated tokens (retokenized): 1299924
|
||||
Request throughput (req/s): 0.41
|
||||
Input token throughput (tok/s): 205.80
|
||||
Output token throughput (tok/s): 1684.24
|
||||
Peak output token throughput (tok/s): 2112.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 1890.05
|
||||
Concurrency: 56.17
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 135563.36
|
||||
Median E2E Latency (ms): 140888.88
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 232.45
|
||||
Median TTFT (ms): 145.59
|
||||
P99 TTFT (ms): 576.49
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.47
|
||||
Median TPOT (ms): 34.02
|
||||
P99 TPOT (ms): 35.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.30
|
||||
Median ITL (ms): 32.63
|
||||
P95 ITL (ms): 34.27
|
||||
P99 ITL (ms): 104.39
|
||||
Max ITL (ms): 155.65
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 65.11
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4210
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 644.17
|
||||
Output token throughput (tok/s): 64.66
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 708.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6508.31
|
||||
Median E2E Latency (ms): 5263.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 189.48
|
||||
Median TTFT (ms): 159.23
|
||||
P99 TTFT (ms): 304.09
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 15.02
|
||||
Median TPOT (ms): 15.03
|
||||
P99 TPOT (ms): 15.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.04
|
||||
Median ITL (ms): 15.03
|
||||
P95 ITL (ms): 15.46
|
||||
P99 ITL (ms): 15.65
|
||||
Max ITL (ms): 24.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 76.43
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41589
|
||||
Total generated tokens (retokenized): 41577
|
||||
Request throughput (req/s): 1.05
|
||||
Input token throughput (tok/s): 3925.47
|
||||
Output token throughput (tok/s): 544.15
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 4469.62
|
||||
Concurrency: 13.95
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13329.63
|
||||
Median E2E Latency (ms): 14141.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 339.88
|
||||
Median TTFT (ms): 252.75
|
||||
P99 TTFT (ms): 906.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.37
|
||||
Median TPOT (ms): 25.73
|
||||
P99 TPOT (ms): 30.94
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.04
|
||||
Median ITL (ms): 21.68
|
||||
P95 ITL (ms): 22.69
|
||||
P99 ITL (ms): 146.98
|
||||
Max ITL (ms): 483.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 136.24
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 169680
|
||||
Total generated tokens (retokenized): 169452
|
||||
Request throughput (req/s): 2.35
|
||||
Input token throughput (tok/s): 9350.32
|
||||
Output token throughput (tok/s): 1245.44
|
||||
Peak output token throughput (tok/s): 1984.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 10595.77
|
||||
Concurrency: 58.46
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 24889.40
|
||||
Median E2E Latency (ms): 25123.37
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 355.82
|
||||
Median TTFT (ms): 268.84
|
||||
P99 TTFT (ms): 858.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 46.62
|
||||
Median TPOT (ms): 49.04
|
||||
P99 TPOT (ms): 58.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 46.36
|
||||
Median ITL (ms): 32.46
|
||||
P95 ITL (ms): 135.23
|
||||
P99 ITL (ms): 204.27
|
||||
Max ITL (ms): 508.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.975
|
||||
Invalid: 0.000
|
||||
Latency: 16.574 s
|
||||
Output throughput: 1194.637 token/s
|
||||
```
|
||||
@@ -0,0 +1,512 @@
|
||||
---
|
||||
title: GLM-4.6V
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6V vision-language model with SGLang - native function calling, 128K context, multimodal document understanding and frontend replication."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios.
|
||||
|
||||
Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features:
|
||||
|
||||
- **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#4-2-3-tool-calling).
|
||||
- **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content.
|
||||
- **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text.
|
||||
- **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
### 2.1 Docker Installation (Recommended)
|
||||
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Ready to use out of the box, no manual environment configuration needed
|
||||
- Avoids dependency conflict issues
|
||||
- Easy to migrate between different environments
|
||||
|
||||
### 2.2 Build from Source
|
||||
|
||||
If you need to use the latest development version or require custom modifications, you can build from source:
|
||||
|
||||
```bash Command
|
||||
# Install SGLang using UV (recommended)
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -e "python[all]" --index-url=https://pypi.org/simple
|
||||
pip install nvidia-cudnn-cu12==9.16.0.29
|
||||
# Install ffmpeg to support video input
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
**Use Cases:**
|
||||
|
||||
- Need to customize and modify SGLang source code
|
||||
- Want to use the latest development features
|
||||
- Participate in SGLang project development
|
||||
|
||||
For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
|
||||
|
||||
import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deployment.jsx";
|
||||
|
||||
<GLM46VDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically).
|
||||
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
|
||||
- **Hardware Notes:**
|
||||
- **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
- **Additional Multimodal Parameters:**
|
||||
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
GLM-4.6V supports image and video inputs via the OpenAI-compatible API.
|
||||
|
||||
**Image Input:**
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{30000}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "default",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "image_url",
|
||||
"image_url": {{
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}}
|
||||
}},
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What is the image"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"temperature": "0",
|
||||
"max_completion_tokens": "1000",
|
||||
"max_tokens": "1000"
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print(response)
|
||||
```
|
||||
|
||||
```text Output
|
||||
{"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"</>\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"</>\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
|
||||
```
|
||||
|
||||
**Video Input:**
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{30000}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "default",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "video_url",
|
||||
"video_url": {{
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
}}
|
||||
}},
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What is in the video"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"temperature": "0",
|
||||
"max_completion_tokens": "1000",
|
||||
"max_tokens": "1000"
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print(response)
|
||||
```
|
||||
|
||||
```text Output
|
||||
{"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n* **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n* **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n* **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n* **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
|
||||
```
|
||||
|
||||
#### 4.2.2 Thinking Mode
|
||||
|
||||
GLM-4.6V supports Thinking mode. Enable the reasoning parser during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6V \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
GLM-4.6V supports tool calling with vision capabilities. Pass tools in your API request:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
openai_api_key = "EMPTY"
|
||||
openai_api_base = "http://127.0.0.1:30000/v1"
|
||||
client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)
|
||||
|
||||
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current temperature for a given location.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and country e.g. Beijing, China",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Please help me check today's weather in Beijing, and tell me whether the tool returned an image."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_bk32t88BGpSdbtDgzT044Rh4",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": 'get_weather',
|
||||
"arguments": '{"location":"Beijing, China"}'
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C."
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=messages,
|
||||
timeout=900,
|
||||
tools=tools
|
||||
)
|
||||
print(response.choices[0].message.content.strip())
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C.
|
||||
|
||||
Yes, the tool returned an image (the SGL logo).
|
||||
```
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Beyond the reasoning parser, you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request — same as the [GLM-4.6 text model approach](./GLM-4.6#4-2-3-thinking-budget):
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=[{"role": "user", "content": "Describe this image briefly."}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1. Text Benchmark: Latency, Throughput and Accuracy
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 ./benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
#### Result Output
|
||||
```text Output
|
||||
Accuracy: 0.925
|
||||
Invalid: 0.000
|
||||
Latency: 15.327 s
|
||||
Output throughput: 1788.375 token/s
|
||||
```
|
||||
|
||||
### 5.2. Multimodal Benchmark - Latency and Throughput
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--port 30000 \
|
||||
--model zai-org/GLM-4.6V \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 128 \
|
||||
--max-concurrency 8
|
||||
```
|
||||
|
||||
#### Result Output
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 8
|
||||
Successful requests: 128
|
||||
Benchmark duration (s): 89.27
|
||||
Total input tokens: 315390
|
||||
Total input text tokens: 8702
|
||||
Total input vision tokens: 306688
|
||||
Total generated tokens: 66020
|
||||
Total generated tokens (retokenized): 31037
|
||||
Request throughput (req/s): 1.43
|
||||
Input token throughput (tok/s): 3533.17
|
||||
Output token throughput (tok/s): 739.59
|
||||
Peak output token throughput (tok/s): 823.00
|
||||
Peak concurrent requests: 12
|
||||
Total token throughput (tok/s): 4272.76
|
||||
Concurrency: 7.67
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5349.20
|
||||
Median E2E Latency (ms): 5380.98
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1724.04
|
||||
Median TTFT (ms): 1688.16
|
||||
P99 TTFT (ms): 6152.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.15
|
||||
Median TPOT (ms): 7.77
|
||||
P99 TPOT (ms): 23.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.00
|
||||
Median ITL (ms): 8.44
|
||||
P95 ITL (ms): 9.23
|
||||
P99 ITL (ms): 116.02
|
||||
Max ITL (ms): 173.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
### 5.3. Multimodal Accuracy Benchmark - MMMU
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}'
|
||||
```
|
||||
|
||||
#### Result Output
|
||||
```text Output
|
||||
Benchmark time: 487.2229107860476
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.962, 'num': 26},
|
||||
'Agriculture': {'acc': 0.5, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.733, 'num': 15},
|
||||
'Art': {'acc': 0.833, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.733, 'num': 30},
|
||||
'Biology': {'acc': 0.586, 'num': 29},
|
||||
'Chemistry': {'acc': 0.654, 'num': 26},
|
||||
'Clinical_Medicine': {'acc': 0.633, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.76, 'num': 25},
|
||||
'Design': {'acc': 0.867, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30},
|
||||
'Economics': {'acc': 0.862, 'num': 29},
|
||||
'Electronics': {'acc': 0.5, 'num': 18},
|
||||
'Energy_and_Power': {'acc': 0.875, 'num': 16},
|
||||
'Finance': {'acc': 0.857, 'num': 28},
|
||||
'Geography': {'acc': 0.714, 'num': 28},
|
||||
'History': {'acc': 0.767, 'num': 30},
|
||||
'Literature': {'acc': 0.897, 'num': 29},
|
||||
'Manage': {'acc': 0.759, 'num': 29},
|
||||
'Marketing': {'acc': 1.0, 'num': 26},
|
||||
'Materials': {'acc': 0.833, 'num': 18},
|
||||
'Math': {'acc': 0.76, 'num': 25},
|
||||
'Mechanical_Engineering': {'acc': 0.619, 'num': 21},
|
||||
'Music': {'acc': 0.286, 'num': 28},
|
||||
'Overall': {'acc': 0.761, 'num': 803},
|
||||
'Overall-Art and Design': {'acc': 0.729, 'num': 118},
|
||||
'Overall-Business': {'acc': 0.884, 'num': 138},
|
||||
'Overall-Health and Medicine': {'acc': 0.773, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118},
|
||||
'Overall-Science': {'acc': 0.728, 'num': 136},
|
||||
'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143},
|
||||
'Pharmacy': {'acc': 0.933, 'num': 30},
|
||||
'Physics': {'acc': 0.929, 'num': 28},
|
||||
'Psychology': {'acc': 0.733, 'num': 30},
|
||||
'Public_Health': {'acc': 0.933, 'num': 30},
|
||||
'Sociology': {'acc': 0.724, 'num': 29}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.761
|
||||
```
|
||||
@@ -0,0 +1,935 @@
|
||||
---
|
||||
title: GLM-4.7-Flash
|
||||
metatags:
|
||||
description: "Deploy GLM-4.7-Flash 30B-A3B MoE model with SGLang - lightweight, efficient inference optimized for single-GPU deployment."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment.
|
||||
|
||||
As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency:
|
||||
|
||||
- **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference
|
||||
- **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance
|
||||
- **Superior Coding**: Strong code generation and understanding capabilities
|
||||
- **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows
|
||||
- **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios
|
||||
|
||||
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM47FlashDeployment } from "/src/snippets/autoregressive/glm-47-flash-deployment.jsx";
|
||||
|
||||
<GLM47FlashDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.7-Flash. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--reasoning-parser glm45 \
|
||||
--attention-backend triton \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47 \
|
||||
--attention-backend triton \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls (tool call deltas may stream in multiple chunks)
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
if tool_calls_accumulator:
|
||||
print("\n=============== Tool Calls =================", flush=True)
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the
|
||||
user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you.
|
||||
=============== Tool Calls =================
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 (1x)
|
||||
- Model: GLM-4.7-Flash
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--attention-backend triton \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 38.94
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.26
|
||||
Input token throughput (tok/s): 156.67
|
||||
Output token throughput (tok/s): 108.37
|
||||
Peak output token throughput (tok/s): 125.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 265.03
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3891.12
|
||||
Median E2E Latency (ms): 3061.48
|
||||
P90 E2E Latency (ms): 7172.25
|
||||
P99 E2E Latency (ms): 9042.62
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 131.36
|
||||
Median TTFT (ms): 94.55
|
||||
P99 TTFT (ms): 435.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.75
|
||||
Median TPOT (ms): 8.82
|
||||
P99 TPOT (ms): 9.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.93
|
||||
Median ITL (ms): 8.98
|
||||
P95 ITL (ms): 9.83
|
||||
P99 ITL (ms): 10.20
|
||||
Max ITL (ms): 18.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 52.73
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40775
|
||||
Request throughput (req/s): 1.52
|
||||
Input token throughput (tok/s): 752.27
|
||||
Output token throughput (tok/s): 773.83
|
||||
Peak output token throughput (tok/s): 1040.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1526.10
|
||||
Concurrency: 13.98
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9217.90
|
||||
Median E2E Latency (ms): 9642.50
|
||||
P90 E2E Latency (ms): 15147.02
|
||||
P99 E2E Latency (ms): 18237.06
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 299.02
|
||||
Median TTFT (ms): 105.98
|
||||
P99 TTFT (ms): 1109.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.03
|
||||
Median TPOT (ms): 18.00
|
||||
P99 TPOT (ms): 26.51
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 17.52
|
||||
Median ITL (ms): 16.07
|
||||
P95 ITL (ms): 18.14
|
||||
P99 ITL (ms): 89.43
|
||||
Max ITL (ms): 763.13
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 91.48
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250941
|
||||
Request throughput (req/s): 5.47
|
||||
Input token throughput (tok/s): 2730.87
|
||||
Output token throughput (tok/s): 2761.82
|
||||
Peak output token throughput (tok/s): 4199.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 5492.69
|
||||
Concurrency: 90.54
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16566.04
|
||||
Median E2E Latency (ms): 16134.36
|
||||
P90 E2E Latency (ms): 30167.60
|
||||
P99 E2E Latency (ms): 34034.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 433.94
|
||||
Median TTFT (ms): 123.26
|
||||
P99 TTFT (ms): 1760.09
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.26
|
||||
Median TPOT (ms): 33.56
|
||||
P99 TPOT (ms): 38.78
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 31.99
|
||||
Median ITL (ms): 24.06
|
||||
P95 ITL (ms): 79.62
|
||||
P99 ITL (ms): 103.03
|
||||
Max ITL (ms): 1369.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 525.43
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44451
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 11.61
|
||||
Output token throughput (tok/s): 84.62
|
||||
Peak output token throughput (tok/s): 125.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 96.23
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 52540.19
|
||||
Median E2E Latency (ms): 53694.45
|
||||
P90 E2E Latency (ms): 94742.08
|
||||
P99 E2E Latency (ms): 101224.18
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 97.45
|
||||
Median TTFT (ms): 95.28
|
||||
P99 TTFT (ms): 105.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.94
|
||||
Median TPOT (ms): 11.25
|
||||
P99 TPOT (ms): 13.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.80
|
||||
Median ITL (ms): 11.51
|
||||
P95 ITL (ms): 15.83
|
||||
P99 ITL (ms): 16.86
|
||||
Max ITL (ms): 19.96
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 473.92
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317860
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 83.70
|
||||
Output token throughput (tok/s): 671.65
|
||||
Peak output token throughput (tok/s): 1040.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 755.35
|
||||
Concurrency: 13.80
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 81746.73
|
||||
Median E2E Latency (ms): 78508.54
|
||||
P90 E2E Latency (ms): 155292.49
|
||||
P99 E2E Latency (ms): 166769.99
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 117.50
|
||||
Median TTFT (ms): 101.97
|
||||
P99 TTFT (ms): 182.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.36
|
||||
Median TPOT (ms): 20.48
|
||||
P99 TPOT (ms): 22.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.52
|
||||
Median ITL (ms): 20.42
|
||||
P95 ITL (ms): 23.41
|
||||
P99 ITL (ms): 26.29
|
||||
Max ITL (ms): 90.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 714.72
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1289431
|
||||
Request throughput (req/s): 0.45
|
||||
Input token throughput (tok/s): 222.38
|
||||
Output token throughput (tok/s): 1820.33
|
||||
Peak output token throughput (tok/s): 3200.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 2042.71
|
||||
Concurrency: 55.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 124364.58
|
||||
Median E2E Latency (ms): 129250.98
|
||||
P90 E2E Latency (ms): 219175.80
|
||||
P99 E2E Latency (ms): 247741.77
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 149.40
|
||||
Median TTFT (ms): 114.78
|
||||
P99 TTFT (ms): 288.60
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 30.51
|
||||
Median TPOT (ms): 31.75
|
||||
P99 TPOT (ms): 33.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 30.56
|
||||
Median ITL (ms): 30.82
|
||||
P95 ITL (ms): 33.20
|
||||
P99 ITL (ms): 80.54
|
||||
Max ITL (ms): 117.72
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 58.27
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 719.73
|
||||
Output token throughput (tok/s): 72.42
|
||||
Peak output token throughput (tok/s): 112.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 792.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5825.08
|
||||
Median E2E Latency (ms): 4624.26
|
||||
P90 E2E Latency (ms): 12690.22
|
||||
P99 E2E Latency (ms): 13177.96
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 296.01
|
||||
Median TTFT (ms): 195.59
|
||||
P99 TTFT (ms): 717.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.63
|
||||
Median TPOT (ms): 13.07
|
||||
P99 TPOT (ms): 16.68
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.13
|
||||
Median ITL (ms): 13.17
|
||||
P95 ITL (ms): 17.02
|
||||
P99 ITL (ms): 17.47
|
||||
Max ITL (ms): 19.84
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 89.59
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41656
|
||||
Request throughput (req/s): 0.89
|
||||
Input token throughput (tok/s): 3348.77
|
||||
Output token throughput (tok/s): 465.10
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3813.87
|
||||
Concurrency: 14.39
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16120.74
|
||||
Median E2E Latency (ms): 16246.55
|
||||
P90 E2E Latency (ms): 27279.72
|
||||
P99 E2E Latency (ms): 34577.93
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1943.94
|
||||
Median TTFT (ms): 382.19
|
||||
P99 TTFT (ms): 8980.41
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 27.87
|
||||
Median TPOT (ms): 28.26
|
||||
P99 TPOT (ms): 40.55
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 27.27
|
||||
Median ITL (ms): 21.74
|
||||
P95 ITL (ms): 23.32
|
||||
P99 ITL (ms): 232.65
|
||||
Max ITL (ms): 4282.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 167.01
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169226
|
||||
Request throughput (req/s): 1.92
|
||||
Input token throughput (tok/s): 7627.82
|
||||
Output token throughput (tok/s): 1017.93
|
||||
Peak output token throughput (tok/s): 1984.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 8645.75
|
||||
Concurrency: 59.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 31147.52
|
||||
Median E2E Latency (ms): 30603.34
|
||||
P90 E2E Latency (ms): 54889.44
|
||||
P99 E2E Latency (ms): 67665.30
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 428.87
|
||||
Median TTFT (ms): 441.69
|
||||
P99 TTFT (ms): 1232.68
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 58.06
|
||||
Median TPOT (ms): 62.79
|
||||
P99 TPOT (ms): 82.23
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 57.93
|
||||
Median ITL (ms): 33.30
|
||||
P95 ITL (ms): 247.98
|
||||
P99 ITL (ms): 409.63
|
||||
Max ITL (ms): 1421.21
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Result
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.845
|
||||
Invalid: 0.000
|
||||
Latency: 8.431 s
|
||||
Output throughput: 2195.387 token/s
|
||||
```
|
||||
@@ -0,0 +1,962 @@
|
||||
---
|
||||
title: GLM-4.7
|
||||
metatags:
|
||||
description: "Deploy GLM-4.7 with SGLang on NVIDIA Blackwell (B200, GB200) and AMD GPUs - state-of-the-art reasoning, robust tool calling, and NVFP4 weights for Blackwell."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and agent workflows.
|
||||
|
||||
GLM-4.7 brings improvements across all major domains:
|
||||
|
||||
- **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations
|
||||
- **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing
|
||||
- **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance
|
||||
- **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows
|
||||
- **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms
|
||||
|
||||
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks
|
||||
- **Multiple Quantizations**: BF16, FP8, and NVFP4 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Tuned for NVIDIA Blackwell (B200, GB200) and AMD MI300X/MI325X/MI355X GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8)
|
||||
- **NVFP4 (4-bit, NVIDIA Blackwell)**: [nvidia/GLM-4.7-NVFP4](https://huggingface.co/nvidia/GLM-4.7-NVFP4)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA H100 / H200 / B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA GB200 / B300 / GB300 (aarch64)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-cu130`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-rocm720-mi30x`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-rocm720-mi35x`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM47Deployment } from "/src/snippets/autoregressive/glm-47-deployment.jsx";
|
||||
|
||||
<GLM47Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Pick a weight format by hardware: **NVFP4** on NVIDIA Blackwell (B200, GB200), **FP8** on H100/H200/AMD, **BF16** as the full-precision fallback. The recommended tensor-parallel size per platform:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>NVFP4</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200 (8×, single node)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=2 / 4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB200 (NVL72, 4× per tray)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=2 / 4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200 (8×)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X / MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=2 / 4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4 / 8</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.7. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
For general GLM-4.x family launch guidance (AMD ROCm notes and more), see [Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang](/cookbook/autoregressive/GLM/GLM-4.5). Per-hardware bench commands and flags are inline in §5.1 below.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.7 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200, NVIDIA GB200, AMD MI300X/MI325X/MI355X (8x)
|
||||
- Model: GLM-4.7-NVFP4 on NVIDIA Blackwell; GLM-4.7-FP8 or GLM-4.7 (BF16) on AMD
|
||||
- SGLang Version: 0.5.12 (NVIDIA Blackwell), 0.5.6.post1 (AMD)
|
||||
- Best per-GPU throughput config on B200: **TP=2 NVFP4 bf16-KV** (NVFP4 weights, no EP). Numbers below come from this config.
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Four core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Throughput**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mixed RAG / agent / multi-turn conversation (used for the inline B200 / GB200 results below)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) — the Throughput (4K/1K) scenario uses `--max-concurrency 128` to match the inline B200/GB200 results below.
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 4: Throughput (4K/1K) — NVIDIA Blackwell with NVFP4**
|
||||
|
||||
The remaining sub-sections (§5.1.4.1 NVIDIA B200, §5.1.4.2 NVIDIA GB200) measure this scenario with `nvidia/GLM-4.7-NVFP4` weights and report the full `bench_serving` output verbatim. The same commands apply to other NVIDIA hardware after substituting the deployment line from §3.1.
|
||||
|
||||
> **Note**: These runs use EOS-enabled generation (no `--disable-ignore-eos`), so generated-token counts reflect natural model behavior rather than a strict fixed-OSL pin. Compare against other EOS-enabled runs at the same workload, not against fixed-output-length benchmarks.
|
||||
|
||||
#### 5.1.4.1 NVIDIA B200
|
||||
|
||||
**Model Deployment (NVIDIA B200, TP=2 NVFP4 — max tok/s/gpu config):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 5 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 1
|
||||
Successful requests: 5
|
||||
Benchmark duration (s): 25.07
|
||||
Total input tokens: 8105
|
||||
Total generated tokens: 2674
|
||||
Request throughput (req/s): 0.20
|
||||
Input token throughput (tok/s): 323.25
|
||||
Output token throughput (tok/s): 106.65
|
||||
Total token throughput (tok/s): 429.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5011.93
|
||||
Median E2E Latency (ms): 6441.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 179.61
|
||||
Median TTFT (ms): 169.05
|
||||
P99 TTFT (ms): 238.01
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.05
|
||||
Median TPOT (ms): 9.03
|
||||
P99 TPOT (ms): 9.16
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.05
|
||||
Median ITL (ms): 9.05
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.60
|
||||
Total input tokens: 179772
|
||||
Total generated tokens: 39657
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 2966.39
|
||||
Output token throughput (tok/s): 654.37
|
||||
Total token throughput (tok/s): 3620.76
|
||||
Concurrency: 14.01
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10615.87
|
||||
Median E2E Latency (ms): 9985.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 267.39
|
||||
Median TTFT (ms): 177.26
|
||||
P99 TTFT (ms): 584.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.98
|
||||
Median TPOT (ms): 21.06
|
||||
P99 TPOT (ms): 24.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.92
|
||||
Median ITL (ms): 17.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 640 \
|
||||
--max-concurrency 128 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 128
|
||||
Successful requests: 640
|
||||
Benchmark duration (s): 172.95
|
||||
Total input tokens: 1453591
|
||||
Total generated tokens: 308740
|
||||
Request throughput (req/s): 3.70
|
||||
Input token throughput (tok/s): 8404.67
|
||||
Output token throughput (tok/s): 1785.14
|
||||
Total token throughput (tok/s): 10189.80
|
||||
Concurrency: 117.85
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 31848.20
|
||||
Median E2E Latency (ms): 28554.42
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1598.40
|
||||
Median TTFT (ms): 298.88
|
||||
P99 TTFT (ms): 11015.96
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 65.94
|
||||
Median TPOT (ms): 65.81
|
||||
P99 TPOT (ms): 137.73
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 62.99
|
||||
Median ITL (ms): 35.44
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.4.2 NVIDIA GB200
|
||||
|
||||
**Model Deployment (NVIDIA GB200, TP=2 NVFP4 — max tok/s/gpu config):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 5 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 1
|
||||
Successful requests: 5
|
||||
Benchmark duration (s): 24.74
|
||||
Total input tokens: 8105
|
||||
Total generated tokens: 2674
|
||||
Request throughput (req/s): 0.20
|
||||
Input token throughput (tok/s): 327.65
|
||||
Output token throughput (tok/s): 108.10
|
||||
Total token throughput (tok/s): 435.75
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4944.47
|
||||
Median E2E Latency (ms): 6347.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 211.41
|
||||
Median TTFT (ms): 207.25
|
||||
P99 TTFT (ms): 226.46
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.86
|
||||
Median TPOT (ms): 8.84
|
||||
P99 TPOT (ms): 8.96
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.87
|
||||
Median ITL (ms): 8.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.40
|
||||
Total input tokens: 179772
|
||||
Total generated tokens: 39657
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 2976.52
|
||||
Output token throughput (tok/s): 656.61
|
||||
Total token throughput (tok/s): 3633.13
|
||||
Concurrency: 13.97
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10611.51
|
||||
Median E2E Latency (ms): 9956.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 338.14
|
||||
Median TTFT (ms): 215.25
|
||||
P99 TTFT (ms): 915.40
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.87
|
||||
Median TPOT (ms): 21.36
|
||||
P99 TPOT (ms): 27.05
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.77
|
||||
Median ITL (ms): 16.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 640 \
|
||||
--max-concurrency 128 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 128
|
||||
Successful requests: 640
|
||||
Benchmark duration (s): 181.89
|
||||
Total input tokens: 1453591
|
||||
Total generated tokens: 309221
|
||||
Request throughput (req/s): 3.52
|
||||
Input token throughput (tok/s): 7991.59
|
||||
Output token throughput (tok/s): 1700.04
|
||||
Total token throughput (tok/s): 9691.63
|
||||
Concurrency: 118.86
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 33690.47
|
||||
Median E2E Latency (ms): 30421.55
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1353.16
|
||||
Median TTFT (ms): 383.52
|
||||
P99 TTFT (ms): 8940.53
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 69.88
|
||||
Median TPOT (ms): 71.77
|
||||
P99 TPOT (ms): 131.75
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 67.23
|
||||
Median ITL (ms): 33.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **4K/1K (Throughput)**: Realistic mixed workload typical of production deployments (RAG context + medium response). Long enough input that prefill matters, long enough output that decode steady-state dominates. Used for the inline B200 / GB200 results above.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-shots 5 \
|
||||
--num-questions 1319 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Test Result (NVIDIA B200, TP=2 NVFP4)
|
||||
```text Output
|
||||
Accuracy: 0.946
|
||||
Latency: 178.284 s
|
||||
Output throughput: 769.204 token/s
|
||||
```
|
||||
|
||||
- Test Result (NVIDIA GB200, TP=2 NVFP4)
|
||||
```text Output
|
||||
Accuracy: 0.951
|
||||
Latency: 175.190 s
|
||||
Invalid: 0.000
|
||||
```
|
||||
@@ -0,0 +1,740 @@
|
||||
---
|
||||
title: GLM-5.1
|
||||
metatags:
|
||||
description: "Deploy GLM-5.1 with SGLang on NVIDIA H100/H200/B300/GB300 and AMD MI300X/MI325X/MI355X."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8)
|
||||
- **NVFP4 (4-bit quantized)**: [nvidia/GLM-5.1-NVFP4](https://huggingface.co/nvidia/GLM-5.1-NVFP4)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B300, GB300, and AMD MI300X/MI325X/MI355X GPUs.
|
||||
|
||||
import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.jsx'
|
||||
|
||||
<GLM51Deployment />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.1.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>NVFP4</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>MXFP4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **H100 and H200**: FP8 is the recommended deployment path.
|
||||
- **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300.
|
||||
- **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang.
|
||||
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
|
||||
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1-FP8 \
|
||||
--tp 8 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 B300/GB300 (NVFP4) Server Command
|
||||
|
||||
#### B300
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/GLM-5.1-NVFP4 \
|
||||
--tp 8 \
|
||||
--quantization modelopt_fp4 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.80 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### GB300
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/GLM-5.1-NVFP4 \
|
||||
--tp 4 \
|
||||
--quantization modelopt_fp4 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.80 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 MI300X/MI325X/MI355X (ROCm) Server Command
|
||||
|
||||
The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above.
|
||||
|
||||
#### MXFP4 (MI355X / gfx950)
|
||||
|
||||
On MI355X (gfx950), set `SGLANG_DSA_TRITON_PREFILL=1` to enable a faster Triton attention kernel for the prefill phase (opt-in, off by default). Keep `--dsa-prefill-backend tilelang` as shown. The EAGLE speculative-decoding flags below are optional but recommended on gfx950.
|
||||
|
||||
```shell Command
|
||||
# SGLANG_DSA_TRITON_PREFILL=1 is optional; it enables a faster Triton prefill kernel on gfx950
|
||||
SGLANG_DSA_TRITON_PREFILL=1 sglang serve \
|
||||
--model-path amd/GLM-5.1-MXFP4 \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### FP8 (Recommended)
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1-FP8 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### BF16
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.3 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.4 Advanced Usage
|
||||
|
||||
#### 4.4.1 Reasoning Parser
|
||||
|
||||
GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Thinking mode is enabled by default, no extra parameters needed
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
1. **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step.
|
||||
|
||||
2. **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$.
|
||||
|
||||
3. **Step-by-Step Breakdown:**
|
||||
* *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$.
|
||||
* *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$.
|
||||
* *Step 3: Perform the calculation.*
|
||||
* $0.15 \times 240$
|
||||
* I can break this down further to make it easy to follow:
|
||||
* $0.10 \times 240 = 24$ (which is 10%)
|
||||
* $0.05 \times 240 = 12$ (which is 5%, half of 10%)
|
||||
* $24 + 12 = 36$
|
||||
* Alternatively, standard multiplication:
|
||||
* $240 \times 15 = 3600$
|
||||
* Move decimal two places left -> $36$
|
||||
* *Step 4: State the final answer clearly.*
|
||||
|
||||
4. **Draft the Response (incorporating the steps clearly):**
|
||||
* *Introduction:* State the problem clearly.
|
||||
* *Step 1:* Explain how to convert 15% to a decimal.
|
||||
* *Step 2:* Explain the multiplication step.
|
||||
* *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value).
|
||||
* *Conclusion:* Give the final answer.
|
||||
|
||||
5. **Refine the Output (Self-Correction/Polishing during drafting):**
|
||||
* *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15.
|
||||
* *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240.
|
||||
* *Drafting Step 3 (Standard way):* 0.15 × 240 = 36.
|
||||
* *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%.
|
||||
* 10% of 240 = 24 (move the decimal point one place to the left)
|
||||
* 5% is half of 10%, so half of 24 = 12
|
||||
* Add them together: 24 + 12 = 36.
|
||||
* *Final Answer:* 15% of 240 is 36.
|
||||
|
||||
6. **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output).
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution to find 15% of 240:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal.**
|
||||
To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left).
|
||||
* 15% = 15 ÷ 100 = **0.15**
|
||||
|
||||
**Step 2: Multiply the decimal by the number.**
|
||||
Now, multiply the decimal (0.15) by the number you are finding the percentage of (240).
|
||||
* 0.15 × 240 = **36**
|
||||
|
||||
*(Alternative mental math method for Step 2)*:
|
||||
If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%:
|
||||
* **10% of 240** = 24 (just move the decimal point one place to the left)
|
||||
* **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2)
|
||||
* **Add them together**: 24 + 12 = **36**
|
||||
|
||||
**Answer:**
|
||||
15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Disable thinking mode via chat_template_kwargs
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# In Instruct mode, the model responds directly without reasoning_content
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
15% of 240 is 36.
|
||||
|
||||
Here is how to calculate it:
|
||||
1. Convert the percentage to a decimal: 15% = 0.15
|
||||
2. Multiply the decimal by the number: 0.15 × 240 = 36
|
||||
```
|
||||
|
||||
#### 4.4.2 Tool Calling
|
||||
|
||||
GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location.
|
||||
=============== Content =================
|
||||
Tool Call: get_weather
|
||||
Arguments:
|
||||
Tool Call: None
|
||||
Arguments: {
|
||||
Tool Call: None
|
||||
Arguments: "location": "Be
|
||||
Tool Call: None
|
||||
Arguments: ijing"
|
||||
Tool Call: None
|
||||
Arguments: }
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: H200 (8x)
|
||||
- Model: GLM-5.1-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: commit 947927bdb
|
||||
|
||||
#### 5.1.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5.1-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.78
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4213
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 170.54
|
||||
Output token throughput (tok/s): 117.96
|
||||
Peak output token throughput (tok/s): 148.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 288.50
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.48
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3576.31
|
||||
Median E2E Latency (ms): 2935.97
|
||||
P90 E2E Latency (ms): 5908.97
|
||||
P99 E2E Latency (ms): 8588.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 290.88
|
||||
Median TTFT (ms): 282.34
|
||||
P99 TTFT (ms): 332.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.54
|
||||
Median TPOT (ms): 6.97
|
||||
P99 TPOT (ms): 9.04
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.80
|
||||
Median ITL (ms): 6.81
|
||||
P95 ITL (ms): 13.51
|
||||
P99 ITL (ms): 26.99
|
||||
Max ITL (ms): 29.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5.1-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 411.74
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 499614
|
||||
Request throughput (req/s): 2.43
|
||||
Input token throughput (tok/s): 1220.41
|
||||
Output token throughput (tok/s): 1214.97
|
||||
Peak output token throughput (tok/s): 2648.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 2435.38
|
||||
Concurrency: 96.30
|
||||
Accept length: 3.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 39648.76
|
||||
Median E2E Latency (ms): 39058.12
|
||||
P90 E2E Latency (ms): 57009.82
|
||||
P99 E2E Latency (ms): 68880.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 20613.80
|
||||
Median TTFT (ms): 21429.21
|
||||
P99 TTFT (ms): 29543.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.73
|
||||
Median TPOT (ms): 36.52
|
||||
P99 TPOT (ms): 67.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.13
|
||||
Median ITL (ms): 16.57
|
||||
P95 ITL (ms): 86.01
|
||||
P99 ITL (ms): 164.88
|
||||
Max ITL (ms): 1307.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
<Note>
|
||||
The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate benchmark run is planned.
|
||||
</Note>
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.000
|
||||
Latency: 32.470 s
|
||||
Output throughput: 642.044 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.860
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.932
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.640
|
||||
subject: college_computer_science, #q:100, acc: 0.900
|
||||
subject: college_mathematics, #q:100, acc: 0.810
|
||||
subject: college_medicine, #q:173, acc: 0.873
|
||||
subject: college_physics, #q:102, acc: 0.912
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.928
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.897
|
||||
subject: elementary_mathematics, #q:378, acc: 0.937
|
||||
subject: formal_logic, #q:126, acc: 0.778
|
||||
subject: global_facts, #q:100, acc: 0.710
|
||||
subject: high_school_biology, #q:310, acc: 0.961
|
||||
subject: high_school_chemistry, #q:203, acc: 0.847
|
||||
subject: high_school_computer_science, #q:100, acc: 0.960
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.960
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.984
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.923
|
||||
subject: high_school_mathematics, #q:270, acc: 0.696
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.821
|
||||
subject: high_school_psychology, #q:545, acc: 0.956
|
||||
subject: high_school_statistics, #q:216, acc: 0.889
|
||||
subject: high_school_us_history, #q:204, acc: 0.941
|
||||
subject: high_school_world_history, #q:237, acc: 0.945
|
||||
subject: human_aging, #q:223, acc: 0.857
|
||||
subject: human_sexuality, #q:131, acc: 0.908
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.907
|
||||
subject: logical_fallacies, #q:163, acc: 0.933
|
||||
subject: machine_learning, #q:112, acc: 0.830
|
||||
subject: management, #q:103, acc: 0.942
|
||||
subject: marketing, #q:234, acc: 0.940
|
||||
subject: medical_genetics, #q:100, acc: 0.990
|
||||
subject: miscellaneous, #q:783, acc: 0.959
|
||||
subject: moral_disputes, #q:346, acc: 0.873
|
||||
subject: moral_scenarios, #q:895, acc: 0.837
|
||||
subject: nutrition, #q:306, acc: 0.922
|
||||
subject: philosophy, #q:311, acc: 0.897
|
||||
subject: prehistory, #q:324, acc: 0.929
|
||||
subject: professional_accounting, #q:282, acc: 0.844
|
||||
subject: professional_law, #q:1534, acc: 0.714
|
||||
subject: professional_medicine, #q:272, acc: 0.941
|
||||
subject: professional_psychology, #q:612, acc: 0.913
|
||||
subject: public_relations, #q:110, acc: 0.791
|
||||
subject: security_studies, #q:245, acc: 0.878
|
||||
subject: sociology, #q:201, acc: 0.940
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.596
|
||||
subject: world_religions, #q:171, acc: 0.936
|
||||
Total latency: 165.275
|
||||
Average accuracy: 0.877
|
||||
```
|
||||
|
||||
### 5.3 AMD GPU Benchmarks
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
|
||||
|
||||
- MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends)
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
title: GLM-5.2
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/zai-org/glm-5.2.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.2-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc`. Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5, balanced 2-1-3 = 2); high-throughput has no spec.
|
||||
</Note>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Context</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.2-FP8">GLM-5.2-FP8</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · FP8</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.2">GLM-5.2</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · BF16</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/nvidia/GLM-5.2-NVFP4">GLM-5.2-NVFP4</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).
|
||||
|
||||
**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`.
|
||||
- **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute.
|
||||
- **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).
|
||||
- **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.
|
||||
- **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points.
|
||||
- **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP.
|
||||
- **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.
|
||||
|
||||
- **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).
|
||||
|
||||
<Note>
|
||||
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
|
||||
</Note>
|
||||
|
||||
- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`).
|
||||
|
||||
**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: <level>` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`:
|
||||
|
||||
| `reasoning_effort` | Injected system line | Effect |
|
||||
|---|---|---|
|
||||
| *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** |
|
||||
| `"high"` | `Reasoning Effort: High` | dials reasoning **down** |
|
||||
| `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) |
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.2-FP8",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240.
|
||||
2. **Convert the percentage to a decimal:** 15% = 0.15
|
||||
3. **Multiply by the total:** 0.15 * 240 = 36
|
||||
(Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)
|
||||
|
||||
Answer: 15% of 240 is **36**.
|
||||
|
||||
Here is how you can calculate it:
|
||||
0.15 × 240 = 36
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `<tool_call>…<arg_key>…<arg_value>…` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
resp = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.2-FP8",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.
|
||||
|
||||
Tool calls: [
|
||||
{
|
||||
"id": "call_13fcd52146934b7781d06d4a",
|
||||
"type": "function",
|
||||
"function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern.
|
||||
|
||||
### 3.4 Claude Code Integration
|
||||
|
||||
GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then:
|
||||
|
||||
```bash Command
|
||||
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
|
||||
export ANTHROPIC_AUTH_TOKEN="dummy"
|
||||
export API_TIMEOUT_MS="3000000"
|
||||
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
|
||||
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
|
||||
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
|
||||
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"
|
||||
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"
|
||||
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"
|
||||
claude
|
||||
```
|
||||
|
||||
Two of these matter specifically for GLM-5.2:
|
||||
|
||||
- **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
|
||||
- **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side.
|
||||
|
||||
For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api).
|
||||
|
||||
### 3.5 Context Parallelism
|
||||
|
||||
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:
|
||||
```bash
|
||||
--attn-cp-size 8 \
|
||||
--enable-prefill-cp \
|
||||
--cp-strategy interleave \
|
||||
```
|
||||
which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.
|
||||
|
||||
When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with
|
||||
```bash
|
||||
--enable-dsa-cache-layer-split \
|
||||
--attn-cp-size 8 \
|
||||
--cp-strategy interleave \
|
||||
```
|
||||
With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.
|
||||
@@ -0,0 +1,675 @@
|
||||
---
|
||||
title: GLM-5
|
||||
metatags:
|
||||
description: "Deploy GLM-5 with SGLang on NVIDIA H100/H200/B200 and AMD MI300X/MI325X/MI355X — state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity.
|
||||
|
||||
With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks
|
||||
- **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2)
|
||||
- **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs.
|
||||
|
||||
import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.jsx'
|
||||
|
||||
<GLM5Deployment />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **B200 (FP8)**: Use `--ep 1 --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized DSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization.
|
||||
|
||||
- **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
|
||||
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
|
||||
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
|
||||
|
||||
<Warning>
|
||||
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
|
||||
</Warning>
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy GLM-5 with the following command (FP8 on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5-FP8 \
|
||||
--tp 8 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--enable-flashinfer-allreduce-fusion \
|
||||
--mem-fraction-static 0.85 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
|
||||
|
||||
The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Reasoning Parser
|
||||
|
||||
GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Thinking mode is enabled by default, no extra parameters needed
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user wants me to solve a math problem: "What is 15% of 240?".
|
||||
|
||||
Step 1: Understand the problem. I need to calculate a percentage of a number.
|
||||
Formula: Percentage × Number = Result.
|
||||
|
||||
Step 2: Convert the percentage to a decimal or fraction.
|
||||
15% = 15/100 or 0.15.
|
||||
|
||||
Step 3: Perform the multiplication.
|
||||
Method A: Decimal multiplication.
|
||||
0.15 × 240.
|
||||
Break it down:
|
||||
10% of 240 = 24.
|
||||
5% is half of 10%, so 12.
|
||||
15% = 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Method B: Fraction multiplication.
|
||||
15/100 × 240.
|
||||
Simplify 240/100 = 2.4.
|
||||
15 × 2.4.
|
||||
10 × 2.4 = 24.
|
||||
5 × 2.4 = 12.
|
||||
24 + 12 = 36.
|
||||
|
||||
Method C: Direct multiplication.
|
||||
240 × 0.15.
|
||||
240 × 0.10 = 24.
|
||||
240 × 0.05 = 12.
|
||||
24 + 12 = 36.
|
||||
|
||||
Step 4: Final Verification.
|
||||
Is 36 reasonable?
|
||||
10% is 24. 20% is 48.
|
||||
15% is halfway between 10% and 20%.
|
||||
Halfway between 24 and 48 is 36.
|
||||
The result is correct.
|
||||
|
||||
Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%).
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal.**
|
||||
To convert 15% to a decimal, divide by 100.
|
||||
$$15\% = \frac{15}{100} = 0.15$$
|
||||
|
||||
**Step 2: Multiply the decimal by the number.**
|
||||
Now, multiply 0.15 by 240.
|
||||
$$0.15 \times 240$$
|
||||
|
||||
**Step 3: Perform the calculation.**
|
||||
You can break this down to make it easier:
|
||||
$$0.15 = 0.10 + 0.05$$
|
||||
|
||||
* First, find 10% of 240:
|
||||
$$0.10 \times 240 = 24$$
|
||||
* Next, find 5% (which is half of 10%):
|
||||
$$\frac{24}{2} = 12$$
|
||||
* Add the two results together:
|
||||
$$24 + 12 = 36$$
|
||||
|
||||
**Answer:**
|
||||
15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Disable thinking mode via chat_template_kwargs
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# In Instruct mode, the model responds directly without reasoning_content
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
To find **15% of 240**, follow these steps:
|
||||
|
||||
### Step 1: Convert the Percentage to a Decimal
|
||||
First, convert the percentage to a decimal by dividing by 100.
|
||||
|
||||
\[
|
||||
15\% = \frac{15}{100} = 0.15
|
||||
\]
|
||||
|
||||
### Step 2: Multiply by the Number
|
||||
Next, multiply the decimal by the number you want to find the percentage of.
|
||||
|
||||
\[
|
||||
0.15 \times 240
|
||||
\]
|
||||
|
||||
### Step 3: Perform the Multiplication
|
||||
Calculate the multiplication:
|
||||
|
||||
\[
|
||||
0.15 \times 240 = 36
|
||||
\]
|
||||
|
||||
### Final Answer
|
||||
\[
|
||||
\boxed{36}
|
||||
\]
|
||||
```
|
||||
|
||||
#### 4.3.2 Tool Calling
|
||||
|
||||
GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required:
|
||||
|
||||
- location: required, should be "Beijing"
|
||||
- unit: optional (not in required array), can be "celsius" or "fahrenheit"
|
||||
|
||||
Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you.
|
||||
=============== Content =================
|
||||
Tool Call: get_weather
|
||||
Arguments:
|
||||
Tool Call: None
|
||||
Arguments: {
|
||||
Tool Call: None
|
||||
Arguments: "location": "Be
|
||||
Tool Call: None
|
||||
Arguments: ijing"
|
||||
Tool Call: None
|
||||
Arguments: }
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: H200 (8x)
|
||||
- Model: GLM-5-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: commit 947927bdb
|
||||
|
||||
#### 5.1.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.78
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4213
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 170.54
|
||||
Output token throughput (tok/s): 117.96
|
||||
Peak output token throughput (tok/s): 148.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 288.50
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.48
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3576.31
|
||||
Median E2E Latency (ms): 2935.97
|
||||
P90 E2E Latency (ms): 5908.97
|
||||
P99 E2E Latency (ms): 8588.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 290.88
|
||||
Median TTFT (ms): 282.34
|
||||
P99 TTFT (ms): 332.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.54
|
||||
Median TPOT (ms): 6.97
|
||||
P99 TPOT (ms): 9.04
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.80
|
||||
Median ITL (ms): 6.81
|
||||
P95 ITL (ms): 13.51
|
||||
P99 ITL (ms): 26.99
|
||||
Max ITL (ms): 29.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 411.74
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 499614
|
||||
Request throughput (req/s): 2.43
|
||||
Input token throughput (tok/s): 1220.41
|
||||
Output token throughput (tok/s): 1214.97
|
||||
Peak output token throughput (tok/s): 2648.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 2435.38
|
||||
Concurrency: 96.30
|
||||
Accept length: 3.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 39648.76
|
||||
Median E2E Latency (ms): 39058.12
|
||||
P90 E2E Latency (ms): 57009.82
|
||||
P99 E2E Latency (ms): 68880.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 20613.80
|
||||
Median TTFT (ms): 21429.21
|
||||
P99 TTFT (ms): 29543.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.73
|
||||
Median TPOT (ms): 36.52
|
||||
P99 TPOT (ms): 67.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.13
|
||||
Median ITL (ms): 16.57
|
||||
P95 ITL (ms): 86.01
|
||||
P99 ITL (ms): 164.88
|
||||
Max ITL (ms): 1307.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
<Note>
|
||||
The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate GLM-5.1 benchmark run is planned.
|
||||
</Note>
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.000
|
||||
Latency: 32.470 s
|
||||
Output throughput: 642.044 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.860
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.932
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.640
|
||||
subject: college_computer_science, #q:100, acc: 0.900
|
||||
subject: college_mathematics, #q:100, acc: 0.810
|
||||
subject: college_medicine, #q:173, acc: 0.873
|
||||
subject: college_physics, #q:102, acc: 0.912
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.928
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.897
|
||||
subject: elementary_mathematics, #q:378, acc: 0.937
|
||||
subject: formal_logic, #q:126, acc: 0.778
|
||||
subject: global_facts, #q:100, acc: 0.710
|
||||
subject: high_school_biology, #q:310, acc: 0.961
|
||||
subject: high_school_chemistry, #q:203, acc: 0.847
|
||||
subject: high_school_computer_science, #q:100, acc: 0.960
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.960
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.984
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.923
|
||||
subject: high_school_mathematics, #q:270, acc: 0.696
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.821
|
||||
subject: high_school_psychology, #q:545, acc: 0.956
|
||||
subject: high_school_statistics, #q:216, acc: 0.889
|
||||
subject: high_school_us_history, #q:204, acc: 0.941
|
||||
subject: high_school_world_history, #q:237, acc: 0.945
|
||||
subject: human_aging, #q:223, acc: 0.857
|
||||
subject: human_sexuality, #q:131, acc: 0.908
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.907
|
||||
subject: logical_fallacies, #q:163, acc: 0.933
|
||||
subject: machine_learning, #q:112, acc: 0.830
|
||||
subject: management, #q:103, acc: 0.942
|
||||
subject: marketing, #q:234, acc: 0.940
|
||||
subject: medical_genetics, #q:100, acc: 0.990
|
||||
subject: miscellaneous, #q:783, acc: 0.959
|
||||
subject: moral_disputes, #q:346, acc: 0.873
|
||||
subject: moral_scenarios, #q:895, acc: 0.837
|
||||
subject: nutrition, #q:306, acc: 0.922
|
||||
subject: philosophy, #q:311, acc: 0.897
|
||||
subject: prehistory, #q:324, acc: 0.929
|
||||
subject: professional_accounting, #q:282, acc: 0.844
|
||||
subject: professional_law, #q:1534, acc: 0.714
|
||||
subject: professional_medicine, #q:272, acc: 0.941
|
||||
subject: professional_psychology, #q:612, acc: 0.913
|
||||
subject: public_relations, #q:110, acc: 0.791
|
||||
subject: security_studies, #q:245, acc: 0.878
|
||||
subject: sociology, #q:201, acc: 0.940
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.596
|
||||
subject: world_religions, #q:171, acc: 0.936
|
||||
Total latency: 165.275
|
||||
Average accuracy: 0.877
|
||||
```
|
||||
|
||||
### 5.3 AMD GPU Benchmarks
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
|
||||
|
||||
- MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends)
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
|
||||
@@ -0,0 +1,829 @@
|
||||
---
|
||||
title: GLM Glyph
|
||||
metatags:
|
||||
description: "Deploy GLM-Glyph with SGLang - community contribution guide for Zhipu AI's GLM Glyph model deployment."
|
||||
---
|
||||
|
||||
import { GLMGlyphDeployment } from '/src/snippets/autoregressive/glm-glyph-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options.
|
||||
|
||||
<GLMGlyphDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count. See the [GLM-4.5 cookbook page](/cookbook/autoregressive/GLM/GLM-4.5) for the full Thinking Budget usage example.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Thinking Mode
|
||||
|
||||
Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/Glyph \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
**Disable Thinking Mode:**
|
||||
|
||||
To disable thinking mode for a specific request:
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[{"role": "user", "content": "What is the capital of France?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Glyph supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/Glyph \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Model: Glyph
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/Glyph \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 17.03
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.59
|
||||
Input token throughput (tok/s): 358.17
|
||||
Output token throughput (tok/s): 247.74
|
||||
Peak output token throughput (tok/s): 251.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 605.91
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1702.14
|
||||
Median E2E Latency (ms): 1361.72
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 22.35
|
||||
Median TTFT (ms): 22.61
|
||||
P99 TTFT (ms): 23.76
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.99
|
||||
Median TPOT (ms): 3.99
|
||||
P99 TPOT (ms): 4.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.99
|
||||
Median ITL (ms): 3.99
|
||||
P95 ITL (ms): 4.03
|
||||
P99 ITL (ms): 4.12
|
||||
Max ITL (ms): 7.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 16.27
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40804
|
||||
Request throughput (req/s): 4.92
|
||||
Input token throughput (tok/s): 2438.06
|
||||
Output token throughput (tok/s): 2507.94
|
||||
Peak output token throughput (tok/s): 3069.00
|
||||
Peak concurrent requests: 26
|
||||
Total token throughput (tok/s): 4946.00
|
||||
Concurrency: 13.44
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2733.43
|
||||
Median E2E Latency (ms): 2892.98
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 33.10
|
||||
Median TTFT (ms): 27.73
|
||||
P99 TTFT (ms): 49.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.33
|
||||
Median TPOT (ms): 5.39
|
||||
P99 TPOT (ms): 5.86
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.30
|
||||
Median ITL (ms): 4.89
|
||||
P95 ITL (ms): 5.54
|
||||
P99 ITL (ms): 21.17
|
||||
Max ITL (ms): 25.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 25.67
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252657
|
||||
Request throughput (req/s): 19.48
|
||||
Input token throughput (tok/s): 9733.69
|
||||
Output token throughput (tok/s): 9843.99
|
||||
Peak output token throughput (tok/s): 13398.00
|
||||
Peak concurrent requests: 127
|
||||
Total token throughput (tok/s): 19577.68
|
||||
Concurrency: 89.49
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4593.75
|
||||
Median E2E Latency (ms): 4431.03
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 48.66
|
||||
Median TTFT (ms): 35.88
|
||||
P99 TTFT (ms): 120.61
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.10
|
||||
Median TPOT (ms): 9.55
|
||||
P99 TPOT (ms): 11.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.01
|
||||
Median ITL (ms): 6.51
|
||||
P95 ITL (ms): 23.19
|
||||
P99 ITL (ms): 25.54
|
||||
Max ITL (ms): 52.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 201.53
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44455
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 30.27
|
||||
Output token throughput (tok/s): 220.63
|
||||
Peak output token throughput (tok/s): 251.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 250.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20151.45
|
||||
Median E2E Latency (ms): 21576.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 2362.23
|
||||
Median TTFT (ms): 23.03
|
||||
P99 TTFT (ms): 21310.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.00
|
||||
Median TPOT (ms): 4.00
|
||||
P99 TPOT (ms): 4.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.00
|
||||
Median ITL (ms): 4.00
|
||||
P95 ITL (ms): 4.05
|
||||
P99 ITL (ms): 4.08
|
||||
Max ITL (ms): 5.67
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 118.67
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 318270
|
||||
Request throughput (req/s): 0.67
|
||||
Input token throughput (tok/s): 334.27
|
||||
Output token throughput (tok/s): 2682.26
|
||||
Peak output token throughput (tok/s): 3264.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3016.53
|
||||
Concurrency: 13.74
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20387.23
|
||||
Median E2E Latency (ms): 20466.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 132.47
|
||||
Median TTFT (ms): 27.19
|
||||
P99 TTFT (ms): 583.15
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.09
|
||||
Median TPOT (ms): 5.13
|
||||
P99 TPOT (ms): 5.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.09
|
||||
Median ITL (ms): 5.08
|
||||
P95 ITL (ms): 5.18
|
||||
P99 ITL (ms): 5.57
|
||||
Max ITL (ms): 522.26
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 150.00
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1300901
|
||||
Request throughput (req/s): 2.13
|
||||
Input token throughput (tok/s): 1059.59
|
||||
Output token throughput (tok/s): 8673.49
|
||||
Peak output token throughput (tok/s): 11899.00
|
||||
Peak concurrent requests: 71
|
||||
Total token throughput (tok/s): 9733.09
|
||||
Concurrency: 54.71
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 25645.42
|
||||
Median E2E Latency (ms): 26913.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 163.75
|
||||
Median TTFT (ms): 93.67
|
||||
P99 TTFT (ms): 426.19
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.27
|
||||
Median TPOT (ms): 6.39
|
||||
P99 TPOT (ms): 6.59
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.27
|
||||
Median ITL (ms): 0.17
|
||||
P95 ITL (ms): 32.94
|
||||
P99 ITL (ms): 67.89
|
||||
Max ITL (ms): 136.00
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
#### 5.1.3.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 17.44
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.57
|
||||
Input token throughput (tok/s): 2405.19
|
||||
Output token throughput (tok/s): 242.00
|
||||
Peak output token throughput (tok/s): 250.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 2647.19
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1742.54
|
||||
Median E2E Latency (ms): 1412.47
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 53.48
|
||||
Median TTFT (ms): 45.05
|
||||
P99 TTFT (ms): 98.57
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.01
|
||||
Median TPOT (ms): 4.01
|
||||
P99 TPOT (ms): 4.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.01
|
||||
Median ITL (ms): 4.01
|
||||
P95 ITL (ms): 4.06
|
||||
P99 ITL (ms): 4.09
|
||||
Max ITL (ms): 4.95
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 16.90
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41668
|
||||
Request throughput (req/s): 4.73
|
||||
Input token throughput (tok/s): 17753.58
|
||||
Output token throughput (tok/s): 2465.75
|
||||
Peak output token throughput (tok/s): 3005.00
|
||||
Peak concurrent requests: 25
|
||||
Total token throughput (tok/s): 20219.33
|
||||
Concurrency: 13.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2890.33
|
||||
Median E2E Latency (ms): 3069.55
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 41.46
|
||||
Median TTFT (ms): 31.75
|
||||
P99 TTFT (ms): 93.18
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.52
|
||||
Median TPOT (ms): 5.58
|
||||
P99 TPOT (ms): 6.14
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.48
|
||||
Median ITL (ms): 5.13
|
||||
P95 ITL (ms): 5.93
|
||||
P99 ITL (ms): 20.76
|
||||
Max ITL (ms): 36.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 35.54
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169994
|
||||
Request throughput (req/s): 9.01
|
||||
Input token throughput (tok/s): 35848.57
|
||||
Output token throughput (tok/s): 4783.96
|
||||
Peak output token throughput (tok/s): 8396.00
|
||||
Peak concurrent requests: 80
|
||||
Total token throughput (tok/s): 40632.53
|
||||
Concurrency: 59.26
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6580.96
|
||||
Median E2E Latency (ms): 6248.74
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 345.27
|
||||
Median TTFT (ms): 96.06
|
||||
P99 TTFT (ms): 2823.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.26
|
||||
Median TPOT (ms): 12.53
|
||||
P99 TPOT (ms): 23.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.76
|
||||
Median ITL (ms): 6.57
|
||||
P95 ITL (ms): 27.66
|
||||
P99 ITL (ms): 91.24
|
||||
Max ITL (ms): 2609.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.890
|
||||
Invalid: 0.000
|
||||
Latency: 3.718 s
|
||||
Output throughput: 5245.606 token/s
|
||||
```
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
title: GLM-OCR
|
||||
metatags:
|
||||
description: "Deploy GLM-OCR with SGLang - state-of-the-art OCR performance for complex document understanding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization.
|
||||
|
||||
The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
|
||||
- **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts.
|
||||
- **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments.
|
||||
- **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems.
|
||||
|
||||
For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding.
|
||||
|
||||
import { GLMOCRDeployment } from '/src/snippets/autoregressive/glm-ocr-deployment.jsx'
|
||||
|
||||
<GLMOCRDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT.
|
||||
- **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency.
|
||||
- **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 OCR Image Processing
|
||||
|
||||
GLM-OCR supports OCR tasks on various document types. Here's a basic example:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Please extract all text from this image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-OCR",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 2.29s
|
||||
Generated text: CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
|
||||
```
|
||||
|
||||
#### 4.2.2 Complex Document Processing
|
||||
|
||||
GLM-OCR excels at processing complex documents including:
|
||||
|
||||
- **Tables**: Accurate extraction of tabular data with structure preservation
|
||||
- **Formulas**: Mathematical formula recognition
|
||||
- **Code Documents**: Source code extraction from screenshots
|
||||
- **Seals and Stamps**: Recognition of seals and stamps in documents
|
||||
- **Multi-layout Documents**: Mixed content with text, images, and tables
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
# Example: Processing a document with tables
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "YOUR_DOCUMENT_IMAGE_URL"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Please extract the table content from this document and format it as markdown."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-OCR",
|
||||
messages=messages,
|
||||
max_tokens=4096
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.1.1 OCRBench Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 -m lmms_eval \
|
||||
--model openai_compatible \
|
||||
--model_args "model_version=zai-org/GLM-OCR" \
|
||||
--tasks ocrbench \
|
||||
--batch_size 128 \
|
||||
--log_samples \
|
||||
--log_samples_suffix "openai_compatible" \
|
||||
--output_path ./logs
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stderr</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yaml</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench_accuracy</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.806</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 OmniDocBench V1.5
|
||||
|
||||
GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks.
|
||||
Reference in New Issue
Block a user