[Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c949e91f18
commit
b819d2fb5b
@@ -0,0 +1,518 @@
|
||||
---
|
||||
title: Devstral 2 (Mistral)
|
||||
metatags:
|
||||
description: "Deploy Devstral 2 agentic coding models with SGLang - optimized for tool use, codebase exploration, and multi-file edits with 256K context."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**.
|
||||
|
||||
The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Agentic coding**: Optimized for tool-driven coding and software engineering agents
|
||||
- **Improved performance**: A step up compared to earlier Devstral models
|
||||
- **Better generalization**: More robust across diverse prompts and coding environments
|
||||
- **Long context**: Up to a **256K** context window
|
||||
|
||||
**Use Cases:**
|
||||
AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration.
|
||||
|
||||
For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral.
|
||||
|
||||
**Models:**
|
||||
|
||||
- **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2)
|
||||
- **FP8 Instruct**:
|
||||
- **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)**
|
||||
- **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)**
|
||||
|
||||
---
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
<Warning title="Transformers version requirement">
|
||||
Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`:
|
||||
|
||||
```shell Command
|
||||
python -c "import transformers; print(transformers.__version__)"
|
||||
```
|
||||
|
||||
If your version is lower, upgrade:
|
||||
|
||||
```shell Command
|
||||
pip install -U --pre "transformers>=5.0.0rc0"
|
||||
```
|
||||
</Warning>
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B).
|
||||
|
||||
<Note>
|
||||
The TP size is set to the minimum required for the selected model size.
|
||||
</Note>
|
||||
|
||||
|
||||
import { Devstral2Deployment } from "/src/snippets/autoregressive/devstral-2-deployment.jsx";
|
||||
|
||||
<Devstral2Deployment />
|
||||
|
||||
### 3.2 Configuration tips
|
||||
|
||||
- **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable.
|
||||
- **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers.
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage (OpenAI-Compatible API)
|
||||
|
||||
SGLang exposes an OpenAI-compatible endpoint. Example:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="mistralai/Devstral-Small-2-24B-Instruct-2512",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful coding assistant."},
|
||||
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
|
||||
],
|
||||
temperature=0.2,
|
||||
max_tokens=512,
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors.
|
||||
|
||||
```python
|
||||
import time
|
||||
import requests
|
||||
from requests.exceptions import RequestException
|
||||
|
||||
def retry_with_exponential_backoff(
|
||||
url,
|
||||
max_retries=3,
|
||||
initial_delay=1,
|
||||
backoff_factor=2,
|
||||
method="GET",
|
||||
**kwargs
|
||||
):
|
||||
"""
|
||||
Retry a request with exponential backoff.
|
||||
|
||||
Parameters:
|
||||
- url: The URL to request.
|
||||
- max_retries: Maximum number of retry attempts (default: 3).
|
||||
- initial_delay: Initial delay in seconds (default: 1).
|
||||
- backoff_factor: Multiplier for the delay between retries (default: 2).
|
||||
- method: HTTP method to use (default: "GET").
|
||||
- **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.).
|
||||
|
||||
Returns:
|
||||
- Response object if the request succeeds.
|
||||
- Raises an exception if all retries fail.
|
||||
"""
|
||||
retry_count = 0
|
||||
delay = initial_delay
|
||||
|
||||
while retry_count < max_retries:
|
||||
try:
|
||||
response = requests.request(method, url, **kwargs)
|
||||
# Check if the response status code indicates success
|
||||
if response.status_code < 400:
|
||||
return response
|
||||
else:
|
||||
raise RequestException(f"HTTP {response.status_code}: {response.text}")
|
||||
|
||||
except RequestException as e:
|
||||
if retry_count == max_retries - 1:
|
||||
raise Exception(f"All retries failed. Last error: {e}")
|
||||
|
||||
print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...")
|
||||
time.sleep(delay)
|
||||
...
|
||||
```
|
||||
|
||||
### 4.2 Tool calling (optional)
|
||||
|
||||
Devstral 2 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--tp 2 \
|
||||
--tool-call-parser mistral
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Devstral-2-123B-Instruct-2512",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
|
||||
## AMD GPU Support
|
||||
|
||||
## 1. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
|
||||
### 1.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 1.2 Advanced Usage
|
||||
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
## 2.Benchmark
|
||||
|
||||
### 5.1 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 94.30
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4206
|
||||
Request throughput (req/s): 0.11
|
||||
Input token throughput (tok/s): 64.70
|
||||
Output token throughput (tok/s): 44.75
|
||||
Peak output token throughput (tok/s): 82.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 109.44
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9427.59
|
||||
Median E2E Latency (ms): 5637.23
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4253.85
|
||||
Median TTFT (ms): 116.95
|
||||
P99 TTFT (ms): 37764.48
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.28
|
||||
Median TPOT (ms): 12.29
|
||||
P99 TPOT (ms): 12.30
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 12.29
|
||||
Median ITL (ms): 12.29
|
||||
P95 ITL (ms): 12.38
|
||||
P99 ITL (ms): 12.42
|
||||
Max ITL (ms): 12.90
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 52.11
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40761
|
||||
Request throughput (req/s): 1.54
|
||||
Input token throughput (tok/s): 761.31
|
||||
Output token throughput (tok/s): 783.13
|
||||
Peak output token throughput (tok/s): 1120.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1544.44
|
||||
Concurrency: 13.60
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8856.19
|
||||
Median E2E Latency (ms): 9314.71
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 398.80
|
||||
Median TTFT (ms): 127.81
|
||||
P99 TTFT (ms): 1500.32
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.32
|
||||
Median TPOT (ms): 16.90
|
||||
P99 TPOT (ms): 32.78
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 16.61
|
||||
Median ITL (ms): 14.26
|
||||
P95 ITL (ms): 15.07
|
||||
P99 ITL (ms): 114.46
|
||||
Max ITL (ms): 1224.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Devstral-2-123B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 116.08
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252523
|
||||
Request throughput (req/s): 4.31
|
||||
Input token throughput (tok/s): 2152.21
|
||||
Output token throughput (tok/s): 2176.60
|
||||
Peak output token throughput (tok/s): 3600.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 4328.81
|
||||
Concurrency: 92.42
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 21456.71
|
||||
Median E2E Latency (ms): 20126.82
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 291.60
|
||||
Median TTFT (ms): 199.24
|
||||
P99 TTFT (ms): 866.02
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 42.42
|
||||
Median TPOT (ms): 45.18
|
||||
P99 TPOT (ms): 53.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 41.97
|
||||
Median ITL (ms): 27.59
|
||||
P95 ITL (ms): 130.43
|
||||
P99 ITL (ms): 137.87
|
||||
Max ITL (ms): 616.73
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### 5.2 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.3 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316 \
|
||||
--port 8888
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.922
|
||||
Invalid: 0.000
|
||||
Latency: 35.800 s
|
||||
Output throughput: 4507.697 token/s
|
||||
```
|
||||
@@ -0,0 +1,288 @@
|
||||
---
|
||||
title: Ministral-3
|
||||
metatags:
|
||||
description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
|
||||
---
|
||||
|
||||
import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.
|
||||
|
||||
The Ministral 3 14B Instruct model offers the following capabilities:
|
||||
|
||||
Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
|
||||
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
|
||||
System Prompt: Maintains strong adherence and support for system prompts.
|
||||
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
|
||||
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
|
||||
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
|
||||
Large Context Window: Supports a 256k context window.
|
||||
|
||||
For further details, please refer to the [official documentation](https://github.com/mistralai)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Ministral3Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.
|
||||
|
||||
**Pre-installation steps**: Adding the following steps after launching the docker
|
||||
```shell Command
|
||||
pip install mistral-common --upgrade
|
||||
pip install transformers==5.0.0.rc0
|
||||
```
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Ministral \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 Launch the server
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--tp 1 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: MI300X GPU (8x)
|
||||
- Model: mistralai/Ministral-3-14B-Instruct-2512
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--tp 1 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
##### Low Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 65.08
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4218
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 93.75
|
||||
Output token throughput (tok/s): 64.84
|
||||
Peak output token throughput (tok/s): 151.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 158.59
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6505.51
|
||||
Median E2E Latency (ms): 3037.37
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 3709.33
|
||||
Median TTFT (ms): 53.72
|
||||
P99 TTFT (ms): 33320.77
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.63
|
||||
Median TPOT (ms): 6.64
|
||||
P99 TPOT (ms): 6.66
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.64
|
||||
Median ITL (ms): 6.65
|
||||
P95 ITL (ms): 6.75
|
||||
P99 ITL (ms): 6.82
|
||||
Max ITL (ms): 8.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### Medium Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 31.20
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40783
|
||||
Request throughput (req/s): 2.56
|
||||
Input token throughput (tok/s): 1271.38
|
||||
Output token throughput (tok/s): 1307.82
|
||||
Peak output token throughput (tok/s): 1760.00
|
||||
Peak concurrent requests: 22
|
||||
Total token throughput (tok/s): 2579.20
|
||||
Concurrency: 13.72
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5351.07
|
||||
Median E2E Latency (ms): 5626.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 280.87
|
||||
Median TTFT (ms): 68.16
|
||||
P99 TTFT (ms): 1194.79
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.47
|
||||
Median TPOT (ms): 10.10
|
||||
P99 TPOT (ms): 20.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.96
|
||||
Median ITL (ms): 9.10
|
||||
P95 ITL (ms): 9.87
|
||||
P99 ITL (ms): 51.39
|
||||
Max ITL (ms): 888.63
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### High Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 88.75
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252547
|
||||
Request throughput (req/s): 5.63
|
||||
Input token throughput (tok/s): 2815.01
|
||||
Output token throughput (tok/s): 2846.91
|
||||
Peak output token throughput (tok/s): 4271.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5661.93
|
||||
Concurrency: 93.04
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16514.45
|
||||
Median E2E Latency (ms): 15834.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 148.57
|
||||
Median TTFT (ms): 99.15
|
||||
P99 TTFT (ms): 455.86
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.93
|
||||
Median TPOT (ms): 34.73
|
||||
P99 TPOT (ms): 38.05
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 32.45
|
||||
Median ITL (ms): 27.30
|
||||
P95 ITL (ms): 71.73
|
||||
P99 ITL (ms): 73.45
|
||||
Max ITL (ms): 328.10
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
@@ -0,0 +1,456 @@
|
||||
---
|
||||
title: Mistral Medium 3.5
|
||||
metatags:
|
||||
description: "Deploy Mistral Medium 3.5 with SGLang - 128B dense flagship merged model with hybrid reasoning, 256K context, vision input, and FP8 quantization."
|
||||
---
|
||||
|
||||
import { MistralMedium35Deployment } from '/src/snippets/autoregressive/mistral-medium-3-5-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head_dim=128)
|
||||
- **256K context window** — YARN RoPE scaling on top of the original 4K base
|
||||
- **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`)
|
||||
- **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios
|
||||
- **Function Calling**: Native tool calling and JSON output
|
||||
- **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in
|
||||
- **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali
|
||||
- **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue)
|
||||
|
||||
**Architecture:**
|
||||
|
||||
- Mistral 3 backbone with YARN RoPE for 256K context
|
||||
- Dense (no MoE), 128B parameters
|
||||
- Standard GQA attention (not MLA)
|
||||
- Pixtral-style vision encoder (48 layers, patch_size=14, spatial_merge=2, image_size=1540) trained from scratch
|
||||
- Multimodal input: text + image
|
||||
|
||||
**Models:**
|
||||
|
||||
- **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8)
|
||||
|
||||
The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present.
|
||||
|
||||
---
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
**Docker Image:** `lmsysorg/sglang:latest` covers all the GPUs in this cookbook (H100 / H200 / B200 / B300).
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5.
|
||||
|
||||
<MistralMedium35Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Tensor Parallelism**: Mistral Medium 3.5 FP8 (~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300).
|
||||
- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
|
||||
- **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output.
|
||||
- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
|
||||
- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
|
||||
- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
|
||||
- **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6).
|
||||
|
||||
### 3.3 Speculative Decoding (EAGLE)
|
||||
|
||||
Mistral ships an EAGLE draft head, [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE), that lets you run speculative decoding on top of the dense 128B target. The draft is a 2-layer GQA body sharing the target's vocab/head, FP8-quantized like the target (~4 GB), and is meant for low-concurrency latency-bound serving.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path mistralai/Mistral-Medium-3.5-128B \
|
||||
--tp 4 \
|
||||
--dtype bfloat16 \
|
||||
--tool-call-parser mistral \
|
||||
--reasoning-parser mistral \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-draft-model-path mistralai/Mistral-Medium-3.5-128B-EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- **`--dtype bfloat16` is required.** The draft `params.json` does not carry a `dtype` field, so `--dtype auto` falls back to fp32 and downcasts to fp16, which conflicts with the bf16 target when the embed/head are shared. Setting bf16 explicitly keeps both sides aligned (this is a no-op for the target — it already loads as bf16).
|
||||
- The draft uses the same vocab and lm_head as the target. Memory overhead on top of the base model is ~4 GB per TP shard.
|
||||
- `(num-steps, eagle-topk, num-draft-tokens) = (3, 1, 4)` is the recommended starting point. Tune for your workload — wider trees (higher `eagle-topk` / `num-draft-tokens`) help high-acceptance (templated) outputs, narrower trees keep latency tight on more diverse text.
|
||||
- EAGLE shines at low concurrency. At high concurrency, throughput is dominated by the target's batched forward pass and the draft's contribution shrinks; consider running without EAGLE for batch-serving workloads.
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Thinking Mode
|
||||
|
||||
Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
|
||||
],
|
||||
temperature=0.7,
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("Answer:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and
|
||||
division before addition, evaluated left to right.
|
||||
|
||||
17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391.
|
||||
144 / 12 = 12.
|
||||
Finally, 391 + 12 = 403.
|
||||
|
||||
Answer: **17 × 23 + 144 / 12 = 403**
|
||||
|
||||
Step by step:
|
||||
1. 17 × 23 = 391
|
||||
2. 144 / 12 = 12
|
||||
3. 391 + 12 = 403
|
||||
```
|
||||
|
||||
### 4.2 Instruct Mode (Reasoning Off)
|
||||
|
||||
To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is the capital of France?"},
|
||||
],
|
||||
temperature=0.1,
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
The capital of France is **Paris**. It is one of the most famous and visited cities in
|
||||
the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower,
|
||||
Louvre Museum, and Notre-Dame Cathedral.
|
||||
```
|
||||
|
||||
### 4.3 Streaming with Reasoning
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain the difference between async and threading in Python."},
|
||||
],
|
||||
temperature=0.7,
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
print("=== Reasoning ===")
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
elif delta.content:
|
||||
print("\n=== Response ===")
|
||||
print(delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
### 4.4 Tool Calling
|
||||
|
||||
Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
)
|
||||
|
||||
tool_calls = response.choices[0].message.tool_calls
|
||||
for tc in tool_calls:
|
||||
print(f"Tool: {tc.function.name}")
|
||||
print(f"Args: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Paris"}
|
||||
```
|
||||
|
||||
### 4.5 Vision (Image Input)
|
||||
|
||||
Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Medium-3.5-128B",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe what you see in this image."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0.7,
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
The image features a stylized representation of the acronym "SGL." The letters
|
||||
are large, bold, and orange with a brown outline, giving them a three-dimensional
|
||||
effect. To the left of the letters, there is a graphic that resembles a neuron
|
||||
or a node with connections, also in a similar orange and brown color scheme. The
|
||||
node has a code symbol (</>) inside a square, suggesting a connection to
|
||||
programming or technology.
|
||||
```
|
||||
|
||||
### 4.6 Loading the Reference System Prompt
|
||||
|
||||
Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly:
|
||||
|
||||
```python Example
|
||||
from datetime import datetime, timedelta
|
||||
from huggingface_hub import hf_hub_download
|
||||
from openai import OpenAI
|
||||
|
||||
MODEL = "mistralai/Mistral-Medium-3.5-128B"
|
||||
|
||||
def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str:
|
||||
path = hf_hub_download(repo_id=repo_id, filename=filename)
|
||||
today = datetime.today().strftime("%Y-%m-%d")
|
||||
yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
|
||||
name = repo_id.split("/")[-1]
|
||||
with open(path) as f:
|
||||
return f.read().format(name=name, today=today, yesterday=yesterday)
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=MODEL,
|
||||
messages=[
|
||||
{"role": "system", "content": load_system_prompt(MODEL)},
|
||||
{"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."},
|
||||
],
|
||||
temperature=0.1,
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmarks
|
||||
|
||||
Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint.
|
||||
|
||||
### 5.1 Accuracy Benchmarks
|
||||
|
||||
#### GSM8K
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.945
|
||||
Invalid: 0.000
|
||||
Latency: 13.594 s
|
||||
Output throughput: 1560.660 token/s
|
||||
```
|
||||
|
||||
#### MMMU
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Overall accuracy: 0.586
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmarks
|
||||
|
||||
#### Latency (Low Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 38.86
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 2684
|
||||
Output token throughput (tok/s): 69.07
|
||||
Mean E2E Latency (ms): 3883.80
|
||||
Median TTFT (ms): 95.90
|
||||
Median TPOT (ms): 14.19
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### Throughput (High Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 117.28
|
||||
Total input tokens: 512842
|
||||
Total generated tokens: 262023
|
||||
Output token throughput (tok/s): 2234.18
|
||||
Total token throughput (tok/s): 6607.01
|
||||
Mean E2E Latency (ms): 11303.79
|
||||
Median TTFT (ms): 152.95
|
||||
Median TPOT (ms): 42.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.3 EAGLE Speculative Decoding (Latency)
|
||||
|
||||
Same 4× H200 setup, EAGLE configuration from [Section 3.3](#3-3-speculative-decoding-eagle). Single-stream latency benchmark (`--max-concurrency 1`).
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 27.64
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 2684
|
||||
Output token throughput (tok/s): 97.10
|
||||
Mean E2E Latency (ms): 2762.99
|
||||
Median TTFT (ms): 90.69
|
||||
Median TPOT (ms): 9.73
|
||||
Accept length: 1.72
|
||||
==================================================
|
||||
```
|
||||
|
||||
EAGLE delivers **~1.41× output throughput and ~29% lower E2E latency** vs. the baseline in [Section 5.2](#5-2-speed-benchmarks) on the same workload. Acceptance length of 1.72 means each draft cycle averages roughly 1.7 accepted tokens.
|
||||
@@ -0,0 +1,393 @@
|
||||
---
|
||||
title: Mistral Small 4
|
||||
metatags:
|
||||
description: "Deploy Mistral Small 4 with SGLang - unified hybrid model combining instruct, reasoning, and agentic capabilities with multimodal support."
|
||||
---
|
||||
|
||||
import { MistralSmall4Deployment } from '/src/snippets/autoregressive/mistral-small-4-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model.
|
||||
|
||||
With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request
|
||||
- **Vision**: Accepts both text and image inputs, providing insights based on visual content
|
||||
- **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities
|
||||
- **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more
|
||||
- **Context Window**: 256K context window
|
||||
- **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters)
|
||||
- **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes
|
||||
- Reasoning effort supported are only **"none" and "high"**
|
||||
|
||||
**Architecture:**
|
||||
|
||||
- Same general architecture as Mistral 3
|
||||
- MoE: 128 experts, 4 active per token
|
||||
- 119B total parameters, 6.5B activated per token
|
||||
- Multimodal input: text + image
|
||||
|
||||
**Models:**
|
||||
|
||||
- **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8)
|
||||
- **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)**
|
||||
- **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603
|
||||
- **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference
|
||||
|
||||
---
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
<Info>
|
||||
Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/install).
|
||||
</Info>
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4.
|
||||
|
||||
<MistralSmall4Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Tensor Parallelism**: Mistral Small 4 FP8 (~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (~60 GB, Blackwell only) runs with tp=1.
|
||||
- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
|
||||
- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
|
||||
- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
|
||||
- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
|
||||
- **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency.
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Thinking Mode
|
||||
|
||||
Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
|
||||
],
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("Answer:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Reasoning: First, I'll break down the problem into two parts: the multiplication and
|
||||
the division. According to the order of operations (PEMDAS/BODMAS), multiplication and
|
||||
division are performed from left to right before addition.
|
||||
|
||||
17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391
|
||||
144 / 12 = 12
|
||||
|
||||
Finally, add the results: 391 + 12 = 403
|
||||
|
||||
Answer: The solution to the problem is as follows:
|
||||
|
||||
1. First, perform the multiplication: 17 × 23.
|
||||
- 17 × 20 = 340
|
||||
- 17 × 3 = 51
|
||||
- 340 + 51 = 391
|
||||
|
||||
2. Then, perform the division: 144 / 12 = 12.
|
||||
|
||||
3. Finally, add the results:
|
||||
- 391 + 12 = 403
|
||||
|
||||
**Answer:** \boxed{403}
|
||||
```
|
||||
|
||||
### 4.2 Instruct Mode (Reasoning Off)
|
||||
|
||||
To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to reverse a string."},
|
||||
],
|
||||
extra_body={"reasoning_effort": "none"},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
````text Output
|
||||
# Python Function to Reverse a String
|
||||
|
||||
Here are several ways to write a Python function to reverse a string:
|
||||
|
||||
## Method 1: Using String Slicing (Most Pythonic)
|
||||
```python
|
||||
def reverse_string(s):
|
||||
"""Reverse a string using slicing."""
|
||||
return s[::-1]
|
||||
```
|
||||
|
||||
## Method 2: Using a Loop
|
||||
```python Example
|
||||
def reverse_string(s):
|
||||
"""Reverse a string using a loop."""
|
||||
reversed_str = ""
|
||||
for char in s:
|
||||
reversed_str = char + reversed_str
|
||||
return reversed_str
|
||||
```
|
||||
|
||||
## Method 3: Using reversed() function
|
||||
```python Example
|
||||
def reverse_string(s):
|
||||
"""Reverse a string using reversed() function."""
|
||||
return ''.join(reversed(s))
|
||||
```
|
||||
|
||||
The first method using string slicing (`s[::-1]`) is generally the most efficient and
|
||||
recommended approach in Python.
|
||||
|
||||
Example usage:
|
||||
```python Example
|
||||
original = "Hello, World!"
|
||||
reversed_str = reverse_string(original)
|
||||
print(reversed_str) # Output: "!dlroW ,olleH"
|
||||
```
|
||||
````
|
||||
|
||||
### 4.3 Streaming with Reasoning
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain the difference between async and threading in Python."},
|
||||
],
|
||||
extra_body={"reasoning_effort": "high"},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
print("=== Reasoning ===")
|
||||
for chunk in stream:
|
||||
delta = chunk.choices[0].delta
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
elif delta.content:
|
||||
print("\n=== Response ===")
|
||||
print(delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
=== Reasoning ===
|
||||
Okay, the user is asking about the difference between async and threading in Python.
|
||||
I need to break this down clearly, covering the key aspects of both, like their
|
||||
purposes, performance characteristics, and use cases...
|
||||
=== Response ===
|
||||
In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency
|
||||
models, each suited for specific use cases. Here's a breakdown of their key differences:
|
||||
|
||||
### 1. Model of Concurrency
|
||||
- **Threading**: Based on preemptive multitasking using OS threads.
|
||||
- **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield...
|
||||
```
|
||||
|
||||
### 4.4 Tool Calling
|
||||
|
||||
Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "City name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
)
|
||||
|
||||
tool_calls = response.choices[0].message.tool_calls
|
||||
for tc in tool_calls:
|
||||
print(f"Tool: {tc.function.name}")
|
||||
print(f"Args: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Paris"}
|
||||
```
|
||||
|
||||
### 4.5 Vision (Image Input)
|
||||
|
||||
Mistral Small 4 accepts image inputs alongside text:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mistralai/Mistral-Small-4-119B-2603",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe what you see in this image."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```text Output
|
||||
The image is a copyright symbol, represented by a stylized version of the lowercase
|
||||
letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and
|
||||
the circle is orange. The design is simple yet striking, using oval and elliptical
|
||||
shapes to create a distinct symbol which signifies copyright protection.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmarks
|
||||
|
||||
### 5.1 Accuracy Benchmarks
|
||||
|
||||
#### GSM8K
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
|
||||
#### MMLU
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmarks
|
||||
|
||||
#### Latency (Low Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
|
||||
#### Throughput (High Concurrency)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 512 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
TODO
|
||||
```
|
||||
Reference in New Issue
Block a user