289 lines
10 KiB
Plaintext
289 lines
10 KiB
Plaintext
---
|
|
title: Ministral-3
|
|
metatags:
|
|
description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
|
|
---
|
|
|
|
import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';
|
|
|
|
## 1. Model Introduction
|
|
The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.
|
|
|
|
The Ministral 3 14B Instruct model offers the following capabilities:
|
|
|
|
Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
|
|
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
|
|
System Prompt: Maintains strong adherence and support for system prompts.
|
|
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
|
|
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
|
|
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
|
|
Large Context Window: Supports a 256k context window.
|
|
|
|
For further details, please refer to the [official documentation](https://github.com/mistralai)
|
|
|
|
## 2. SGLang Installation
|
|
|
|
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
|
|
|
## 3. Model Deployment
|
|
|
|
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
|
|
|
### 3.1 Basic Configuration
|
|
|
|
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
|
|
|
<Ministral3Deployment />
|
|
|
|
### 3.2 Configuration Tips
|
|
**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.
|
|
|
|
**Pre-installation steps**: Adding the following steps after launching the docker
|
|
```shell Command
|
|
pip install mistral-common --upgrade
|
|
pip install transformers==5.0.0.rc0
|
|
```
|
|
## 4. Model Invocation
|
|
|
|
### 4.1 Basic Usage
|
|
|
|
For basic API usage and request examples, please refer to:
|
|
|
|
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
|
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
|
|
|
### 4.2 Advanced Usage
|
|
|
|
#### 4.2.1 Launch the docker
|
|
```shell Command
|
|
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
|
```
|
|
|
|
```shell Command
|
|
docker run -d -it --ipc=host --network=host --privileged \
|
|
--cap-add=CAP_SYS_ADMIN \
|
|
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
|
--group-add video --cap-add=SYS_PTRACE \
|
|
--security-opt seccomp=unconfined \
|
|
-v /:/work \
|
|
-e SHELL=/bin/bash \
|
|
--name Ministral \
|
|
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
|
/bin/bash
|
|
```
|
|
|
|
#### 4.2.2 Launch the server
|
|
```shell Command
|
|
sglang serve \
|
|
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
|
--tp 1 \
|
|
--trust-remote-code
|
|
```
|
|
|
|
## 5. Benchmark
|
|
|
|
This section uses **industry-standard configurations** for comparable benchmark results.
|
|
|
|
### 5.1 Speed Benchmark
|
|
|
|
**Test Environment:**
|
|
|
|
- Hardware: MI300X GPU (8x)
|
|
- Model: mistralai/Ministral-3-14B-Instruct-2512
|
|
- Tensor Parallelism: 1
|
|
- SGLang Version: 0.5.7
|
|
|
|
- Model Deployment Command:
|
|
|
|
```bash Command
|
|
sglang serve \
|
|
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
|
--tp 1 \
|
|
--trust-remote-code
|
|
```
|
|
|
|
##### Low Concurrency
|
|
- Benchmark Command:
|
|
```bash Command
|
|
python3 -m sglang.bench_serving \
|
|
--backend sglang \
|
|
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
|
--dataset-name random \
|
|
--random-input-len 1000 \
|
|
--random-output-len 1000 \
|
|
--num-prompts 10 \
|
|
--max-concurrency 1 \
|
|
--request-rate inf
|
|
```
|
|
|
|
- Test Results:
|
|
```text Output
|
|
============ Serving Benchmark Result ============
|
|
Backend: sglang
|
|
Traffic request rate: inf
|
|
Max request concurrency: 1
|
|
Successful requests: 10
|
|
Benchmark duration (s): 65.08
|
|
Total input tokens: 6101
|
|
Total input text tokens: 6101
|
|
Total input vision tokens: 0
|
|
Total generated tokens: 4220
|
|
Total generated tokens (retokenized): 4218
|
|
Request throughput (req/s): 0.15
|
|
Input token throughput (tok/s): 93.75
|
|
Output token throughput (tok/s): 64.84
|
|
Peak output token throughput (tok/s): 151.00
|
|
Peak concurrent requests: 2
|
|
Total token throughput (tok/s): 158.59
|
|
Concurrency: 1.00
|
|
----------------End-to-End Latency----------------
|
|
Mean E2E Latency (ms): 6505.51
|
|
Median E2E Latency (ms): 3037.37
|
|
---------------Time to First Token----------------
|
|
Mean TTFT (ms): 3709.33
|
|
Median TTFT (ms): 53.72
|
|
P99 TTFT (ms): 33320.77
|
|
-----Time per Output Token (excl. 1st token)------
|
|
Mean TPOT (ms): 6.63
|
|
Median TPOT (ms): 6.64
|
|
P99 TPOT (ms): 6.66
|
|
---------------Inter-Token Latency----------------
|
|
Mean ITL (ms): 6.64
|
|
Median ITL (ms): 6.65
|
|
P95 ITL (ms): 6.75
|
|
P99 ITL (ms): 6.82
|
|
Max ITL (ms): 8.45
|
|
==================================================
|
|
```
|
|
|
|
##### Medium Concurrency
|
|
- Benchmark Command:
|
|
```bash Command
|
|
python3 -m sglang.bench_serving \
|
|
--backend sglang \
|
|
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
|
--dataset-name random \
|
|
--random-input-len 1000 \
|
|
--random-output-len 1000 \
|
|
--num-prompts 80 \
|
|
--max-concurrency 16 \
|
|
--request-rate inf
|
|
```
|
|
- Test Results:
|
|
```text Output
|
|
============ Serving Benchmark Result ============
|
|
Backend: sglang
|
|
Traffic request rate: inf
|
|
Max request concurrency: 16
|
|
Successful requests: 80
|
|
Benchmark duration (s): 31.20
|
|
Total input tokens: 39668
|
|
Total input text tokens: 39668
|
|
Total input vision tokens: 0
|
|
Total generated tokens: 40805
|
|
Total generated tokens (retokenized): 40783
|
|
Request throughput (req/s): 2.56
|
|
Input token throughput (tok/s): 1271.38
|
|
Output token throughput (tok/s): 1307.82
|
|
Peak output token throughput (tok/s): 1760.00
|
|
Peak concurrent requests: 22
|
|
Total token throughput (tok/s): 2579.20
|
|
Concurrency: 13.72
|
|
----------------End-to-End Latency----------------
|
|
Mean E2E Latency (ms): 5351.07
|
|
Median E2E Latency (ms): 5626.45
|
|
---------------Time to First Token----------------
|
|
Mean TTFT (ms): 280.87
|
|
Median TTFT (ms): 68.16
|
|
P99 TTFT (ms): 1194.79
|
|
-----Time per Output Token (excl. 1st token)------
|
|
Mean TPOT (ms): 10.47
|
|
Median TPOT (ms): 10.10
|
|
P99 TPOT (ms): 20.00
|
|
---------------Inter-Token Latency----------------
|
|
Mean ITL (ms): 9.96
|
|
Median ITL (ms): 9.10
|
|
P95 ITL (ms): 9.87
|
|
P99 ITL (ms): 51.39
|
|
Max ITL (ms): 888.63
|
|
==================================================
|
|
```
|
|
|
|
##### High Concurrency
|
|
- Benchmark Command:
|
|
```bash Command
|
|
python3 -m sglang.bench_serving \
|
|
--backend sglang \
|
|
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
|
--dataset-name random \
|
|
--random-input-len 1000 \
|
|
--random-output-len 1000 \
|
|
--num-prompts 500 \
|
|
--max-concurrency 100 \
|
|
--request-rate inf
|
|
```
|
|
|
|
- Test Results:
|
|
```text Output
|
|
============ Serving Benchmark Result ============
|
|
Backend: sglang
|
|
Traffic request rate: inf
|
|
Max request concurrency: 100
|
|
Successful requests: 500
|
|
Benchmark duration (s): 88.75
|
|
Total input tokens: 249831
|
|
Total input text tokens: 249831
|
|
Total input vision tokens: 0
|
|
Total generated tokens: 252662
|
|
Total generated tokens (retokenized): 252547
|
|
Request throughput (req/s): 5.63
|
|
Input token throughput (tok/s): 2815.01
|
|
Output token throughput (tok/s): 2846.91
|
|
Peak output token throughput (tok/s): 4271.00
|
|
Peak concurrent requests: 110
|
|
Total token throughput (tok/s): 5661.93
|
|
Concurrency: 93.04
|
|
----------------End-to-End Latency----------------
|
|
Mean E2E Latency (ms): 16514.45
|
|
Median E2E Latency (ms): 15834.45
|
|
---------------Time to First Token----------------
|
|
Mean TTFT (ms): 148.57
|
|
Median TTFT (ms): 99.15
|
|
P99 TTFT (ms): 455.86
|
|
-----Time per Output Token (excl. 1st token)------
|
|
Mean TPOT (ms): 32.93
|
|
Median TPOT (ms): 34.73
|
|
P99 TPOT (ms): 38.05
|
|
---------------Inter-Token Latency----------------
|
|
Mean ITL (ms): 32.45
|
|
Median ITL (ms): 27.30
|
|
P95 ITL (ms): 71.73
|
|
P99 ITL (ms): 73.45
|
|
Max ITL (ms): 328.10
|
|
==================================================
|
|
```
|
|
|
|
### 5.2 Accuracy Benchmark
|
|
|
|
Document model accuracy on standard benchmarks:
|
|
|
|
#### 5.2.1 GSM8K Benchmark
|
|
|
|
- Benchmark Command
|
|
|
|
```bash Command
|
|
python3 benchmark/gsm8k/bench_sglang.py \
|
|
--num-shots 8 \
|
|
--num-questions 1316 \
|
|
--parallel 1316
|
|
```
|
|
|
|
**Test Results:**
|
|
|
|
```text Output
|
|
Accuracy: 0.959
|
|
Invalid: 0.000
|
|
Latency: 29.185 s
|
|
Output throughput: 4854.672 token/s
|
|
```
|