[Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c949e91f18
commit
b819d2fb5b
@@ -0,0 +1,288 @@
|
||||
---
|
||||
title: Ministral-3
|
||||
metatags:
|
||||
description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
|
||||
---
|
||||
|
||||
import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.
|
||||
|
||||
The Ministral 3 14B Instruct model offers the following capabilities:
|
||||
|
||||
Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
|
||||
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
|
||||
System Prompt: Maintains strong adherence and support for system prompts.
|
||||
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
|
||||
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
|
||||
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
|
||||
Large Context Window: Supports a 256k context window.
|
||||
|
||||
For further details, please refer to the [official documentation](https://github.com/mistralai)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Ministral3Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.
|
||||
|
||||
**Pre-installation steps**: Adding the following steps after launching the docker
|
||||
```shell Command
|
||||
pip install mistral-common --upgrade
|
||||
pip install transformers==5.0.0.rc0
|
||||
```
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Ministral \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 Launch the server
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--tp 1 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: MI300X GPU (8x)
|
||||
- Model: mistralai/Ministral-3-14B-Instruct-2512
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--tp 1 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
##### Low Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 65.08
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4218
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 93.75
|
||||
Output token throughput (tok/s): 64.84
|
||||
Peak output token throughput (tok/s): 151.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 158.59
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6505.51
|
||||
Median E2E Latency (ms): 3037.37
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 3709.33
|
||||
Median TTFT (ms): 53.72
|
||||
P99 TTFT (ms): 33320.77
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.63
|
||||
Median TPOT (ms): 6.64
|
||||
P99 TPOT (ms): 6.66
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.64
|
||||
Median ITL (ms): 6.65
|
||||
P95 ITL (ms): 6.75
|
||||
P99 ITL (ms): 6.82
|
||||
Max ITL (ms): 8.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### Medium Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 31.20
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40783
|
||||
Request throughput (req/s): 2.56
|
||||
Input token throughput (tok/s): 1271.38
|
||||
Output token throughput (tok/s): 1307.82
|
||||
Peak output token throughput (tok/s): 1760.00
|
||||
Peak concurrent requests: 22
|
||||
Total token throughput (tok/s): 2579.20
|
||||
Concurrency: 13.72
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5351.07
|
||||
Median E2E Latency (ms): 5626.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 280.87
|
||||
Median TTFT (ms): 68.16
|
||||
P99 TTFT (ms): 1194.79
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.47
|
||||
Median TPOT (ms): 10.10
|
||||
P99 TPOT (ms): 20.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.96
|
||||
Median ITL (ms): 9.10
|
||||
P95 ITL (ms): 9.87
|
||||
P99 ITL (ms): 51.39
|
||||
Max ITL (ms): 888.63
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### High Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model mistralai/Ministral-3-14B-Instruct-2512 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 88.75
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252547
|
||||
Request throughput (req/s): 5.63
|
||||
Input token throughput (tok/s): 2815.01
|
||||
Output token throughput (tok/s): 2846.91
|
||||
Peak output token throughput (tok/s): 4271.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5661.93
|
||||
Concurrency: 93.04
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16514.45
|
||||
Median E2E Latency (ms): 15834.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 148.57
|
||||
Median TTFT (ms): 99.15
|
||||
P99 TTFT (ms): 455.86
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.93
|
||||
Median TPOT (ms): 34.73
|
||||
P99 TPOT (ms): 38.05
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 32.45
|
||||
Median ITL (ms): 27.30
|
||||
P95 ITL (ms): 71.73
|
||||
P99 ITL (ms): 73.45
|
||||
Max ITL (ms): 328.10
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
Reference in New Issue
Block a user