sglang/docs/cookbook/autoregressive/Mistral/Mistral-Medium-3.5.mdx

---
title: Mistral Medium 3.5
metatags:
    description: "Deploy Mistral Medium 3.5 with SGLang - 128B dense flagship merged model with hybrid reasoning, 256K context, vision input, and FP8 quantization."
---

import { MistralMedium35Deployment } from '/src/snippets/autoregressive/mistral-medium-3-5-deployment.jsx';

## 1. Model Introduction

**Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios.

**Key Features:**

- **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head_dim=128)
- **256K context window** — YARN RoPE scaling on top of the original 4K base
- **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`)
- **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios
- **Function Calling**: Native tool calling and JSON output
- **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in
- **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali
- **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue)

**Architecture:**

- Mistral 3 backbone with YARN RoPE for 256K context
- Dense (no MoE), 128B parameters
- Standard GQA attention (not MLA)
- Pixtral-style vision encoder (48 layers, patch_size=14, spatial_merge=2, image_size=1540) trained from scratch
- Multimodal input: text + image

**Models:**

- **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8)

The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present.

---

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install).

**Docker Image:** `lmsysorg/sglang:latest` covers all the GPUs in this cookbook (H100 / H200 / B200 / B300).

---

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5.

<MistralMedium35Deployment />

### 3.2 Configuration Tips

- **Tensor Parallelism**: Mistral Medium 3.5 FP8 (~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300).
- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
- **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output.
- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
- **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6).

### 3.3 Speculative Decoding (EAGLE)

Mistral ships an EAGLE draft head, [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE), that lets you run speculative decoding on top of the dense 128B target. The draft is a 2-layer GQA body sharing the target's vocab/head, FP8-quantized like the target (~4 GB), and is meant for low-concurrency latency-bound serving.

```bash Command
python -m sglang.launch_server \
  --model-path mistralai/Mistral-Medium-3.5-128B \
  --tp 4 \
  --dtype bfloat16 \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path mistralai/Mistral-Medium-3.5-128B-EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --port 30000
```

- **`--dtype bfloat16` is required.** The draft `params.json` does not carry a `dtype` field, so `--dtype auto` falls back to fp32 and downcasts to fp16, which conflicts with the bf16 target when the embed/head are shared. Setting bf16 explicitly keeps both sides aligned (this is a no-op for the target — it already loads as bf16).
- The draft uses the same vocab and lm_head as the target. Memory overhead on top of the base model is ~4 GB per TP shard.
- `(num-steps, eagle-topk, num-draft-tokens) = (3, 1, 4)` is the recommended starting point. Tune for your workload — wider trees (higher `eagle-topk` / `num-draft-tokens`) help high-acceptance (templated) outputs, narrower trees keep latency tight on more diverse text.
- EAGLE shines at low concurrency. At high concurrency, throughput is dominated by the target's batched forward pass and the draft's contribution shrinks; consider running without EAGLE for batch-serving workloads.

---

## 4. Model Invocation

### 4.1 Thinking Mode

Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode.

```python Example
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
    ],
    temperature=0.7,
    extra_body={"reasoning_effort": "high"},
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("Answer:", response.choices[0].message.content)
```

**Output:**

```text Output
Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and
division before addition, evaluated left to right.

17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391.
144 / 12 = 12.
Finally, 391 + 12 = 403.

Answer: **17 × 23 + 144 / 12 = 403**

Step by step:
1. 17 × 23 = 391
2. 144 / 12 = 12
3. 391 + 12 = 403
```

### 4.2 Instruct Mode (Reasoning Off)

To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is:

```python Example
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
    ],
    temperature=0.1,
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

**Output:**

```text Output
The capital of France is **Paris**. It is one of the most famous and visited cities in
the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower,
Louvre Museum, and Notre-Dame Cathedral.
```

### 4.3 Streaming with Reasoning

```python Example
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {"role": "user", "content": "Explain the difference between async and threading in Python."},
    ],
    temperature=0.7,
    extra_body={"reasoning_effort": "high"},
    stream=True,
)

print("=== Reasoning ===")
for chunk in stream:
    delta = chunk.choices[0].delta
    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
        print(delta.reasoning_content, end="", flush=True)
    elif delta.content:
        print("\n=== Response ===")
        print(delta.content, end="", flush=True)
print()
```

### 4.4 Tool Calling

Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`:

```python Example
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

tool_calls = response.choices[0].message.tool_calls
for tc in tool_calls:
    print(f"Tool: {tc.function.name}")
    print(f"Args: {tc.function.arguments}")
```

**Output:**

```text Output
Tool: get_weather
Args: {"location": "Paris"}
```

### 4.5 Vision (Image Input)

Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios:

```python Example
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see in this image."},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
                },
            ],
        }
    ],
    temperature=0.7,
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

**Output:**

```text Output
The image features a stylized representation of the acronym "SGL." The letters
are large, bold, and orange with a brown outline, giving them a three-dimensional
effect. To the left of the letters, there is a graphic that resembles a neuron
or a node with connections, also in a similar orange and brown color scheme. The
node has a code symbol (</>) inside a square, suggesting a connection to
programming or technology.
```

### 4.6 Loading the Reference System Prompt

Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly:

```python Example
from datetime import datetime, timedelta
from huggingface_hub import hf_hub_download
from openai import OpenAI

MODEL = "mistralai/Mistral-Medium-3.5-128B"

def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str:
    path = hf_hub_download(repo_id=repo_id, filename=filename)
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    name = repo_id.split("/")[-1]
    with open(path) as f:
        return f.read().format(name=name, today=today, yesterday=yesterday)

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": load_system_prompt(MODEL)},
        {"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."},
    ],
    temperature=0.1,
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

---

## 5. Benchmarks

Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint.

### 5.1 Accuracy Benchmarks

#### GSM8K

```bash Command
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```

**Results:**

```text Output
Accuracy: 0.945
Invalid: 0.000
Latency: 13.594 s
Output throughput: 1560.660 token/s
```

#### MMMU

```bash Command
python3 benchmark/mmmu/bench_sglang.py --port 30000
```

**Results:**

```text Output
Overall accuracy: 0.586
```

### 5.2 Speed Benchmarks

#### Latency (Low Concurrency)

```bash Command
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 10 \
  --max-concurrency 1 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Successful requests:                     10
Benchmark duration (s):                  38.86
Total input tokens:                      6101
Total generated tokens:                  2684
Output token throughput (tok/s):         69.07
Mean E2E Latency (ms):                   3883.80
Median TTFT (ms):                        95.90
Median TPOT (ms):                        14.19
==================================================
```

#### Throughput (High Concurrency)

```bash Command
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Successful requests:                     1000
Benchmark duration (s):                  117.28
Total input tokens:                      512842
Total generated tokens:                  262023
Output token throughput (tok/s):         2234.18
Total token throughput (tok/s):          6607.01
Mean E2E Latency (ms):                   11303.79
Median TTFT (ms):                        152.95
Median TPOT (ms):                        42.53
==================================================
```

### 5.3 EAGLE Speculative Decoding (Latency)

Same 4× H200 setup, EAGLE configuration from [Section 3.3](#3-3-speculative-decoding-eagle). Single-stream latency benchmark (`--max-concurrency 1`).

```bash Command
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 10 \
  --max-concurrency 1 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Successful requests:                     10
Benchmark duration (s):                  27.64
Total input tokens:                      6101
Total generated tokens:                  2684
Output token throughput (tok/s):         97.10
Mean E2E Latency (ms):                   2762.99
Median TTFT (ms):                        90.69
Median TPOT (ms):                        9.73
Accept length:                           1.72
==================================================
```

EAGLE delivers **~1.41× output throughput and ~29% lower E2E latency** vs. the baseline in [Section 5.2](#5-2-speed-benchmarks) on the same workload. Acceptance length of 1.72 means each draft cycle averages roughly 1.7 accepted tokens.