Bugfix (#24027)

2026-04-29 21:13:51 +08:00
parent 4c1eefca4f
commit 1279ae0787
8 changed files with 885 additions and 5 deletions
@@ -0,0 +1,398 @@
+---
+title: Mistral Medium 3.5
+metatags:
+    description: "Deploy Mistral Medium 3.5 with SGLang - 128B dense flagship merged model with hybrid reasoning, 256K context, vision input, and FP8 quantization."
+---
+
+import { MistralMedium35Deployment } from '/src/snippets/autoregressive/mistral-medium-3-5-deployment.jsx';
+
+## 1. Model Introduction
+
+**Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios.
+
+**Key Features:**
+
+- **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head_dim=128)
+- **256K context window** — YARN RoPE scaling on top of the original 4K base
+- **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`)
+- **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios
+- **Function Calling**: Native tool calling and JSON output
+- **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in
+- **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali
+- **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue)
+
+**Architecture:**
+
+- Mistral 3 backbone with YARN RoPE for 256K context
+- Dense (no MoE), 128B parameters
+- Standard GQA attention (not MLA)
+- Pixtral-style vision encoder (48 layers, patch_size=14, spatial_merge=2, image_size=1540) trained from scratch
+- Multimodal input: text + image
+
+**Models:**
+
+- **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8)
+
+The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present.
+
+---
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
+
+---
+
+## 3. Model Deployment
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5.
+
+<MistralMedium35Deployment />
+
+### 3.2 Configuration Tips
+
+- **Tensor Parallelism**: Mistral Medium 3.5 FP8 (~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300).
+- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
+- **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output.
+- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
+- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
+- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
+- **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6).
+
+---
+
+## 4. Model Invocation
+
+### 4.1 Thinking Mode
+
+Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Medium-3.5-128B",
+    messages=[
+        {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
+    ],
+    temperature=0.7,
+    extra_body={"reasoning_effort": "high"},
+)
+
+print("Reasoning:", response.choices[0].message.reasoning_content)
+print("Answer:", response.choices[0].message.content)
+```
+
+**Output:**
+
+```text Output
+Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and
+division before addition, evaluated left to right.
+
+17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391.
+144 / 12 = 12.
+Finally, 391 + 12 = 403.
+
+Answer: **17 × 23 + 144 / 12 = 403**
+
+Step by step:
+1. 17 × 23 = 391
+2. 144 / 12 = 12
+3. 391 + 12 = 403
+```
+
+### 4.2 Instruct Mode (Reasoning Off)
+
+To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Medium-3.5-128B",
+    messages=[
+        {"role": "user", "content": "What is the capital of France?"},
+    ],
+    temperature=0.1,
+    extra_body={"reasoning_effort": "none"},
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output:**
+
+```text Output
+The capital of France is **Paris**. It is one of the most famous and visited cities in
+the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower,
+Louvre Museum, and Notre-Dame Cathedral.
+```
+
+### 4.3 Streaming with Reasoning
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="mistralai/Mistral-Medium-3.5-128B",
+    messages=[
+        {"role": "user", "content": "Explain the difference between async and threading in Python."},
+    ],
+    temperature=0.7,
+    extra_body={"reasoning_effort": "high"},
+    stream=True,
+)
+
+print("=== Reasoning ===")
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
+        print(delta.reasoning_content, end="", flush=True)
+    elif delta.content:
+        print("\n=== Response ===")
+        print(delta.content, end="", flush=True)
+print()
+```
+
+### 4.4 Tool Calling
+
+Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a city",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {"type": "string", "description": "City name"},
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                },
+                "required": ["location"],
+            },
+        },
+    }
+]
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Medium-3.5-128B",
+    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
+    tools=tools,
+    tool_choice="auto",
+)
+
+tool_calls = response.choices[0].message.tool_calls
+for tc in tool_calls:
+    print(f"Tool: {tc.function.name}")
+    print(f"Args: {tc.function.arguments}")
+```
+
+**Output:**
+
+```text Output
+Tool: get_weather
+Args: {"location": "Paris"}
+```
+
+### 4.5 Vision (Image Input)
+
+Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Medium-3.5-128B",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Describe what you see in this image."},
+                {
+                    "type": "image_url",
+                    "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
+                },
+            ],
+        }
+    ],
+    temperature=0.7,
+    extra_body={"reasoning_effort": "none"},
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output:**
+
+```text Output
+The image features a stylized representation of the acronym "SGL." The letters
+are large, bold, and orange with a brown outline, giving them a three-dimensional
+effect. To the left of the letters, there is a graphic that resembles a neuron
+or a node with connections, also in a similar orange and brown color scheme. The
+node has a code symbol (</>) inside a square, suggesting a connection to
+programming or technology.
+```
+
+### 4.6 Loading the Reference System Prompt
+
+Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly:
+
+```python Example
+from datetime import datetime, timedelta
+from huggingface_hub import hf_hub_download
+from openai import OpenAI
+
+MODEL = "mistralai/Mistral-Medium-3.5-128B"
+
+def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str:
+    path = hf_hub_download(repo_id=repo_id, filename=filename)
+    today = datetime.today().strftime("%Y-%m-%d")
+    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
+    name = repo_id.split("/")[-1]
+    with open(path) as f:
+        return f.read().format(name=name, today=today, yesterday=yesterday)
+
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+
+response = client.chat.completions.create(
+    model=MODEL,
+    messages=[
+        {"role": "system", "content": load_system_prompt(MODEL)},
+        {"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."},
+    ],
+    temperature=0.1,
+    extra_body={"reasoning_effort": "none"},
+)
+
+print(response.choices[0].message.content)
+```
+
+---
+
+## 5. Benchmarks
+
+Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint.
+
+### 5.1 Accuracy Benchmarks
+
+#### GSM8K
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --port 30000
+```
+
+**Results:**
+
+```text Output
+Accuracy: 0.945
+Invalid: 0.000
+Latency: 13.594 s
+Output throughput: 1560.660 token/s
+```
+
+#### MMMU
+
+```bash Command
+python3 benchmark/mmmu/bench_sglang.py --port 30000
+```
+
+**Results:**
+
+```text Output
+Overall accuracy: 0.586
+```
+
+### 5.2 Speed Benchmarks
+
+#### Latency (Low Concurrency)
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --dataset-name random \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --port 30000
+```
+
+**Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Successful requests:                     10
+Benchmark duration (s):                  38.86
+Total input tokens:                      6101
+Total generated tokens:                  2684
+Output token throughput (tok/s):         69.07
+Mean E2E Latency (ms):                   3883.80
+Median TTFT (ms):                        95.90
+Median TPOT (ms):                        14.19
+==================================================
+```
+
+#### Throughput (High Concurrency)
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --dataset-name random \
+  --num-prompts 1000 \
+  --max-concurrency 100 \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --port 30000
+```
+
+**Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Successful requests:                     1000
+Benchmark duration (s):                  117.28
+Total input tokens:                      512842
+Total generated tokens:                  262023
+Output token throughput (tok/s):         2234.18
+Total token throughput (tok/s):          6607.01
+Mean E2E Latency (ms):                   11303.79
+Median TTFT (ms):                        152.95
+Median TPOT (ms):                        42.53
+==================================================
+```