[docs] sync kimi-k2.6 from sgl-cookbook (#23394)

This commit is contained in:
zijiexia
2026-04-21 13:59:55 -07:00
committed by GitHub
parent c396e4924b
commit d20ae9ceaa
@@ -693,10 +693,42 @@ python3 eval.py ocrbench \
#### 5.1.5 MMMU Pro Vision
```text Output
Pending update...
- Dataset: [MMMU Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro) standard 10-option subset (1,730 questions with images)
- Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based)
- Settings: max_tokens=32,768, thinking mode (default), max_connections=256
> **Important**: Kimi-K2.6 is a reasoning model. Setting `max_tokens` too low (e.g., 4096) causes the thinking process to consume the entire token budget, leaving no tokens for the final answer. Use `max_tokens=32768` or higher.
**Evaluation Command:**
```shell Command
cd Kimi-Vendor-Verifier
OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \
python3 eval.py mmmu \
--model openai/moonshotai/Kimi-K2.6 \
--max-tokens 32768 \
--think-mode none \
--max-connections 256
```
**Results (1,481/1,730 samples completed):**
<table>
<thead>
<tr>
<th>Evaluation Mode</th>
<th>Accuracy</th>
</tr>
</thead>
<tbody>
<tr>
<td>pass@1</td>
<td><strong>82.2%</strong></td>
</tr>
</tbody>
</table>
### 5.2 Speed Benchmark
**Test Environment:**