[docs] sync kimi-k2.6 from sgl-cookbook (#23394)
This commit is contained in:
@@ -693,10 +693,42 @@ python3 eval.py ocrbench \
|
|||||||
|
|
||||||
#### 5.1.5 MMMU Pro Vision
|
#### 5.1.5 MMMU Pro Vision
|
||||||
|
|
||||||
```text Output
|
- Dataset: [MMMU Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro) standard 10-option subset (1,730 questions with images)
|
||||||
Pending update...
|
- Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based)
|
||||||
|
- Settings: max_tokens=32,768, thinking mode (default), max_connections=256
|
||||||
|
|
||||||
|
> **Important**: Kimi-K2.6 is a reasoning model. Setting `max_tokens` too low (e.g., 4096) causes the thinking process to consume the entire token budget, leaving no tokens for the final answer. Use `max_tokens=32768` or higher.
|
||||||
|
|
||||||
|
**Evaluation Command:**
|
||||||
|
|
||||||
|
```shell Command
|
||||||
|
cd Kimi-Vendor-Verifier
|
||||||
|
|
||||||
|
OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \
|
||||||
|
python3 eval.py mmmu \
|
||||||
|
--model openai/moonshotai/Kimi-K2.6 \
|
||||||
|
--max-tokens 32768 \
|
||||||
|
--think-mode none \
|
||||||
|
--max-connections 256
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Results (1,481/1,730 samples completed):**
|
||||||
|
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Evaluation Mode</th>
|
||||||
|
<th>Accuracy</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td>pass@1</td>
|
||||||
|
<td><strong>82.2%</strong></td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
### 5.2 Speed Benchmark
|
### 5.2 Speed Benchmark
|
||||||
|
|
||||||
**Test Environment:**
|
**Test Environment:**
|
||||||
|
|||||||
Reference in New Issue
Block a user