diff --git a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx index 6af071ea8..19fd93924 100644 --- a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx +++ b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx @@ -693,10 +693,42 @@ python3 eval.py ocrbench \ #### 5.1.5 MMMU Pro Vision -```text Output -Pending update... +- Dataset: [MMMU Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro) standard 10-option subset (1,730 questions with images) +- Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based) +- Settings: max_tokens=32,768, thinking mode (default), max_connections=256 + +> **Important**: Kimi-K2.6 is a reasoning model. Setting `max_tokens` too low (e.g., 4096) causes the thinking process to consume the entire token budget, leaving no tokens for the final answer. Use `max_tokens=32768` or higher. + +**Evaluation Command:** + +```shell Command +cd Kimi-Vendor-Verifier + +OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ +python3 eval.py mmmu \ + --model openai/moonshotai/Kimi-K2.6 \ + --max-tokens 32768 \ + --think-mode none \ + --max-connections 256 ``` +**Results (1,481/1,730 samples completed):** + + + + + + + + + + + + + + +
Evaluation ModeAccuracy
pass@182.2%
+ ### 5.2 Speed Benchmark **Test Environment:**