minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Cookbook] Run accuracy benchmarks through sgl-eval (#36977)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-08-28 23:29:38 -07:00
committed by GitHub
parent 4d7d2ebb44
commit a25df83fe3
3 changed files with 49 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx
+25 -44
View File
@@ -487,30 +487,25 @@ This section uses **industry-standard configurations** for comparable benchmark
### 5.1 Accuracy Benchmark
**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills)
**Evaluation Tool**: [sgl-eval](https://github.com/sgl-project/sgl-eval)
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True`
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 repeats, max_tokens=120,000 (GPQA and AIME; MMLU-Pro is greedy and single-pass, see below)
#### 5.1.1 GPQA Diamond
- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking))
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) -- sgl-eval's default for this benchmark
- Evaluation command:
```bash Command
ns prepare_data gpqa
pip install git+https://github.com/sgl-project/sgl-eval.git
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=gpqa:8 \
++prompt_config=eval/aai/mcq-4choices \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
sgl-eval run gpqa \
--base-url http://localhost:30000/v1 \
--model MiniMaxAI/MiniMax-M2.7 \
--n-repeats 8 \
--max-tokens 120000 \
--temperature 0.6 \
--top-p 0.95
```
- Test Results:
@@ -544,22 +539,16 @@ ns eval \
#### 5.1.2 AIME 2025
- Dataset: AIME 2025 (30 problems)
- Prompt: `generic/math` (boxed answer format)
- Prompt: `generic/math` (boxed answer format) -- sgl-eval's default for this benchmark
- Evaluation command:
```bash Command
ns prepare_data aime25
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=aime25:8 \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
sgl-eval run aime25 \
--base-url http://localhost:30000/v1 \
--model MiniMaxAI/MiniMax-M2.7 \
--n-repeats 8 \
--max-tokens 120000 \
--temperature 0.6 \
--top-p 0.95
```
- Test Results:
@@ -593,22 +582,14 @@ ns eval \
#### 5.1.3 MMLU-Pro
- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice)
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) -- sgl-eval's default for this benchmark
- Evaluation command:
```bash Command
ns prepare_data mmlu-pro
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=mmlu-pro \
++prompt_config=eval/aai/mcq-10choices \
++inference.tokens_to_generate=32768 \
++inference.temperature=0.0 \
++parse_reasoning=True
sgl-eval run mmlu_pro \
--base-url http://localhost:30000/v1 \
--model MiniMaxAI/MiniMax-M2.7 \
--max-tokens 32768 \
--temperature 0.0
```
- Test Results: