minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Cookbook] Run accuracy benchmarks through sgl-eval (#36977)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-08-28 23:29:38 -07:00
committed by GitHub
parent 4d7d2ebb44
commit a25df83fe3
3 changed files with 49 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+11 -17
View File
@@ -813,7 +813,7 @@ python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198
#### 5.2.4 AIME 2025 Benchmark #### 5.2.4 AIME 2025 Benchmark
Results on AIME 2025 (8×B200), evaluated with [NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills.git): Results on AIME 2025 (8×B200), evaluated with [sgl-eval](https://github.com/sgl-project/sgl-eval):
<table style={{width: "100%", borderCollapse: "collapse"}}> <table style={{width: "100%", borderCollapse: "collapse"}}>
<thead> <thead>
@@ -846,25 +846,19 @@ Results on AIME 2025 (8×B200), evaluated with [NeMo-Skills](https://github.com/
</tbody> </tbody>
</table> </table>
**Reproduction.** Install [NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills), launch the server with the tool-call and reasoning parsers, then run `ns eval`: **Reproduction.** Install [sgl-eval](https://github.com/sgl-project/sgl-eval), launch the server with the tool-call and reasoning parsers, then run `sgl-eval run`. The AIME 2025 dataset ships with sgl-eval, and thinking is on by default for it:
```bash Command ```bash Command
pip install git+https://github.com/NVIDIA/NeMo-Skills.git --ignore-installed blinker pip install git+https://github.com/sgl-project/sgl-eval.git
export NEMO_SKILLS_DISABLE_UNCOMMITTED_CHANGES_CHECK=1 sgl-eval run aime25 \
ns prepare_data aime25 --base-url http://localhost:30000/v1 \
--model deepseek-ai/DeepSeek-V3.2-Exp \
ns eval \ --n-repeats 4 \
--benchmarks=aime25:4 \ --temperature 1.0 \
--server_type=sglang \ --top-p 0.95 \
--model=deepseek-ai/DeepSeek-V3.2-Exp \ --max-tokens 64000
--server_address=http://localhost:30000/v1 \ # Use --max-tokens 120000 for the DeepSeek-V3.2-Speciale model
--output_dir=nemo_skills_aime25_output \
++chat_template_kwargs.thinking=true \
++inference.temperature=1.0 \
++inference.top_p=0.95 \
++inference.tokens_to_generate=64000
# Use ++inference.tokens_to_generate=120000 for the DeepSeek-V3.2-Speciale model
``` ```
### 5.3 Speed Benchmark on Hopper ### 5.3 Speed Benchmark on Hopper
docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx
+25 -44
View File
@@ -487,30 +487,25 @@ This section uses **industry-standard configurations** for comparable benchmark
### 5.1 Accuracy Benchmark ### 5.1 Accuracy Benchmark
**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills) **Evaluation Tool**: [sgl-eval](https://github.com/sgl-project/sgl-eval)
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True` **Evaluation Settings**: temperature=0.6, top_p=0.95, 8 repeats, max_tokens=120,000 (GPQA and AIME; MMLU-Pro is greedy and single-pass, see below)
#### 5.1.1 GPQA Diamond #### 5.1.1 GPQA Diamond
- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions) - Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) - Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) -- sgl-eval's default for this benchmark
- Evaluation command: - Evaluation command:
```bash Command ```bash Command
ns prepare_data gpqa pip install git+https://github.com/sgl-project/sgl-eval.git
ns eval \ sgl-eval run gpqa \
--cluster=local \ --base-url http://localhost:30000/v1 \
--server_type=openai \ --model MiniMaxAI/MiniMax-M2.7 \
--model=MiniMaxAI/MiniMax-M2.7 \ --n-repeats 8 \
--server_address=http://localhost:30000/v1 \ --max-tokens 120000 \
--output_dir=./m2.7-eval/ \ --temperature 0.6 \
--benchmarks=gpqa:8 \ --top-p 0.95
++prompt_config=eval/aai/mcq-4choices \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
``` ```
- Test Results: - Test Results:
@@ -544,22 +539,16 @@ ns eval \
#### 5.1.2 AIME 2025 #### 5.1.2 AIME 2025
- Dataset: AIME 2025 (30 problems) - Dataset: AIME 2025 (30 problems)
- Prompt: `generic/math` (boxed answer format) - Prompt: `generic/math` (boxed answer format) -- sgl-eval's default for this benchmark
- Evaluation command: - Evaluation command:
```bash Command ```bash Command
ns prepare_data aime25 sgl-eval run aime25 \
--base-url http://localhost:30000/v1 \
ns eval \ --model MiniMaxAI/MiniMax-M2.7 \
--cluster=local \ --n-repeats 8 \
--server_type=openai \ --max-tokens 120000 \
--model=MiniMaxAI/MiniMax-M2.7 \ --temperature 0.6 \
--server_address=http://localhost:30000/v1 \ --top-p 0.95
--output_dir=./m2.7-eval/ \
--benchmarks=aime25:8 \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
``` ```
- Test Results: - Test Results:
@@ -593,22 +582,14 @@ ns eval \
#### 5.1.3 MMLU-Pro #### 5.1.3 MMLU-Pro
- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice) - Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) - Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) -- sgl-eval's default for this benchmark
- Evaluation command: - Evaluation command:
```bash Command ```bash Command
ns prepare_data mmlu-pro sgl-eval run mmlu_pro \
--base-url http://localhost:30000/v1 \
ns eval \ --model MiniMaxAI/MiniMax-M2.7 \
--cluster=local \ --max-tokens 32768 \
--server_type=openai \ --temperature 0.0
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=mmlu-pro \
++prompt_config=eval/aai/mcq-10choices \
++inference.tokens_to_generate=32768 \
++inference.temperature=0.0 \
++parse_reasoning=True
``` ```
- Test Results: - Test Results:
docs/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx
+13 -20
View File
@@ -621,32 +621,25 @@ python tool_calls_eval.py tool-calls/samples.jsonl \
#### 5.1.2 AIME 2025 #### 5.1.2 AIME 2025
- Dataset: [AIME 2025](https://huggingface.co/datasets/nvidia/aime25) (30 problems) - Dataset: [AIME 2025](https://huggingface.co/datasets/nvidia/aime25) (30 problems)
- Evaluation Tool: [NVIDIA NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills) - Evaluation Tool: [sgl-eval](https://github.com/sgl-project/sgl-eval)
- Prompt: `eval/matharena/aime` (MathArena format with `\boxed{}` answers) - Prompt: `eval/matharena/aime` (MathArena format with `\boxed{}` answers)
- Settings: temperature=1.0, top_p=0.95, max_tokens=131,072, 32 seeds - Settings: temperature=1.0, top_p=0.95, max_tokens=131,072, 32 repeats
**Evaluation Command:** **Evaluation Command:**
```shell Command ```shell Command
# Prepare dataset pip install git+https://github.com/sgl-project/sgl-eval.git
python3 nemo_skills/dataset/aime25/prepare.py
# Run 32 seeds in parallel # The dataset ships with sgl-eval; --n-repeats runs all 32 samples per problem.
for RS in $(seq 0 31); do sgl-eval run aime25 \
python3 nemo_skills/inference/generate.py \ --base-url http://localhost:30000/v1 \
input_file=nemo_skills/dataset/aime25/test.jsonl \ --model moonshotai/Kimi-K2.6 \
output_file=results/kimi-k26/aime25/output-rs${RS}.jsonl \ --n-repeats 32 \
prompt_config=eval/matharena/aime \ --temperature 1.0 \
prompt_format=openai \ --top-p 0.95 \
+server.server_type=openai \ --max-tokens 131072 \
+server.model=moonshotai/Kimi-K2.6 \ --prompt matharena-aime \
+server.base_url=http://localhost:30000/v1 \ --num-threads 512
++inference.temperature=1.0 \
++inference.top_p=0.95 \
++inference.tokens_to_generate=131072 \
++inference.random_seed=${RS} \
max_concurrent_requests=512 &
done
``` ```
**Results:** **Results:**