[Agents] Add a rule to prefer sgl-eval for accuracy evals (#44040)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
This commit is contained in:
Mohammad Miadh Angkad
2026-10-13 20:17:04 -07:00
committed by GitHub
co-authored by Mohammad Angkad
parent 53fe21cdfd
commit 4f751da975
4 changed files with 40 additions and 9 deletions
+4 -3
View File
@@ -11,6 +11,7 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
- A working SGLang installation (`pip install -e .` or equivalent)
- At least one available CUDA GPU
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
## Step-by-step Workflow
@@ -49,7 +50,7 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
```
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- This is a quick sanity check, not a rigorous benchmark.
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
@@ -108,8 +109,8 @@ for i in $(seq 1 120); do
done
# 3. Accuracy check
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
# Expected: Accuracy > 0.8
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
# Expected: score > 80%
# 4. Profile
python3 -m sglang.test.send_one --profile
+2 -2
View File
@@ -388,9 +388,9 @@ A `register_*_ci(...)` under `python/sglang/` is rejected by the
## Eval Accuracy Mixins
**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (run_eval, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (the eval call, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults.
Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults. `MMLUMixin` already scores through sgl-eval; `GSM8KMixin` still defaults to the deprecated `run_eval` backend for existing tests, so new tests set `gsm8k_backend = "sgl_eval"`. The exception is a base or completion-only model: sgl-eval only sends chat requests, so those stay on the 5-shot completion backend.
```python
class TestMyFeature(CustomTestCase, MMLUMixin):
+31
View File
@@ -0,0 +1,31 @@
# Accuracy evals: use sgl-eval
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
model's accuracy.
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
server's env, at the version pinned in the `test` extra of
`python/pyproject.toml`.
```bash
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
```
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
- **Sampling:** use the model's recommended settings (temperature, top_p,
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
them with its own defaults, so pass them explicitly, or use
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
sets the `thinking` chat-template key; if the model's template reads a
different one (commonly `enable_thinking`), pass that key with
`--chat-template-kwarg` instead. When comparing against a CI test, use that
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
base models, since sgl-eval only sends chat requests.
Only compare scores from the same harness and settings, and say which ones you
used when you report a number.
+3 -4
View File
@@ -2,9 +2,8 @@
Run few-shot GSM-8K evaluation.
.. deprecated::
This module is deprecated. Use ``sglang.test.run_eval`` with
``eval_name="gsm8k"`` instead, which routes through the unified
Chat API evaluation framework with dump_metric support.
This module is deprecated. Use ``sgl-eval run gsm8k`` or
``sglang.test.sgl_eval_utils.run_sgl_eval`` instead.
Usage:
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
@@ -58,7 +57,7 @@ def get_answer_value(answer_str):
def run_eval(args):
warnings.warn(
"sglang.test.few_shot_gsm8k is deprecated. "
"Use sglang.test.run_eval with eval_name='gsm8k' instead.",
"Use `sgl-eval run gsm8k` or sglang.test.sgl_eval_utils.run_sgl_eval instead.",
DeprecationWarning,
stacklevel=2,
)