[Agents] Add a rule to prefer sgl-eval for accuracy evals (#44040)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
This commit is contained in:
co-authored by
Mohammad Angkad
parent
53fe21cdfd
commit
4f751da975
@@ -11,6 +11,7 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
|
||||
|
||||
- A working SGLang installation (`pip install -e .` or equivalent)
|
||||
- At least one available CUDA GPU
|
||||
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
|
||||
|
||||
## Step-by-step Workflow
|
||||
|
||||
@@ -49,7 +50,7 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
|
||||
```
|
||||
|
||||
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
|
||||
- This is a quick sanity check, not a rigorous benchmark.
|
||||
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
|
||||
|
||||
@@ -108,8 +109,8 @@ for i in $(seq 1 120); do
|
||||
done
|
||||
|
||||
# 3. Accuracy check
|
||||
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
|
||||
# Expected: Accuracy > 0.8
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
|
||||
# Expected: score > 80%
|
||||
|
||||
# 4. Profile
|
||||
python3 -m sglang.test.send_one --profile
|
||||
|
||||
@@ -388,9 +388,9 @@ A `register_*_ci(...)` under `python/sglang/` is rejected by the
|
||||
|
||||
## Eval Accuracy Mixins
|
||||
|
||||
**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (run_eval, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
|
||||
**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (the eval call, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
|
||||
|
||||
Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults.
|
||||
Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults. `MMLUMixin` already scores through sgl-eval; `GSM8KMixin` still defaults to the deprecated `run_eval` backend for existing tests, so new tests set `gsm8k_backend = "sgl_eval"`. The exception is a base or completion-only model: sgl-eval only sends chat requests, so those stay on the 5-shot completion backend.
|
||||
|
||||
```python
|
||||
class TestMyFeature(CustomTestCase, MMLUMixin):
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# Accuracy evals: use sgl-eval
|
||||
|
||||
Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a
|
||||
model's accuracy.
|
||||
|
||||
- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the
|
||||
server's env, at the version pinned in the `test` extra of
|
||||
`python/pyproject.toml`.
|
||||
|
||||
```bash
|
||||
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200
|
||||
```
|
||||
|
||||
- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits
|
||||
that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`.
|
||||
|
||||
- **Sampling:** use the model's recommended settings (temperature, top_p,
|
||||
thinking, max tokens) unless the user says otherwise. sgl-eval overrides
|
||||
them with its own defaults, so pass them explicitly, or use
|
||||
`--load-preset-from-model-id` when the model has a preset. `--thinking` only
|
||||
sets the `thinking` chat-template key; if the model's template reads a
|
||||
different one (commonly `enable_thinking`), pass that key with
|
||||
`--chat-template-kwarg` instead. When comparing against a CI test, use that
|
||||
test's settings (printed in its log as `sgl-eval <name> generation: {...}`).
|
||||
|
||||
Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`,
|
||||
or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and
|
||||
base models, since sgl-eval only sends chat requests.
|
||||
|
||||
Only compare scores from the same harness and settings, and say which ones you
|
||||
used when you report a number.
|
||||
@@ -2,9 +2,8 @@
|
||||
Run few-shot GSM-8K evaluation.
|
||||
|
||||
.. deprecated::
|
||||
This module is deprecated. Use ``sglang.test.run_eval`` with
|
||||
``eval_name="gsm8k"`` instead, which routes through the unified
|
||||
Chat API evaluation framework with dump_metric support.
|
||||
This module is deprecated. Use ``sgl-eval run gsm8k`` or
|
||||
``sglang.test.sgl_eval_utils.run_sgl_eval`` instead.
|
||||
|
||||
Usage:
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
@@ -58,7 +57,7 @@ def get_answer_value(answer_str):
|
||||
def run_eval(args):
|
||||
warnings.warn(
|
||||
"sglang.test.few_shot_gsm8k is deprecated. "
|
||||
"Use sglang.test.run_eval with eval_name='gsm8k' instead.",
|
||||
"Use `sgl-eval run gsm8k` or sglang.test.sgl_eval_utils.run_sgl_eval instead.",
|
||||
DeprecationWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user