diff --git a/.agents/skills/generate-profile/SKILL.md b/.agents/skills/generate-profile/SKILL.md index dc744cf4fe..bc22fd5e0b 100644 --- a/.agents/skills/generate-profile/SKILL.md +++ b/.agents/skills/generate-profile/SKILL.md @@ -11,6 +11,7 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g - A working SGLang installation (`pip install -e .` or equivalent) - At least one available CUDA GPU +- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env ## Step-by-step Workflow @@ -49,7 +50,7 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra sgl-eval run gsm8k --base-url http://127.0.0.1:/v1 --num-examples 20 ``` -- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.) +- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.) - This is a quick sanity check, not a rigorous benchmark. - If accuracy is unexpectedly low, something is wrong — do not proceed to profiling. @@ -108,8 +109,8 @@ for i in $(seq 1 120); do done # 3. Accuracy check -python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20 -# Expected: Accuracy > 0.8 +sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20 +# Expected: score > 80% # 4. Profile python3 -m sglang.test.send_one --profile diff --git a/.agents/skills/write-sglang-test/SKILL.md b/.agents/skills/write-sglang-test/SKILL.md index d0ddcdbe91..f4a6293aac 100644 --- a/.agents/skills/write-sglang-test/SKILL.md +++ b/.agents/skills/write-sglang-test/SKILL.md @@ -388,9 +388,9 @@ A `register_*_ci(...)` under `python/sglang/` is rejected by the ## Eval Accuracy Mixins -**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (run_eval, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length). +**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (the eval call, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length). -Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults. +Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults. `MMLUMixin` already scores through sgl-eval; `GSM8KMixin` still defaults to the deprecated `run_eval` backend for existing tests, so new tests set `gsm8k_backend = "sgl_eval"`. The exception is a base or completion-only model: sgl-eval only sends chat requests, so those stay on the 5-shot completion backend. ```python class TestMyFeature(CustomTestCase, MMLUMixin): diff --git a/.claude/rules/prefer-sgl-eval.md b/.claude/rules/prefer-sgl-eval.md new file mode 100644 index 0000000000..19993e87e0 --- /dev/null +++ b/.claude/rules/prefer-sgl-eval.md @@ -0,0 +1,31 @@ +# Accuracy evals: use sgl-eval + +Use [sgl-eval](https://github.com/sgl-project/sgl-eval) whenever you measure a +model's accuracy. + +- **Against a server:** run the `sgl-eval` CLI from its own virtualenv, not the + server's env, at the version pinned in the `test` extra of + `python/pyproject.toml`. + + ```bash + sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 200 + ``` + +- **In a test:** use `sglang.test.sgl_eval_utils.run_sgl_eval` or the kits + that wrap it. A new `GSM8KMixin` test sets `gsm8k_backend = "sgl_eval"`. + +- **Sampling:** use the model's recommended settings (temperature, top_p, + thinking, max tokens) unless the user says otherwise. sgl-eval overrides + them with its own defaults, so pass them explicitly, or use + `--load-preset-from-model-id` when the model has a preset. `--thinking` only + sets the `thinking` chat-template key; if the model's template reads a + different one (commonly `enable_thinking`), pass that key with + `--chat-template-kwarg` instead. When comparing against a CI test, use that + test's settings (printed in its log as `sgl-eval generation: {...}`). + +Do not use the deprecated `sglang.test.run_eval` or `few_shot_gsm8k`, `lm-eval`, +or a hand-written scorer. The exceptions are evals sgl-eval doesn't ship, and +base models, since sgl-eval only sends chat requests. + +Only compare scores from the same harness and settings, and say which ones you +used when you report a number. diff --git a/python/sglang/test/few_shot_gsm8k.py b/python/sglang/test/few_shot_gsm8k.py index e3631419b1..b7997fb7da 100644 --- a/python/sglang/test/few_shot_gsm8k.py +++ b/python/sglang/test/few_shot_gsm8k.py @@ -2,9 +2,8 @@ Run few-shot GSM-8K evaluation. .. deprecated:: - This module is deprecated. Use ``sglang.test.run_eval`` with - ``eval_name="gsm8k"`` instead, which routes through the unified - Chat API evaluation framework with dump_metric support. + This module is deprecated. Use ``sgl-eval run gsm8k`` or + ``sglang.test.sgl_eval_utils.run_sgl_eval`` instead. Usage: python3 -m sglang.test.few_shot_gsm8k --num-questions 200 @@ -58,7 +57,7 @@ def get_answer_value(answer_str): def run_eval(args): warnings.warn( "sglang.test.few_shot_gsm8k is deprecated. " - "Use sglang.test.run_eval with eval_name='gsm8k' instead.", + "Use `sgl-eval run gsm8k` or sglang.test.sgl_eval_utils.run_sgl_eval instead.", DeprecationWarning, stacklevel=2, )