docs: make playground issue template model field a free-form input (#27827)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
5fefe91289
commit
73d0989d9b
@@ -48,7 +48,7 @@ the full contract):
|
|||||||
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
||||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
|
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
|
||||||
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `ACCURACY_LABELS` + `benchmarkCommands.accuracy`. |
|
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `ACCURACY_LABELS` + `benchmarkCommands.accuracy`. |
|
||||||
| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the value that matches the `model` dropdown in your issue template so it's pre-selected when the issue opens. |
|
| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the model's HF id (`<hf-org>/<model-slug>`); it prefills the issue template's free-form `model` input when the issue opens. **Don't prune this block** — without it the engine falls back to `deepseek-ai/deepseek-v4` and submissions from your page get mislabeled. |
|
||||||
|
|
||||||
## 2.2 Author the 5-dim matrix (`cells[]`)
|
## 2.2 Author the 5-dim matrix (`cells[]`)
|
||||||
|
|
||||||
|
|||||||
@@ -142,8 +142,9 @@ sgl-eval run gsm8k \\
|
|||||||
mi355x: "lmsysorg/sglang:dev-rocm720-mi35x",
|
mi355x: "lmsysorg/sglang:dev-rocm720-mi35x",
|
||||||
},
|
},
|
||||||
|
|
||||||
// Pre-selects the issue template's `model` dropdown on "Submit verified cell".
|
// Prefills the issue template's free-form `model` field on "Submit verified cell".
|
||||||
// Must match that dropdown's value (usually `<hf-org>/<model-slug>`).
|
// Use the HF id (`<hf-org>/<model-slug>`). Do NOT delete this block when pruning —
|
||||||
|
// without it the engine falls back to "deepseek-ai/deepseek-v4" and mislabels submissions.
|
||||||
github: {
|
github: {
|
||||||
cookbookModel: "__HF_ORG__/__MODEL_SLUG__",
|
cookbookModel: "__HF_ORG__/__MODEL_SLUG__",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -51,7 +51,10 @@ than restating.
|
|||||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback).
|
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback).
|
||||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||||
- `github.cookbookModel` matches the issue-template `model` dropdown value.
|
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
|
||||||
|
template's `model` field is a free-form input prefilled from this value; if the config
|
||||||
|
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
|
||||||
|
page's submissions get mislabeled.
|
||||||
- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes
|
- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes
|
||||||
(the `moe` axis's MegaMoE backend option + `megamoeQuant` block only on Blackwell MoE,
|
(the `moe` axis's MegaMoE backend option + `megamoeQuant` block only on Blackwell MoE,
|
||||||
gated by `requiresHw`; `hisparse` only DSA-style; `pdDisagg.router` only with a PD topology).
|
gated by `requiresHw`; `hisparse` only DSA-style; `pdDisagg.router` only with a PD topology).
|
||||||
|
|||||||
@@ -21,13 +21,12 @@ body:
|
|||||||
recipe, they'll convert this issue into a PR against the cookbook cell
|
recipe, they'll convert this issue into a PR against the cookbook cell
|
||||||
catalog and close this issue.
|
catalog and close this issue.
|
||||||
|
|
||||||
- type: dropdown
|
- type: input
|
||||||
id: model
|
id: model
|
||||||
attributes:
|
attributes:
|
||||||
label: Cookbook model
|
label: Cookbook model
|
||||||
description: Which model's cookbook does this cell belong to?
|
description: Which model's cookbook does this cell belong to? Auto-filled by the playground — use the model id as it appears in the cookbook config.
|
||||||
options:
|
placeholder: "deepseek-ai/deepseek-v4"
|
||||||
- deepseek-ai/deepseek-v4
|
|
||||||
validations:
|
validations:
|
||||||
required: true
|
required: true
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user