[Docs] Add NVFP4 quantization to GLM-5.2 cookbook (#29380)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
5996b54bd3
commit
dd56a9f069
@@ -149,7 +149,8 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
|
||||
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||
a guessed release), `multiNodeHints` only for fabric-specific hw (e.g. gb200).
|
||||
a guessed release; key by `hw`, or `hw|quant` when one quant on a shared GPU needs its own
|
||||
image), `multiNodeHints` only for fabric-specific hw (e.g. gb200).
|
||||
|
||||
### Site-wiring (do all three)
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ the full contract):
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
|
||||
| `dockerImages` | `{[hwId]: string}` | Per-hw image name for `docker run` framing. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
|
||||
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `accuracyLabels` (below) + `benchmarkCommands.accuracy`. |
|
||||
|
||||
@@ -140,10 +140,12 @@ sgl-eval run gsm8k \\
|
||||
],
|
||||
},
|
||||
|
||||
// Per-hw image for `docker run` framing. ASK the user which sglang build the recipes ran
|
||||
// on; don't guess a supporting release. Default below is :dev (nightly) — replace the tag
|
||||
// with the user's release if they give one. NVIDIA share one image; AMD uses ROCm tags.
|
||||
// GB200/GB300/B300 may need a `-cu130` (CUDA 13) tag — confirm per release.
|
||||
// Image for `docker run` framing, keyed by `hw` (or `hw|quant`, resolved first, when one
|
||||
// quant on a shared GPU needs its own image — e.g. an FP4 dev build while FP8/BF16 use the
|
||||
// release tag). ASK the user which sglang build the recipes ran on; don't guess a supporting
|
||||
// release. Default below is :dev (nightly) — replace the tag with the user's release if they
|
||||
// give one. NVIDIA share one image; AMD uses ROCm tags. GB200/GB300/B300 may need a
|
||||
// `-cu130` (CUDA 13) tag — confirm per release.
|
||||
dockerImages: {
|
||||
h100: "lmsysorg/sglang:dev",
|
||||
h200: "lmsysorg/sglang:dev",
|
||||
|
||||
@@ -61,7 +61,9 @@ than restating.
|
||||
explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics
|
||||
in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal
|
||||
toggles.
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback).
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback); a
|
||||
`hw|quant` key (resolved before the plain `hw`) is valid when one quant on a shared GPU needs
|
||||
a different image (e.g. an FP4 dev build) — don't flag those.
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
|
||||
|
||||
Reference in New Issue
Block a user