[Docs] Add Qwen3.8 cookbook (#34587)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
197832bcf5
commit
8e7c07fae7
@@ -164,7 +164,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
|
||||
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. A `modes[]` entry may carry `flags` / `env` that only that role needs (`prefill`'s `--load-balance-method`, `decode`'s `--disaggregation-decode-polling-interval`, ...); they are emitted only while that role is selected, and a flag whose head the base cell already sets is replaced rather than duplicated. Put role-specific settings here, NOT in the cells — a cell carries one recipe, and the role is a Playground overlay. In `router.command` the ports MUST be `{{PREFILL_PORT}}` / `{{DECODE_PORT}}` / `{{ROUTER_PORT}}`: the engine substitutes them from its own `PD_PORTS` (prefill 30000, decode **30100**), so a literal port silently points the router at something the generated decode command never binds. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a companion block and retargets the cURL modal to `router.port`. |
|
||||
| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
|
||||
| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
|
||||
| `flagSelects` | A config-declared **list** of single-selects, each `{ id, title, stripPrefixes, options }` (option = `{ id, label, flags?, hide?, disable?, disableReason? }`); a flagless option is the "none"/accuracy-safe choice | A titled single-select that picks one value of a flag family the other axes don't model — e.g. KV-cache dtype (`--kv-cache-dtype`), mamba scheduler strategy (`--mamba-scheduler-strategy`). Generic: no engine change to add another. |
|
||||
|
||||
@@ -277,14 +277,18 @@ sgl-eval run gsm8k \\
|
||||
// `auto` is a sentinel (emits no --disaggregation-ib-device flag).
|
||||
ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
|
||||
// Router fronting prefill + decode; substitute <prefill-host>/<decode-host>.
|
||||
// ALWAYS use {{PREFILL_PORT}} / {{DECODE_PORT}} / {{ROUTER_PORT}} — the
|
||||
// engine fills them from its own PD_PORTS (prefill serves on 30000, decode
|
||||
// on 30100). A literal port here will not track the engine and sends the
|
||||
// router at a port the generated decode command does not listen on.
|
||||
router: {
|
||||
port: 8000,
|
||||
command:
|
||||
`python3 -m sglang_router.launch_router \\
|
||||
--pd-disaggregation \\
|
||||
--prefill http://<prefill-host>:30000 \\
|
||||
--decode http://<decode-host>:30001 \\
|
||||
--host 0.0.0.0 --port 8000 \\
|
||||
--prefill http://<prefill-host>:{{PREFILL_PORT}} \\
|
||||
--decode http://<decode-host>:{{DECODE_PORT}} \\
|
||||
--host 0.0.0.0 --port {{ROUTER_PORT}} \\
|
||||
--disable-circuit-breaker \\
|
||||
--health-check-interval-secs 999999`,
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user