diff --git a/docs/cards/logos/meta.png b/docs/cards/logos/meta.png new file mode 100644 index 000000000..e101baba7 Binary files /dev/null and b/docs/cards/logos/meta.png differ diff --git a/docs/cookbook/autoregressive/Llama/Llama3.1.mdx b/docs/cookbook/autoregressive/Meta/Llama3.1.mdx similarity index 100% rename from docs/cookbook/autoregressive/Llama/Llama3.1.mdx rename to docs/cookbook/autoregressive/Meta/Llama3.1.mdx diff --git a/docs/cookbook/autoregressive/Llama/Llama3.3-70B.mdx b/docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx similarity index 100% rename from docs/cookbook/autoregressive/Llama/Llama3.3-70B.mdx rename to docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx diff --git a/docs/cookbook/autoregressive/Llama/Llama4.mdx b/docs/cookbook/autoregressive/Meta/Llama4.mdx similarity index 100% rename from docs/cookbook/autoregressive/Llama/Llama4.mdx rename to docs/cookbook/autoregressive/Meta/Llama4.mdx diff --git a/docs/cookbook/autoregressive/Meta/MuseGlimmer.mdx b/docs/cookbook/autoregressive/Meta/MuseGlimmer.mdx new file mode 100644 index 000000000..45419d9d4 --- /dev/null +++ b/docs/cookbook/autoregressive/Meta/MuseGlimmer.mdx @@ -0,0 +1,205 @@ +--- +title: Muse Glimmer +description: "A multimodal reasoning model served from a BF16, NVFP4 + MXFP8, vendor GGUF, or MLX checkpoint." +tag: NEW +--- + +## Deployment + + + + + +See the [official SGLang installation guide](../../../docs/get-started/install) for all installation methods and hardware platforms. The steps below match the **Python** and **Docker** options in the command panel. + + + + + +```bash Command +pip install --upgrade pip +pip install uv +uv pip install sglang +``` + +Run the **Python** output of the command panel below in that environment. + + + + + +```bash Command +docker pull lmsysorg/sglang:dev-muse-glimmer +``` + +See [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker) to start the image. Replace the inner `sglang serve ...` command with the command from the panel below. + + + + + + + +Select a checkpoint format. Select whether to use speculative decoding: + +- **Standard**: Use normal autoregressive decoding. +- **DFlash**: Use speculative decoding with the DFlash draft model. The draft serves as published, with no conversion step. See [§2](#2-configuration-tips). + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/meta-models/muse-glimmer.jsx"; +import { benchmarks } from "/src/snippets/configs/meta-models/muse-glimmer-benchmarks.jsx"; + + + +## Playground + +Use the Playground to test SGLang features that are not in the verified matrix. The Deploy panel above shows only combinations that the SGLang team has verified. The Playground lets you add more options to the command from the Deploy panel. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +Muse Glimmer is a multimodal reasoning model. You can serve Muse Glimmer in four formats: + +- A BF16 checkpoint (`MuseGlimmerForConditionalGeneration`). +- A set of vendor GGUF files. +- A ready-to-serve NVFP4 + MXFP8 checkpoint. +- Three MLX repacks for Apple Silicon. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FormSourceNotes
BF16meta-models/Muse-Glimmer-30BSupports image input.
GGUF Q4_K_Mmeta-models/Muse-Glimmer-30B-GGUFText only. This path is not optimized. SGLang shows a warning at startup.
NVFP4RadixArk/Muse-Glimmer-NVFP4Text only. Ready to serve, no conversion needed.
MLX Q4RadixArk/Muse-Glimmer-q4-MLXText only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.
MLX Q4_K_M (gs128)RadixArk/Muse-Glimmer-q4km-gs128-MLXText only. Apple Silicon (MLX backend). Carries the vendor GGUF's exact quantization codes in MLX format. The measured MLX artifact. See §3.4.
MLX Q4_K (dynamic)RadixArk/Muse-Glimmer-q4k-dynamic-MLXText only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.
+ +**Resources:** [Muse-Glimmer-30B (BF16)](https://huggingface.co/meta-models/Muse-Glimmer-30B) · [Muse-Glimmer-30B-assistant (DFlash draft)](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) · [Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) · [Muse-Glimmer-NVFP4](https://huggingface.co/RadixArk/Muse-Glimmer-NVFP4) · MLX · [q4](https://huggingface.co/RadixArk/Muse-Glimmer-q4-MLX) · [q4km-gs128](https://huggingface.co/RadixArk/Muse-Glimmer-q4km-gs128-MLX) · [q4k-dynamic](https://huggingface.co/RadixArk/Muse-Glimmer-q4k-dynamic-MLX). + +## 2. Configuration Tips + +**The GGUF format is text only.** SGLang has no `mmproj` path. You cannot use the vision GGUF files. Use the BF16 checkpoint for multimodal input. + +**The NVFP4 checkpoint.** `RadixArk/Muse-Glimmer-NVFP4` is a ready-to-serve NVFP4 + MXFP8 checkpoint. No conversion needed — point `--model-path` straight at it. + +**The DFlash draft.** `meta-models/Muse-Glimmer-30B-assistant` is the vendor's native draft export and serves directly. No conversion needed. + +**DFlash with a GGUF target model** needs `--speculative-draft-load-format auto`. Without this flag, the draft model uses the `gguf` load format from the target model. The loader then rejects the draft directory. + +**Apple Silicon uses an MLX checkpoint, not the GGUF files.** The MLX backend has no GGUF path. Serve one of the three `RadixArk/Muse-Glimmer-*-MLX` artifacts with `SGLANG_USE_MLX=1` (see the Apple Silicon cells in the command panel). All three take the same flags; `q4km-gs128` is the one with a measured round. Keep `--disable-radix-cache` — the windowed KV storage for the sliding-window layers requires it — and set `SGLANG_MLX_CACHE_LIMIT_GB=8` so the MLX buffer cache does not grow the footprint under concurrent load. Speculative decoding is not available on the MLX backend. + +## 3. Advanced Usage + +### 3.1 Reasoning + +Muse Glimmer enables the `muse` reasoning parser by default. This parser separates the reasoning text from the final answer. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +resp = client.chat.completions.create( + model="meta-models/Muse-Glimmer-30B", + messages=[{"role": "user", "content": "What is 15% of 240?"}], +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Answer:", msg.content) +``` + + + +### 3.2 Tool Calling + +Muse Glimmer enables the `muse` tool-call parser by default. This parser sends structured tool calls in `message.tool_calls`. + +### 3.3 Multimodal + +The BF16 checkpoint supports image input. It defaults to text only. To switch, select **Modality** in the command panel above. + +**Text only** adds `--language-model-only`. This flag turns off the vision tower. SGLang does not build or load the vision weights. This frees memory for the KV cache. SGLang rejects image requests in this mode. + +Select **Image + text** to turn on image input. + +NVFP4, GGUF, and the MLX artifacts are text only. The Modality option does not appear for GGUF or MLX; NVFP4 only offers **Text only**. + +### 3.4 Apple Silicon (MLX) + +The MLX backend serves three Muse Glimmer artifacts on Apple Silicon Macs (48 GB unified memory or more). All three are text only — the MLX backend has no vision path — and all three take the same flags, so pick one in the command panel: + +- `RadixArk/Muse-Glimmer-q4-MLX` — no measured round yet. +- `RadixArk/Muse-Glimmer-q4km-gs128-MLX` — a lossless repack of the vendor's Q4_K_M (gs128) GGUF: every weight keeps the GGUF's exact quantization code, with the group scales re-expressed in MLX affine bf16 (≤2⁻⁸ relative rounding). The numbers below are for this artifact. +- `RadixArk/Muse-Glimmer-q4k-dynamic-MLX` — no measured round yet. + +Choose along the speed-versus-accuracy axis: footprint and expected accuracy both grow `q4` → `q4km-gs128` → `q4k-dynamic`, and decode speed moves the other way. Decode on Apple Silicon is memory-bandwidth-bound, so a smaller artifact reads fewer weight bytes per token — more tokens per second, and more unified memory left over for the KV cache. Take `q4` for the fastest responses on the smallest machine, `q4k-dynamic` to stay closest to BF16, and `q4km-gs128` for the middle ground — it is also the only one of the three with a measured round, below. + +This table shows accuracy for the gs128 checkpoint, with the vendor llama.cpp fork serving the source GGUF on the same machine as the reference. GSM8K: 200 questions, no-thinking chat template, temperature 0, max 2048 new tokens. CIMemories: 1 profile, full combo, single trial, DeepSeek-R1-0528 judge. + + + + + + + + + + + + + + + + + + + + + + + + + + +
BenchmarkSGLang MLXllama.cpp (same GGUF)
GSM8K (200q, no-thinking, greedy)0.9700.970
CIMemories — violation rate (lower is better)0.00%8.27%
CIMemories — coverage (higher is better)76.0%68.4%
+ +CIMemories is a single-trial benchmark with a nondeterministic judge; treat the SGLang-vs-llama.cpp gap on that row as run noise, not a runtime effect. GSM8K parity is exact. + +Decode throughput for gs128 on an M5 Pro (64 GB), 1k-in/1k-out greedy: 15.3 tok/s at batch 1, rising to 52.6 tok/s aggregate at batch 8 — ahead of llama.cpp on the same GGUF codes at every batch size above 1. diff --git a/docs/cookbook/autoregressive/intro.mdx b/docs/cookbook/autoregressive/intro.mdx index 2dd2fb3b9..e584961ad 100644 --- a/docs/cookbook/autoregressive/intro.mdx +++ b/docs/cookbook/autoregressive/intro.mdx @@ -44,10 +44,10 @@ metatags: img="/cards/logos/deepreinforce.png" /> **Llama** (2, 3.x, 4 series) `meta-llama/Llama-4-Scout-17B-16E-Instruct` - Meta's open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance. SGLang provides Llama-4 model-specific optimizations + Meta's open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance. SGLang provides Llama-4 model-specific optimizations **Mistral** (Mixtral, NeMo, Small3) diff --git a/docs/scripts/gen_redirects.py b/docs/scripts/gen_redirects.py index 249a880fb..48aefc347 100755 --- a/docs/scripts/gen_redirects.py +++ b/docs/scripts/gen_redirects.py @@ -102,7 +102,7 @@ EXPLICIT = { "/basic_usage/glm45": "/cookbook/autoregressive/GLM/GLM-4.5", "/basic_usage/glmv": "/cookbook/autoregressive/GLM/GLM-4.6V", "/basic_usage/gpt_oss": "/cookbook/autoregressive/OpenAI/GPT-OSS", - "/basic_usage/llama4": "/cookbook/autoregressive/Llama/Llama4", + "/basic_usage/llama4": "/cookbook/autoregressive/Meta/Llama4", "/basic_usage/minimax_m2": "/cookbook/autoregressive/MiniMax/MiniMax-M2", "/basic_usage/popular_model_usage": "/cookbook/autoregressive/intro", "/basic_usage/qwen3": "/cookbook/autoregressive/Qwen/Qwen3", diff --git a/docs/src/snippets/configs/meta-models/muse-glimmer-benchmarks.jsx b/docs/src/snippets/configs/meta-models/muse-glimmer-benchmarks.jsx new file mode 100644 index 000000000..b6396a165 --- /dev/null +++ b/docs/src/snippets/configs/meta-models/muse-glimmer-benchmarks.jsx @@ -0,0 +1,23 @@ +export const benchmarks = [ + { match: { hw: "rtx5090", variant: "default", quant: "gguf", strategy: "standard", nodes: "single" } }, + { match: { hw: "rtx5090", variant: "default", quant: "gguf", strategy: "dflash", nodes: "single" } }, + { match: { hw: "rtx5090", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" } }, + { match: { hw: "rtx5090", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" } }, + + { match: { hw: "rtx6000", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" } }, + { match: { hw: "rtx6000", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" } }, + { match: { hw: "rtx6000", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" } }, + { match: { hw: "rtx6000", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" } }, + + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" } }, + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" } }, + + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" } }, + + { match: { hw: "mac", variant: "default", quant: "mlx-q4", strategy: "standard", nodes: "single" } }, + { match: { hw: "mac", variant: "default", quant: "mlx-q4km", strategy: "standard", nodes: "single" } }, + { match: { hw: "mac", variant: "default", quant: "mlx-q4k-dyn", strategy: "standard", nodes: "single" } }, +]; diff --git a/docs/src/snippets/configs/meta-models/muse-glimmer.jsx b/docs/src/snippets/configs/meta-models/muse-glimmer.jsx new file mode 100644 index 000000000..27e77548b --- /dev/null +++ b/docs/src/snippets/configs/meta-models/muse-glimmer.jsx @@ -0,0 +1,522 @@ +export const config = { + modelName: "Muse Glimmer", + + supportedHardware: ["b200", "h200", "rtx5090", "rtx6000", "dgx-spark", "mac"], + + hardware: [ + // RTX 5090 and RTX PRO 6000 are Blackwell-generation but not in the shared + // HARDWARE_CATALOG (consumer/workstation cards, not the datacenter SKUs + // that catalog covers), so they need a local vendor override here. + { id: "rtx5090", label: "RTX 5090", vram: "32GB", vendor: "blackwell" }, + { id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "blackwell" }, + // dgx-spark is NOT listed here -- it's already in the shared + // HARDWARE_CATALOG under blackwell (with its multi-node docker flags), + // so this model just inherits that entry. + // Apple Silicon Mac (MLX backend, unified memory). Benchmarked on an + // M5 Pro 64GB; the q4km-gs128 artifact fits a 48GB machine. + { id: "mac", label: "Apple Silicon", vram: "48GB+", vendor: "apple" }, + ], + + variants: [{ id: "default", label: "Default" }], + + quantizations: [ + { id: "bf16", label: "BF16" }, + { id: "gguf", label: "GGUF Q4_K_M" }, + { id: "nvfp4", label: "NVFP4" }, + // Three MLX artifacts, all Apple-Silicon-only. gs128 is the one with a + // measured GSM8K / CIMemories round (see the cookbook §3.4 table); the + // other two serve with the same recipe but are not benchmarked yet. + { id: "mlx-q4", label: "MLX Q4" }, + { id: "mlx-q4km", label: "MLX Q4_K_M (gs128)" }, + { id: "mlx-q4k-dyn", label: "MLX Q4_K (dynamic)" }, + ], + + // No Docker path on Apple Silicon — the MLX backend runs native-only. + runModes: (s) => (s.hw === "mac" ? ["python"] : ["python", "docker"]), + + strategies: [ + { id: "standard", label: "Standard" }, + { id: "dflash", label: "DFlash" }, + ], + + nodesOptions: [{ id: "single", label: "Single Node" }], + + overlayDims: [ + { + id: "modality", + title: "Modality", + default: "text", + // GGUF and every MLX artifact are text-only; no modality choice there. + showWhen: (s) => s.quant !== "gguf" && !(s.quant || "").startsWith("mlx-"), + options: [ + // NVFP4 ships with no vision weights despite config.json declaring + // vision_config, so "Image + text" only shows for bf16. + { id: "mm", label: "Image + text", showWhen: (s) => s.quant === "bf16" }, + { + id: "text", + label: "Text only", + flags: ["--language-model-only"], + }, + ], + }, + ], + + modelNames: { + "default|bf16": "meta-models/Muse-Glimmer-30B", + "default|gguf": "meta-models/Muse-Glimmer-30B-GGUF/muse-glimmer-30B-kquant-17gb.gguf", + "default|nvfp4": "RadixArk/Muse-Glimmer-NVFP4", + "default|mlx-q4": "RadixArk/Muse-Glimmer-q4-MLX", + "default|mlx-q4km": "RadixArk/Muse-Glimmer-q4km-gs128-MLX", + "default|mlx-q4k-dyn": "RadixArk/Muse-Glimmer-q4k-dynamic-MLX", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + DRAFT_PATH: { target: "command", label: "DFlash draft checkpoint", default: "meta-models/Muse-Glimmer-30B-assistant" }, + QUANT_PATH: { target: "command", label: "NVFP4 checkpoint", default: "RadixArk/Muse-Glimmer-NVFP4" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --flush-cache`, + numPromptsByConc: { 1: 8, 16: 32, 64: 128, 256: 512, 1024: 2048, 4096: 4096 }, + }, + + // No `accuracy` template and no `accuracyLabels`: no benchmarks entry carries + // accuracy data, so the Reproduce modal renders speed only. Re-add both + // together (matching keys) when an eval round lands. + + dockerImages: { + b200: "lmsysorg/sglang:dev-muse-glimmer", + h200: "lmsysorg/sglang:dev-muse-glimmer", + rtx5090: "lmsysorg/sglang:dev-muse-glimmer", + rtx6000: "lmsysorg/sglang:dev-muse-glimmer", + // GB10 needs an aarch64 manifest under this tag to resolve natively + "dgx-spark": "lmsysorg/sglang:dev-muse-glimmer", + }, + + github: { + cookbookModel: "meta-models/Muse-Glimmer-30B", + }, + + playgroundFeatures: { + attention: { + knobs: [ + { id: "tp", label: "TP", values: [null, 1, 2] }, + ], + }, + + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off" }, + { id: "dflash", label: "DFlash", + flags: ["--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5"] }, + ], + }, + }, + + cells: [ + { + match: { hw: "rtx5090", variant: "default", quant: "gguf", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "rtx5090", variant: "default", quant: "gguf", strategy: "dflash", nodes: "single" }, + // Measured on a single RTX 5090 (PR #8, config B): AIME pass@1 93.75% + // (SEM +/-0.42, maj@8 93.33%, pass@8 96.67%, 0% truncation) and GSM8K + // 97.12%; KV pool 87099 vs 241189 for GGUF standard, since the draft + // takes its share. --speculative-draft-load-format auto is what makes a + // GGUF *target* work here: without it the draft inherits the target's + // gguf load format and the loader rejects the draft dir. + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-draft-load-format auto", + "--speculative-dflash-block-size 5", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + { + match: { hw: "rtx5090", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.9", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // 0.9 confirmed on real hardware (single RTX 5090, 32GB): boots and + // survives batch sizes 1/4/8 at 512in/256out with no OOM. Margin is + // tight though -- ~1.1GB free after CUDA graph capture -- so this is + // the ceiling, not a value with headroom to spare. + // + // --kv-cache-dtype fp8_e4m3 confirmed on the same hardware: doubles + // max_total_num_tokens (57462 -> 114925) with no meaningful accuracy + // cost -- GSM8K (200q, real sgl-eval) scored 0.905 fp8 vs 0.885 bf16, + // a gap well within normal run-to-run noise at this sample size. + // + // --speculative-draft-model-quantization fp8 dynamically quantizes the + // draft's bf16 linears to fp8 at load time (draft weights 4.83GB -> + // 3.03GB) and, on this SM120 GPU, disables DFlash's fused KV + // materialization fast path (quantized qkv_proj isn't supported there), + // which changes the KV sizing math enough to push max_total_num_tokens + // 114925 -> 180193. Confirmed on the same hardware: accept length and + // decode speed are statistically identical to the unquantized draft + // across batch sizes 1/4/8 and 5 varied prompts (radix cache flushed + // between runs) -- no measurable regression. + { + match: { hw: "rtx5090", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--speculative-draft-model-quantization fp8", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.9", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + { + match: { hw: "rtx6000", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "rtx6000", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "rtx6000", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // --kv-cache-dtype fp8_e4m3 directly confirmed on RTX 5090 (doubles + // max_total_num_tokens, no meaningful accuracy cost -- see the rtx5090 + // nvfp4 cell above); not independently re-benchmarked on this SKU, but + // the mechanism (halving KV cache bytes/token) is hardware-independent. + // + // --speculative-draft-model-quantization fp8 directly confirmed on + // RTX 5090 (no accept-length or speed regression, more KV cache + // headroom -- see the rtx5090 nvfp4 cell above); not independently + // re-benchmarked on this SKU, but the mechanism (dynamically quantizing + // the draft's bf16 linears to fp8 at load time) is hardware-independent. + { + match: { hw: "rtx6000", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--speculative-draft-model-quantization fp8", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // GB10: 128GB unified CPU/GPU memory -- the mem-fraction values in the + // dgx-spark cells below are deliberately lower than the discrete-GPU + // cells (the OS and client share the same pool; 0.85-style fractions + // OOM the box during load). + { + match: { hw: "dgx-spark", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--mem-fraction-static 0.75", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "dgx-spark", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--mem-fraction-static 0.65", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.40", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // --kv-cache-dtype fp8_e4m3 directly confirmed on RTX 5090 (doubles + // max_total_num_tokens, no meaningful accuracy cost -- see the rtx5090 + // nvfp4 cell above); not independently re-benchmarked on this SKU, but + // the mechanism (halving KV cache bytes/token) is hardware-independent. + // + // --speculative-draft-model-quantization fp8 directly confirmed on + // RTX 5090 (no accept-length or speed regression, more KV cache + // headroom -- see the rtx5090 nvfp4 cell above); not independently + // re-benchmarked on this SKU, but the mechanism (dynamically quantizing + // the draft's bf16 linears to fp8 at load time) is hardware-independent. + { + match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--speculative-draft-model-quantization fp8", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.38", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + { + match: { hw: "mac", variant: "default", quant: "mlx-q4", strategy: "standard", nodes: "single" }, + verified: false, + env: ["SGLANG_USE_MLX=1", "SGLANG_MLX_CACHE_LIMIT_GB=8"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--disable-radix-cache", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mac", variant: "default", quant: "mlx-q4km", strategy: "standard", nodes: "single" }, + verified: true, + env: ["SGLANG_USE_MLX=1", "SGLANG_MLX_CACHE_LIMIT_GB=8"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--disable-radix-cache", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mac", variant: "default", quant: "mlx-q4k-dyn", strategy: "standard", nodes: "single" }, + verified: false, + env: ["SGLANG_USE_MLX=1", "SGLANG_MLX_CACHE_LIMIT_GB=8"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--disable-radix-cache", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "standard", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // --kv-cache-dtype fp8_e4m3 directly confirmed on RTX 5090 (doubles + // max_total_num_tokens, no meaningful accuracy cost -- see the rtx5090 + // nvfp4 cell above); not independently re-benchmarked on this SKU, but + // the mechanism (halving KV cache bytes/token) is hardware-independent. + // + // --speculative-draft-model-quantization fp8 directly confirmed on + // RTX 5090 (no accept-length or speed regression, more KV cache + // headroom -- see the rtx5090 nvfp4 cell above); not independently + // re-benchmarked on this SKU, but the mechanism (dynamically quantizing + // the draft's bf16 linears to fp8 at load time) is hardware-independent. + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "dflash", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{QUANT_PATH}}", + "--reasoning-parser muse", + "--tool-call-parser muse", + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path {{DRAFT_PATH}}", + "--speculative-dflash-block-size 5", + "--speculative-draft-model-quantization fp8", + "--kv-cache-dtype fp8_e4m3", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};