diff --git a/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx new file mode 100644 index 000000000..65ba1fea6 --- /dev/null +++ b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx @@ -0,0 +1,206 @@ +--- +title: Ling-3.0-flash +description: "Deploy Ling-3.0-flash with SGLang — 124B total / 5.1B active hybrid KDA + MLA MoE in BF16 or FP8, with thinking mode, Ling3 parsers, and NEXTN speculative decoding." +tag: NEW +--- + +## Deployment + + + + + +```bash Command +docker pull lmsysorg/sglang:dev-Ling-3.0-flash +``` + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + +Pick your hardware + recipe to generate the launch command. Three serving strategies are covered: + +- **Low-Latency** — fastest reply for a single user. Pick for chat. These recipes run NEXTN speculative decoding. +- **High-Throughput** — most tokens per second across many users. Best for batch jobs. These recipes turn speculative decoding off, since at saturation the draft/verify overhead outweighs the speedup. +- **HiCache + Mooncake** — writes reusable prefixes to Mooncake L3 storage. Start the Mooncake services in §3.3 before launching the generated server command. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/inclusionAI/ling-3.0-flash.jsx"; +import { benchmarks } from "/src/snippets/configs/inclusionAI/ling-3.0-flash-benchmarks.jsx"; + + + +## Playground + +The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +Ling-3.0-flash is a hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family. It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network. This keeps per-token inference cost close to a small model — **124B total parameters with only 5.1B active** — while retaining large-model capacity. + +It is a hybrid-reasoning model with thinking enabled by default, and it supports structured tool calling. Native context length is 128K, extendable to 256K with YaRN. + +**Available Models:** + +- **BF16**: [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) — 124B total / 5.1B active +- **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-flash-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8) + +**License:** MIT + +**Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash). + +## 2. Configuration Tips + +- BF16 tensor parallelism follows the GPU: `--tp 4` on 141 GB-class cards (H20-3e, H200) and 4-GPU Blackwell nodes (B200, GB300); `--tp 8` on 80 GB cards (H100, H800). +- The FP8 recipes pair `--tp` with a matching `--ep-size` (`--tp 4 --ep-size 4` on 4-GPU nodes, `--tp 8 --ep-size 8` on H100/H800). The checkpoint uses blockwise (128×128) E4M3 expert weights, so a pure tensor-parallel shard must satisfy `(768 / TP) % 128 == 0` — only TP2 qualifies; expert parallelism splits experts whole instead of by column, which lifts that restriction and uses the full node. SGLang detects the quantization format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed. +- `--reasoning-parser ling3` and `--tool-call-parser ling3` enable Ling-3.0-specific reasoning and structured tool-call parsing; toggle them in the **Parsers** card of the [Playground](#playground). +- Both the chat template and the `ling3` reasoning parser default to thinking on. A single request can turn it off with `"chat_template_kwargs": {"enable_thinking": false}` (see §3.1). +- `--mem-fraction-static 0.8` reserves headroom for CUDA graphs and concurrent decoding; with the default allocation the NEXTN recipes can OOM under concurrent requests (e.g. a 32-thread GSM8K run). +- The checkpoint ships a built-in MTP layer (`num_nextn_predict_layers: 1`); enable it with `--speculative-algorithm NEXTN` — no separate draft model is needed. The Low-Latency recipes have it on; toggle it in the **Speculative Decoding** card of the [Playground](#playground). +- Native context is 128K. The recipes set `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1` to acknowledge the longer context explicitly, then use `--context-length 262144` and YaRN with factor 2.0 to extend it to 256K. +- The **HiCache** card in the [Playground](#playground) exposes the validated Mooncake L3 path. It adds the hybrid-KDA scheduler and prefix-key settings together; see §3.3 for the required services. + +## 3. Advanced Usage + +### 3.1 Reasoning + +Ling-3.0-flash thinks by default. With `--reasoning-parser ling3` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`: + + + +```bash Command +curl -s http://localhost:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "inclusionAI/Ling-3.0-flash", + "messages": [{"role": "user", "content": "What is 15% of 240?"}] + }' +``` + + + + + +```json Output +{ + "choices": [ + { + "message": { + "role": "assistant", + "content": "15% of 240 = **36**", + "reasoning_content": "The user is asking a simple percentage calculation: 15% of 240. This is straightforward: 0.15 × 240 = 36.", + "tool_calls": null + }, + "finish_reason": "stop" + } + ] +} +``` + + + + +Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. + + +### 3.2 Tool Calling + +With `--tool-call-parser ling3` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`: + + + +```bash Command +curl -s http://localhost:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "inclusionAI/Ling-3.0-flash", + "messages": [{"role": "user", "content": "Search for the latest news about AI"}], + "tools": [{ + "type": "function", + "function": { + "name": "search", + "description": "Search for information on the internet", + "parameters": { + "type": "object", + "properties": { + "query": {"type": "string", "description": "The search query"} + }, + "required": ["query"] + } + } + }], + "tool_choice": "auto" + }' +``` + + + + + +```json Output +{ + "choices": [ + { + "message": { + "role": "assistant", + "content": "", + "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool with a query about the latest AI news.", + "tool_calls": [ + { + "id": "call_0822dd418aa34254aa5b19e1", + "index": 0, + "type": "function", + "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI 2025\"}" } + } + ] + }, + "finish_reason": "tool_calls" + } + ] +} +``` + + + +For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request). + +### 3.3 HiCache with Mooncake + +The HiCache recipes use Mooncake as L3 prefix storage. Start the metadata server, master, and storage client before you launch SGLang. The following command runs all three services from the same image in a separate container: + +```bash Command +docker run --rm --network host --ipc=host \ + lmsysorg/sglang:dev-Ling-3.0-flash \ + bash -lc ' + python3 -m mooncake.http_metadata_server --port 8290 & + mooncake_master --port 50171 --metrics_port 9024 & + exec mooncake_client \ + --host=127.0.0.1 \ + --port=50172 \ + --master_server_address=127.0.0.1:50171 \ + --metadata_server=http://127.0.0.1:8290/metadata \ + --protocol=tcp \ + --device_names= \ + --global_segment_size=4294967296 \ + --enable_http_server=true \ + --http_port=8291 + ' +``` + +Then select **HiCache + Mooncake** in Deployment, or enable **HiCache** in the Playground. Docker commands use host networking so the SGLang container can reach these localhost services. You can change the master and metadata endpoints in the **Env** dialog. + +This validated setup uses TCP, so `MOONCAKE_DEVICE=` and the client's `--device_names=` are intentionally empty. Set both to your actual device list only when you configure an RDMA deployment. + + +With the default `chunked_prefill_size` of 8192, a cold request writes through only when its uncached extend length fits in one chunk. A longer cold first request skips write-through for that influx; a repeat with the same prefix can hit the device radix cache and proceed normally. + + + +For this hybrid KDA model, use the Mooncake L3 recipe shown here. Host-memory L2 eviction is not exposed because the KDA cache path is not currently compatible with it. Track the limitation in [issue #33713](https://github.com/sgl-project/sglang/issues/33713). + + +For storage sizing and backend details, see [HiCache best practices](/docs/advanced_features/hicache_best_practices). diff --git a/docs/cookbook/autoregressive/InclusionAI/Ring-2.6-1T.mdx b/docs/cookbook/autoregressive/InclusionAI/Ring-2.6-1T.mdx index 87e270455..42a9eff2c 100644 --- a/docs/cookbook/autoregressive/InclusionAI/Ring-2.6-1T.mdx +++ b/docs/cookbook/autoregressive/InclusionAI/Ring-2.6-1T.mdx @@ -2,7 +2,6 @@ title: Ring-2.6-1T metatags: description: "Deploy Ring-2.6-1T with SGLang - a trillion-parameter InclusionAI reasoning model for agent workflows, high/xhigh reasoning effort, and tool use." -tag: NEW --- ## 1. Model Introduction diff --git a/docs/cookbook/autoregressive/intro.mdx b/docs/cookbook/autoregressive/intro.mdx index 4bd913b49..2dd2fb3b9 100644 --- a/docs/cookbook/autoregressive/intro.mdx +++ b/docs/cookbook/autoregressive/intro.mdx @@ -106,7 +106,7 @@ metatags: boolean` // dockerMounts optional — additional `-v` mount specs // dockerRunCommand optional — command placed after the image and before // generated server flags; string or `(selection) => string` @@ -100,6 +101,8 @@ export const Deployment = ({ config, benchmarks }) => { hopper: [ { id: "h200", label: "H200", vram: "141GB" }, { id: "h100", label: "H100", vram: "80GB" }, + { id: "h20-3e", label: "H20-3e", vram: "141GB" }, + { id: "h800", label: "H800", vram: "80GB" }, ], amd: [ { id: "mi300x", label: "MI300X", vram: "192GB" }, @@ -726,6 +729,8 @@ export const Deployment = ({ config, benchmarks }) => { : (config.dockerRunCommand || "sglang serve"); const portFlag = flags.find((x) => x.split(/[\s=]/)[0] === "--port"); const servePort = portFlag ? portFlag.slice("--port".length).trim() : "{{PORT}}"; + const hostNetwork = multinode || (typeof config.dockerHostNetworkWhen === "function" + && config.dockerHostNetworkWhen(sel, { flags, env: cellEnv })); const vendorOf = (hwId) => { for (const [vendor, list] of Object.entries(HARDWARE_CATALOG)) { if (list.some((h) => h.id === hwId)) return vendor; @@ -760,7 +765,7 @@ export const Deployment = ({ config, benchmarks }) => { // Multi-node needs host networking so the cross-node rendezvous port // (--dist-init-addr) and NCCL/GLOO traffic are reachable; single-node // just maps the serve port. - multinode ? " --network host" : ` -p ${servePort}:${servePort}`, + hostNetwork ? " --network host" : ` -p ${servePort}:${servePort}`, ...(multinode ? fabricFlagsOf(sel.hw).map((f) => " " + f) : []), " -v ~/.cache/huggingface:/root/.cache/huggingface", ...(config.dockerMounts || []).map((mount) => ` -v ${mount}`), diff --git a/docs/src/snippets/_playground.jsx b/docs/src/snippets/_playground.jsx index e82597719..45f1a5c41 100644 --- a/docs/src/snippets/_playground.jsx +++ b/docs/src/snippets/_playground.jsx @@ -1061,12 +1061,21 @@ export const Playground = ({ config }) => { }, // ---- Axis: Hierarchical KV Cache ---------------------------------------- - // Enable + optional backend + write policy. Owns the `--hicache-*` family - // (unconditional strip). + // Enable + optional backend + write policy. `null` inherits the base cell, + // so a verified HiCache recipe remains byte-identical until it is changed. hicache: { - initState: () => ({ enable: false, backend: null, writePolicy: "auto" }), + initState: () => ({ enable: null, backend: null, writePolicy: "auto" }), - apply: ({ flags, env, value, fc, sel, h }) => { + deriveFromBase: (cell, fc, h) => { + const flags = (cell && cell.flags) || []; + return { + enable: h.hasFlag(flags, "--enable-hierarchical-cache"), + backend: h.findFlagArg(flags, "--hicache-storage-backend"), + writePolicy: h.findFlagArg(flags, "--hicache-write-policy") || "auto", + }; + }, + + apply: ({ flags, env, value, fc, sel, h, derived }) => { if (fc.excludesHw && sel && fc.excludesHw.includes(sel.hw)) return { flags, env }; // When the Deploy panel owns enablement (`showWhen`), the base already // carries a complete, verified hicache recipe. Rebuilding it from this @@ -1083,12 +1092,34 @@ export const Playground = ({ config }) => { } return { flags, env }; } - flags = h.stripFlagsByFirstToken(flags, [ + + const hasOverride = value.enable !== null + || value.backend !== null + || (value.writePolicy && value.writePolicy !== "auto"); + if (!hasOverride) return { flags, env }; + + const backendOptions = fc.backends || []; + const ownedHeads = [ "--enable-hierarchical-cache", "--hicache-ratio", "--hicache-size", "--hicache-write-policy", "--hicache-mem-layout", "--hicache-io-backend", "--hicache-storage-backend", "--hicache-storage-prefetch-policy", - ]); - if (value.enable) { + "--hicache-storage-backend-extra-config", + ...((fc.requiredFlags || []).map((f) => f.split(/\s/)[0])), + ...backendOptions.flatMap((o) => (o.flags || []).map((f) => f.split(/\s/)[0])), + ]; + const ownedEnvKeys = [ + ...(fc.requiredEnv || []), + ...backendOptions.flatMap((o) => o.env || []), + ].map((e) => e.split("=")[0]); + flags = h.stripFlagsByFirstToken(flags, ownedHeads); + if (ownedEnvKeys.length) env = h.stripEnvByPrefix(env, ownedEnvKeys); + + const enabled = value.enable !== null + ? value.enable : !!(derived && derived.enable); + const backend = value.backend !== null + ? value.backend + : ((derived && derived.backend) || fc.defaultBackend || null); + if (enabled) { const isAmd = sel && /^mi\d/.test(sel.hw); const pdMode = h.findFlagArg(flags, "--disaggregation-mode") || "off"; const pdBackend = h.findFlagArg(flags, "--disaggregation-transfer-backend"); @@ -1113,7 +1144,7 @@ export const Playground = ({ config }) => { if (useAmdIo) { adds.push(`--hicache-mem-layout ${amdIo.memLayout}`, `--hicache-io-backend ${amdIo.ioBackend}`); - } else if (value.backend) { + } else if (backend) { adds.push("--hicache-mem-layout page_first_direct", "--hicache-io-backend direct"); } @@ -1121,43 +1152,58 @@ export const Playground = ({ config }) => { ? value.writePolicy : ((amdIo && amdIo.writePolicy) || "write_through"); adds.push(`--hicache-write-policy ${writePolicy}`); // When amdStorageFileOnly is set, AMD emits storage flags only for "file". - if ((isAmd && fc.amdStorageFileOnly) ? value.backend === "file" : !!value.backend) { - adds.push(`--hicache-storage-backend ${value.backend}`, + if ((isAmd && fc.amdStorageFileOnly) ? backend === "file" : !!backend) { + adds.push(`--hicache-storage-backend ${backend}`, `--hicache-storage-prefetch-policy ${(amdIo && amdIo.prefetchPolicy) || "wait_complete"}`); } else if (amdIo && amdIo.prefetchPolicy) { adds.push(`--hicache-storage-prefetch-policy ${amdIo.prefetchPolicy}`); } + const backendOption = backendOptions.find((o) => o.id === backend); + adds.push(...(backendOption?.flags || []), ...(fc.requiredFlags || [])); flags = h.insertBeforeTail(flags, adds); + env = [ + ...env, + ...(backendOption?.env || []), + ...(fc.requiredEnv || []), + ]; } return { flags, env }; }, - render: ({ axisId, value, setValue, fc, base, s, renderChip, renderSelect }) => { + render: ({ axisId, value, setValue, fc, base, s, renderChip, renderSelect, derived }) => { if (fc.excludesHw && fc.excludesHw.includes(base.hw)) return null; const setSlot = (k, v) => setValue({ ...value, [k]: v }); const hasBackends = (fc.backends || []).length > 0; const hasPolicies = (fc.writePolicies || []).length > 0; + const enabled = value.enable !== null + ? value.enable : !!(derived && derived.enable); + const hasAutoBackend = (fc.backends || []).some((o) => o.id === null); + const backend = value.backend !== null + ? value.backend + : (hasAutoBackend ? null : ((derived && derived.backend) || fc.defaultBackend || null)); + const writePolicy = value.writePolicy !== "auto" + ? value.writePolicy : ((derived && derived.writePolicy) || "auto"); return (
HiCache {typeof fc.showWhen !== "function" && ( - {renderChip("Enable", value.enable, true, - () => setSlot("enable", !value.enable))} + {renderChip("Enable", enabled, true, + () => setSlot("enable", !enabled))} )} {hasBackends && ( Storage - {renderSelect(value.backend, fc.backends, + {renderSelect(backend, fc.backends, (v) => setSlot("backend", v), base)} )} {hasPolicies && ( Write - {renderSelect(value.writePolicy, fc.writePolicies, + {renderSelect(writePolicy, fc.writePolicies, (v) => setSlot("writePolicy", v), base)} )} @@ -1419,6 +1465,8 @@ export const Playground = ({ config }) => { : (config.dockerRunCommand || "sglang serve"); const portFlag = f.find((x) => x.split(/[\s=]/)[0] === "--port"); const servePort = portFlag ? portFlag.slice("--port".length).trim() : "{{PORT}}"; + const hostNetwork = multinode || pdMode || (typeof config.dockerHostNetworkWhen === "function" + && config.dockerHostNetworkWhen(sel, { flags: f, env: cellEnv })); // Mirrors `multiNodeDockerFlags` on the _deployment.jsx HARDWARE_CATALOG // (Mintlify strips module state, so the engines cannot share it). const HW_MULTINODE_DOCKER_FLAGS = { @@ -1430,7 +1478,7 @@ export const Playground = ({ config }) => { const dockerLines = [ "docker run --gpus all", " --shm-size 32g", - (multinode || pdMode) ? " --network host" : ` -p ${servePort}:${servePort}`, + hostNetwork ? " --network host" : ` -p ${servePort}:${servePort}`, ...(multinode ? fabricFlags.map((x) => " " + x) : []), " -v ~/.cache/huggingface:/root/.cache/huggingface", ...(config.dockerMounts || []).map((mount) => ` -v ${mount}`), diff --git a/docs/src/snippets/configs/inclusionAI/ling-3.0-flash-benchmarks.jsx b/docs/src/snippets/configs/inclusionAI/ling-3.0-flash-benchmarks.jsx new file mode 100644 index 000000000..50fa6f4ad --- /dev/null +++ b/docs/src/snippets/configs/inclusionAI/ling-3.0-flash-benchmarks.jsx @@ -0,0 +1,97 @@ +// Ling-3.0-flash per-cell benchmark numbers, keyed by the same `match` tuple as +// ling-3.0-flash.jsx cells. See _deployment.jsx for the speed/accuracy schema. +// +// Accuracy harness (one harness for the whole GSM8K column, per +// config.benchmarkCommands.accuracy): sgl-eval run gsm8k, full 1319 questions, +// --num-threads 32. Every filled entry below also recorded 100% stop / +// 0% truncated / 0% error. +// +// Speed numbers are not measured yet — entries carry accuracy only. +// +// Cells with no entry (H20-3e / H800 / H100, both quantizations) had no matching +// allocation and were never gated. +export const benchmarks = [ + // ==================================================================== + // H200 + BF16 (TP4) + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + sglang_version: "PR #33561 @ c5071ded", + accuracy: { gsm8k_pct: 96.59 }, + }, + { + // Rejected by the full GSM8K gate at request 1319: a no-EOS runaway generated + // >33k tokens. Recipe stays `verified: false` in ling-3.0-flash.jsx. + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + accuracy: { gsm8k_pct: null }, + notes: "Full GSM8K gate did not complete: one request ran away without emitting EOS (>33k generated tokens).", + }, + + // ==================================================================== + // H200 + FP8 (TP4 + EP4) + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + sglang_version: "PR #33561 @ e57e030b", + accuracy: { gsm8k_pct: 95.83 }, + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + sglang_version: "PR #33561 @ e57e030b", + accuracy: { gsm8k_pct: 96.51 }, + }, + + // ==================================================================== + // H200 + HiCache (Mooncake tiered cache) + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "hicache", nodes: "single" }, + sglang_version: "PR #33561 @ 51bcd89c", + accuracy: { gsm8k_pct: 96.44 }, + }, + + // ==================================================================== + // B200 + BF16 (TP4) + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + sglang_version: "PR #33561 @ c5071ded", + accuracy: { gsm8k_pct: 96.44 }, + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + sglang_version: "PR #33561 @ c5071ded", + accuracy: { gsm8k_pct: 96.51 }, + }, + + // ==================================================================== + // B200 + FP8 (TP4 + EP4) + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + sglang_version: "PR #33561 @ e57e030b", + accuracy: { gsm8k_pct: 96.59 }, + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + sglang_version: "PR #33561 @ e57e030b", + accuracy: { gsm8k_pct: 97.04 }, + }, + + // ==================================================================== + // GB300 + BF16 (TP4) + // ==================================================================== + // TODO: both cells are `verified: true` (gated on the final head) but the GSM8K + // percentages were not recorded in the PR body or in the verifying commits — + // fill from the run logs. + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } }, + + // ==================================================================== + // GB300 + FP8 (TP4 + EP4) + // ==================================================================== + // TODO: both cells are `verified: true` on the final head; the 96.66% / 96.44% + // pair in the PR body predates the TP+EP change — fill with the re-measured values. + { match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } }, +]; diff --git a/docs/src/snippets/configs/inclusionAI/ling-3.0-flash.jsx b/docs/src/snippets/configs/inclusionAI/ling-3.0-flash.jsx new file mode 100644 index 000000000..ff1392bd9 --- /dev/null +++ b/docs/src/snippets/configs/inclusionAI/ling-3.0-flash.jsx @@ -0,0 +1,615 @@ +export const config = { + modelName: "Ling-3.0-flash", + + supportedHardware: ["h20-3e", "h200", "h800", "h100", "b200", "gb300"], + groupHardware: false, + + variants: [ + { id: "default", label: "Ling-3.0-flash" }, + ], + quantizations: [ + { id: "bf16", label: "BF16" }, + { id: "fp8", label: "FP8" }, + ], + strategies: [ + { id: "low-latency", label: "Low-Latency" }, + { id: "high-throughput", label: "High-Throughput" }, + { id: "hicache", label: "HiCache + Mooncake" }, + ], + nodesOptions: [ + { id: "single", label: "Single Node" }, + ], + + modelNames: { + "default|bf16": "inclusionAI/Ling-3.0-flash", + "default|fp8": "inclusionAI/Ling-3.0-flash-fp8", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + MOONCAKE_MASTER: { target: "command", label: "Mooncake master", default: "127.0.0.1:50171" }, + MOONCAKE_METADATA_SERVER: { target: "command", label: "Mooncake metadata", default: "http://127.0.0.1:8290/metadata" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"What is the capital of France?"}] }'`, + + dockerImages: { + "h20-3e": "lmsysorg/sglang:dev-Ling-3.0-flash", + "h200": "lmsysorg/sglang:dev-Ling-3.0-flash", + "h800": "lmsysorg/sglang:dev-Ling-3.0-flash", + "h100": "lmsysorg/sglang:dev-Ling-3.0-flash", + "b200": "lmsysorg/sglang:dev-Ling-3.0-flash", + "gb300": "lmsysorg/sglang:dev-Ling-3.0-flash", + }, + + dockerHostNetworkWhen: (_sel, { flags }) => + flags.some((flag) => flag === "--hicache-storage-backend mooncake"), + + benchmarkCommands: { + speed: `python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --flush-cache`, + accuracy: { + gsm8k_pct: `# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gsm8k \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --num-threads 32`, + }, + }, + + accuracyLabels: [ + ["gsm8k_pct", "GSM8K", "%"], + ], + + github: { + cookbookModel: "inclusionAI/Ling-3.0-flash", + }, + + playgroundFeatures: { + attention: { + knobs: [ + { id: "tp", label: "TP", values: [null, 4, 8] }, + ], + }, + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser ling3" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser ling3" }, + ], + }, + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off (greedy)" }, + { id: "nextn", label: "NEXTN (built-in MTP)", flags: ["--speculative-algorithm NEXTN"] }, + ], + }, + hicache: { + defaultBackend: "mooncake", + requiredFlags: [ + "--mamba-scheduler-strategy extra_buffer", + "--enable-cache-report", + ], + backends: [ + { + id: "mooncake", + label: "Mooncake", + flags: [ + "--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'", + ], + env: [ + "MOONCAKE_MASTER={{MOONCAKE_MASTER}}", + "MOONCAKE_PROTOCOL=tcp", + "MC_MS_AUTO_DISC=0", + "MOONCAKE_DEVICE=", + "MOONCAKE_TE_META_DATA_SERVER={{MOONCAKE_METADATA_SERVER}}", + "MOONCAKE_GLOBAL_SEGMENT_SIZE=0", + ], + }, + ], + }, + }, + + cells: [ + { + match: { hw: "h20-3e", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h800", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h20-3e", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h800", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--ep-size 8", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--ep-size 8", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h20-3e", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h800", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h20-3e", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h800", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--ep-size 8", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--ep-size 8", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: ["SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // Hybrid KDA must pass prefix keys to Mooncake; otherwise storage writes are empty. + // Cold uncached extends above chunked_prefill_size skip write-through for that influx. + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "hicache", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", + "MOONCAKE_MASTER={{MOONCAKE_MASTER}}", + "MOONCAKE_PROTOCOL=tcp", + "MC_MS_AUTO_DISC=0", + "MOONCAKE_DEVICE=", + "MOONCAKE_TE_META_DATA_SERVER={{MOONCAKE_METADATA_SERVER}}", + "MOONCAKE_GLOBAL_SEGMENT_SIZE=0", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--enable-hierarchical-cache", + "--hicache-storage-backend mooncake", + "--hicache-io-backend direct", + "--hicache-mem-layout page_first_direct", + "--mamba-scheduler-strategy extra_buffer", + "--enable-cache-report", + "--hicache-storage-prefetch-policy wait_complete", + "--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "hicache", nodes: "single" }, + verified: false, + env: [ + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", + "MOONCAKE_MASTER={{MOONCAKE_MASTER}}", + "MOONCAKE_PROTOCOL=tcp", + "MC_MS_AUTO_DISC=0", + "MOONCAKE_DEVICE=", + "MOONCAKE_TE_META_DATA_SERVER={{MOONCAKE_METADATA_SERVER}}", + "MOONCAKE_GLOBAL_SEGMENT_SIZE=0", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--enable-hierarchical-cache", + "--hicache-storage-backend mooncake", + "--hicache-io-backend direct", + "--hicache-mem-layout page_first_direct", + "--mamba-scheduler-strategy extra_buffer", + "--enable-cache-report", + "--hicache-storage-prefetch-policy wait_complete", + "--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "hicache", nodes: "single" }, + verified: false, + env: [ + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", + "MOONCAKE_MASTER={{MOONCAKE_MASTER}}", + "MOONCAKE_PROTOCOL=tcp", + "MC_MS_AUTO_DISC=0", + "MOONCAKE_DEVICE=", + "MOONCAKE_TE_META_DATA_SERVER={{MOONCAKE_METADATA_SERVER}}", + "MOONCAKE_GLOBAL_SEGMENT_SIZE=0", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--enable-hierarchical-cache", + "--hicache-storage-backend mooncake", + "--hicache-io-backend direct", + "--hicache-mem-layout page_first_direct", + "--mamba-scheduler-strategy extra_buffer", + "--enable-cache-report", + "--hicache-storage-prefetch-policy wait_complete", + "--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "hicache", nodes: "single" }, + verified: false, + env: [ + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", + "MOONCAKE_MASTER={{MOONCAKE_MASTER}}", + "MOONCAKE_PROTOCOL=tcp", + "MC_MS_AUTO_DISC=0", + "MOONCAKE_DEVICE=", + "MOONCAKE_TE_META_DATA_SERVER={{MOONCAKE_METADATA_SERVER}}", + "MOONCAKE_GLOBAL_SEGMENT_SIZE=0", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--ep-size 4", + "--context-length 262144", + "--speculative-algorithm NEXTN", + "--json-model-override-args '{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":2.0,\"rope_theta\":6000000,\"partial_rotary_factor\":0.5,\"original_max_position_embeddings\":131072}}'", + "--mem-fraction-static 0.8", + "--enable-hierarchical-cache", + "--hicache-storage-backend mooncake", + "--hicache-io-backend direct", + "--hicache-mem-layout page_first_direct", + "--mamba-scheduler-strategy extra_buffer", + "--enable-cache-report", + "--hicache-storage-prefetch-policy wait_complete", + "--hicache-storage-backend-extra-config '{\"hicache_storage_pass_prefix_keys\":true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};