Add Laguna-M.1 cookbook (#28661)
This commit is contained in:
@@ -0,0 +1,234 @@
|
|||||||
|
---
|
||||||
|
title: Laguna-M.1
|
||||||
|
description: "Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4."
|
||||||
|
tag: NEW
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
<a id="install" />
|
||||||
|
|
||||||
|
<Accordion title="Install SGLang">
|
||||||
|
|
||||||
|
Laguna-M.1 support is already on SGLang `main` — **softplus per-element attention-output gating** ([PR #28400](https://github.com/sgl-project/sglang/pull/28400)) and a **global-attention fix** ([PR #28604](https://github.com/sgl-project/sglang/pull/28604), since M.1 is full-attention `sliding_window: 0`) — but not yet in a tagged release. The two paths below match the **Python / Docker** toggle in the command panel: install from `main` (Python tab), or use the **Docker** image, which bundles the same build (CUDA 13, covers H200 + all Blackwell). The model loads natively, so no `--trust-remote-code` is needed.
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab title="Python (pip / uv)">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
pip install -U uv
|
||||||
|
uv venv --python 3.12 && source .venv/bin/activate
|
||||||
|
|
||||||
|
# Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a
|
||||||
|
# tagged release — install from main. The serving runtime is in the base dependencies, no extra needed:
|
||||||
|
git clone https://github.com/sgl-project/sglang.git
|
||||||
|
cd sglang
|
||||||
|
uv pip install -e python
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image (`dev-cu13-618-nightly`) bundles the CUDA-13 runtime and the M.1 code. Once M.1 support lands in a tagged release, `uv pip install sglang` will pull it directly.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab title="Docker">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
# Pinned nightly with the Laguna-M.1 build (PR #28400 + #28604; CUDA 13 — covers H200 + all Blackwell):
|
||||||
|
docker pull lmsysorg/sglang:dev-cu13-618-nightly
|
||||||
|
```
|
||||||
|
|
||||||
|
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
Pick your hardware + quantization to generate the launch command. Laguna-M.1 ships a single **Balanced** recipe per cell — poolside's recommended operating point, a good speed/throughput trade-off for typical multi-user serving. The 8-GPU HGX platforms (H200 / B200 / B300) use `--tp 8`; the 4-GPU Grace-Blackwell single nodes (GB200 / GB300) use `--tp 4`.
|
||||||
|
|
||||||
|
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||||
|
import { config } from "/src/snippets/configs/poolside/laguna-m1.jsx";
|
||||||
|
import { benchmarks } from "/src/snippets/configs/poolside/laguna-m1-benchmarks.jsx";
|
||||||
|
|
||||||
|
<Deployment config={config} benchmarks={benchmarks} />
|
||||||
|
|
||||||
|
## Playground
|
||||||
|
|
||||||
|
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs (parsers, DP-Attention, DeepEP / EP) on top of whichever cell the Deploy panel is currently showing.
|
||||||
|
|
||||||
|
import { Playground } from "/src/snippets/_playground.jsx";
|
||||||
|
|
||||||
|
<Playground config={config} />
|
||||||
|
|
||||||
|
## 1. Model Introduction
|
||||||
|
|
||||||
|
[Laguna-M.1](https://huggingface.co/poolside/Laguna-M.1) is an open-weight, **225B-parameter** Mixture-of-Experts model (**23B activated per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software-engineering work. It is released under Apache 2.0.
|
||||||
|
|
||||||
|
**Key Features:**
|
||||||
|
|
||||||
|
- **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts, top-16 routing** (+1 shared expert) and auxiliary-loss-free load balancing.
|
||||||
|
- **Global attention with output gating**: global attention across all layers, 64 Q-heads / 8 KV-heads (head dim 128), with **softplus attention output gating** (requires [PR #28400](https://github.com/sgl-project/sglang/pull/28400)).
|
||||||
|
- **Long context**: 262,144 tokens, RoPE with YaRN.
|
||||||
|
- **Agentic coding**: competitive on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0.
|
||||||
|
- **Native reasoning**: interleaved thinking between tool calls, toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.
|
||||||
|
|
||||||
|
**Available Quantizations:**
|
||||||
|
|
||||||
|
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||||
|
<colgroup>
|
||||||
|
<col style={{width: "20%"}} />
|
||||||
|
<col style={{width: "80%"}} />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Quantization</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Hugging Face path</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1`](https://huggingface.co/poolside/Laguna-M.1)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1-FP8`](https://huggingface.co/poolside/Laguna-M.1-FP8)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1-NVFP4`](https://huggingface.co/poolside/Laguna-M.1-NVFP4)</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
**License:** Apache 2.0
|
||||||
|
|
||||||
|
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-M.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai).
|
||||||
|
|
||||||
|
## 2. Configuration Tips
|
||||||
|
|
||||||
|
- **Long-context memory**: M.1 is global-attention (no sliding-window), so the 262,144-token KV cache is large. If you hit OOM at full context, lower `--mem-fraction-static` or cap `--context-length`.
|
||||||
|
- **FP8 first launch**: FP8 triggers a multi-session DeepGEMM JIT pre-compile on first launch; pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-M.1-FP8` to avoid paying it on every restart.
|
||||||
|
- **Parsers** (`poolside_v1`): for agentic / tool-using deployments enable the **Reasoning Parser** and **Tool Call Parser** in the Playground above — they emit `--reasoning-parser poolside_v1` (thinking → `reasoning_content`) and `--tool-call-parser poolside_v1` (structured `tool_calls`).
|
||||||
|
- **Thinking default**: thinking is **off by default**; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||||
|
- **Served model id**: the server registers the model under whatever you pass to `--model-path`, so a client's `model` field must match it — `poolside/Laguna-M.1` (BF16) or `poolside/Laguna-M.1-FP8` / `-NVFP4` for the quantized cells. The §3 examples use the BF16 id; swap in the id you launched.
|
||||||
|
- **Recommended sampling**: poolside benchmarks M.1 at `temperature=1.0`, `top_k=20` with thinking enabled. These are per-request sampling params (not launch flags) — e.g. `temperature=1.0, extra_body={"top_k": 20}` on the OpenAI client.
|
||||||
|
## 3. Advanced Usage
|
||||||
|
|
||||||
|
### 3.1 Reasoning
|
||||||
|
|
||||||
|
Launch with `--reasoning-parser poolside_v1` (or toggle **Reasoning Parser** in the **Parsers** card of the Playground above). Reasoning is **opt-in**: the Laguna chat template gates it on `enable_thinking=True` (passed via `chat_template_kwargs`) — the generic `thinking` key is ignored. The `<think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.
|
||||||
|
|
||||||
|
<Accordion title="Reasoning Example (Python)">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="poolside/Laguna-M.1",
|
||||||
|
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||||
|
max_tokens=2048,
|
||||||
|
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||||
|
)
|
||||||
|
|
||||||
|
message = response.choices[0].message
|
||||||
|
print("=============== Reasoning ===============")
|
||||||
|
print(message.reasoning_content)
|
||||||
|
print("=============== Answer ==================")
|
||||||
|
print(message.content)
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Example Output">
|
||||||
|
|
||||||
|
```text Output
|
||||||
|
=============== Reasoning ===============
|
||||||
|
Okay, so I need to find out what 15% of 240 is. Hmm, percentages can sometimes be
|
||||||
|
tricky, but let me think. I remember that "percent" means per hundred, right? So 15%
|
||||||
|
is the same as 15 per 100 or 15/100. Maybe I can convert that percentage into a decimal
|
||||||
|
first? ... 15 divided by 100 is 0.15. ... Now, to find 15% of 240, I just need to
|
||||||
|
multiply 240 by 0.15. ... 240 times 0.1 is 24 (10% of 240), and 240 times 0.05 is 12
|
||||||
|
(half of that), so 24 + 12 = 36.
|
||||||
|
[… verifies the same result several more ways: 15/100 × 240, 240 × 15 ÷ 100,
|
||||||
|
1% × 15, and the fraction 3/20 × 240 — all give 36 …]
|
||||||
|
So ... all methods are pointing to 36. I'm pretty confident that 15% of 240 is 36.
|
||||||
|
=============== Answer ==================
|
||||||
|
To find 15% of 240, convert the percentage to a decimal (0.15) and multiply by 240:
|
||||||
|
**240 × 0.15 = 36**.
|
||||||
|
|
||||||
|
**Step-by-Step Explanation:**
|
||||||
|
1. **Convert 15% to a decimal:** 15% = 15/100 = 0.15.
|
||||||
|
2. **Multiply by 240:**
|
||||||
|
- Break it down:
|
||||||
|
- 10% of 240 = 24 (since 240 × 0.1 = 24).
|
||||||
|
- 5% of 240 = 12 (half of 24).
|
||||||
|
- Add them: 24 + 12 = **36**.
|
||||||
|
|
||||||
|
**Answer:** 15% of 240 is **36**.
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
Laguna-M.1's reasoning traces are long — the model explores and re-verifies an answer
|
||||||
|
multiple ways. Give it a generous `max_tokens` for harder problems (reasoning regularly
|
||||||
|
exceeds 3k tokens). The trace above is abbreviated; the model emits it in full.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
### 3.2 Tool Calling
|
||||||
|
|
||||||
|
Launch with `--tool-call-parser poolside_v1` (or toggle **Tool Call Parser** in the **Parsers** card of the Playground above). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (`enable_thinking=False`, the default).
|
||||||
|
|
||||||
|
<Accordion title="Tool Calling Example (Python)">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
|
||||||
|
tools = [
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"description": "Get the current weather for a location",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"location": {"type": "string", "description": "The city name"},
|
||||||
|
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||||
|
},
|
||||||
|
"required": ["location"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="poolside/Laguna-M.1",
|
||||||
|
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||||
|
tools=tools,
|
||||||
|
)
|
||||||
|
|
||||||
|
message = response.choices[0].message
|
||||||
|
if message.tool_calls:
|
||||||
|
for call in message.tool_calls:
|
||||||
|
print(f"Tool: {call.function.name}")
|
||||||
|
print(f"Args: {call.function.arguments}")
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Example Output">
|
||||||
|
|
||||||
|
```text Output
|
||||||
|
Tool: get_weather
|
||||||
|
Args: {"location": "Beijing"}
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
@@ -2,7 +2,6 @@
|
|||||||
title: Laguna-XS.2
|
title: Laguna-XS.2
|
||||||
metatags:
|
metatags:
|
||||||
description: "Deploy Poolside's Laguna-XS.2 hybrid SWA + MoE model with SGLang on NVIDIA H200 / B200 — agentic coding with hybrid reasoning and tool calling."
|
description: "Deploy Poolside's Laguna-XS.2 hybrid SWA + MoE model with SGLang on NVIDIA H200 / B200 — agentic coding with hybrid reasoning and tool calling."
|
||||||
tag: NEW
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Model Introduction
|
## 1. Model Introduction
|
||||||
|
|||||||
@@ -136,7 +136,7 @@ metatags:
|
|||||||
<Card
|
<Card
|
||||||
title="Poolside"
|
title="Poolside"
|
||||||
mode="card"
|
mode="card"
|
||||||
href="/cookbook/autoregressive/Poolside/Laguna-XS.2"
|
href="/cookbook/autoregressive/Poolside/Laguna-M.1"
|
||||||
img="/cards/logos/poolside.png"
|
img="/cards/logos/poolside.png"
|
||||||
/>
|
/>
|
||||||
</CardGroup>
|
</CardGroup>
|
||||||
|
|||||||
@@ -1123,6 +1123,7 @@
|
|||||||
{
|
{
|
||||||
"group": "Poolside",
|
"group": "Poolside",
|
||||||
"pages": [
|
"pages": [
|
||||||
|
"cookbook/autoregressive/Poolside/Laguna-M.1",
|
||||||
"cookbook/autoregressive/Poolside/Laguna-XS.2"
|
"cookbook/autoregressive/Poolside/Laguna-XS.2"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,81 @@
|
|||||||
|
// Laguna-M.1 benchmarks — one entry per cell `match` (same 5 keys as laguna-m1.jsx cells).
|
||||||
|
//
|
||||||
|
// All numbers below are REAL measured values; cells without measurements are bare `{ match }`
|
||||||
|
// pending stubs (the card renders "pending"). NO fabricated/dummy numbers remain.
|
||||||
|
// (cookbook_guide §3 forbids fabricated numbers in a published page.)
|
||||||
|
//
|
||||||
|
// FP8 is HOPPER-ONLY (not compatible with Blackwell) → the only FP8 entry is H200; there are no
|
||||||
|
// Blackwell FP8 entries. Blackwell cells are BF16 / NVFP4.
|
||||||
|
//
|
||||||
|
// REAL numbers (sgl-eval; GSM8K non-thinking; AIME via the enable_thinking wrapper, max_tokens=32768):
|
||||||
|
// H200 BF16 (tp8): GSM8K 93.02% · AIME25 53.33% overall (~0.80 stop-only) · perf (cc1, cc128).
|
||||||
|
// H200 FP8 (tp8): GSM8K 93.25% · AIME25 50.0% overall (~0.79 stop-only). g_proj FP8 fix validated; no perf (BF16-only scope).
|
||||||
|
// B200 BF16 (tp8): GSM8K 91.88% · AIME25 66.88% (n_repeats=16) · perf A/B (cc1, cc128).
|
||||||
|
// B200 NVFP4 (tp8): GSM8K 89.38%.
|
||||||
|
// ⚠️ AIME OVERALL is depressed by ~33–37% truncation at the 32k cap (M.1 reasoning is long → no boxed
|
||||||
|
// answer → scored 0); stop-only (~0.80) is the truer signal, and a 48–64k cap would lift the overall.
|
||||||
|
// (perf tokens_per_sec_per_gpu = measured output tok/s ÷ 8 GPUs; TTFT = median.)
|
||||||
|
//
|
||||||
|
// sglang_version reflects the REQUIRED build = PR #28400 (per-element gating) + PR #28604
|
||||||
|
// (global-attention SWA fix). The plain #28400 wheel (0.5.14.dev20260618+g343aeeef39) is NOT
|
||||||
|
// enough — it crashes M.1 under load. H200 numbers were taken on a main build @ 3f668733.
|
||||||
|
|
||||||
|
export const benchmarks = [
|
||||||
|
// ===== H200 — BF16 / FP8 =====
|
||||||
|
{
|
||||||
|
// ✅ REAL — 8xH200, BF16, tp8. GSM8K 0.9302; perf bench_serving random 4096/1024.
|
||||||
|
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "main @ 3f668733 (#28400 + #28604)",
|
||||||
|
speed: [
|
||||||
|
// cc=1: median TTFT 81.89 ms, median TPOT 8.91 ms, output 109.96 tok/s (÷8 ≈ 13.7/GPU).
|
||||||
|
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 81.9, tpot_ms: 8.91, tokens_per_sec_per_gpu: 13.7 },
|
||||||
|
// cc=128: median TTFT 200.11 ms (mean 1221), median TPOT 52.09 ms, output 2266 tok/s (÷8 ≈ 283/GPU); total 11311 tok/s.
|
||||||
|
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 128 },
|
||||||
|
ttft_ms: 200.1, tpot_ms: 52.1, tokens_per_sec_per_gpu: 283 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 93.02, aime25_pct: 53.33 }, // AIME overall, n_repeats=1 (32k-truncation-limited, ~33% trunc; stop-only ~0.80)
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// ✅ REAL — 8xH200, FP8, tp8. GSM8K 93.25 + AIME25 0.50 (overall). The g_proj quant fix is
|
||||||
|
// validated (FP8 now loads past layer 0). Perf not measured on FP8 (BF16-only scope) → no speed row.
|
||||||
|
// FP8 is Hopper-only — not compatible with Blackwell (no Blackwell FP8 entry below).
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "main @ 3f668733 (#28400 + #28604 + g_proj FP8 fix)",
|
||||||
|
accuracy: { gsm8k_pct: 93.25, aime25_pct: 50.0 }, // AIME overall (32k-truncation-limited; stop-only ~0.79)
|
||||||
|
},
|
||||||
|
|
||||||
|
// ===== B200 (8-GPU HGX) — BF16 / NVFP4 =====
|
||||||
|
{
|
||||||
|
// ✅ REAL — 8xB200, BF16, tp8. GSM8K 91.88 + AIME25 66.88; perf A/B (laguna-m1-results.md).
|
||||||
|
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "PR #28400 + #28604",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 108, tpot_ms: 9.0, tokens_per_sec_per_gpu: 13.6 },
|
||||||
|
{ workload: { dataset: "random", isl: 4096, osl: 1024, max_concurrency: 128 },
|
||||||
|
ttft_ms: 170, tpot_ms: 43.3, tokens_per_sec_per_gpu: 331 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 91.88, aime25_pct: 66.88 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// ✅ REAL (GSM8K only) — 8xB200, NVFP4, tp8. GSM8K 89.38 (laguna-m1-results.md).
|
||||||
|
// AIME + perf not measured yet → omitted (card shows them pending).
|
||||||
|
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "PR #28400 + #28604",
|
||||||
|
accuracy: { gsm8k_pct: 89.38 },
|
||||||
|
},
|
||||||
|
|
||||||
|
// ===== B300 / GB200 / GB300 — BF16 / NVFP4, UNVERIFIED, no data yet → bare "pending" stubs
|
||||||
|
// (no fabricated numbers; FP8 is Hopper-only so no Blackwell FP8 entries). =====
|
||||||
|
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "gb200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" } },
|
||||||
|
];
|
||||||
@@ -0,0 +1,302 @@
|
|||||||
|
// Laguna-M.1 (poolside) — config-driven cookbook page.
|
||||||
|
// Consumed by the shared _deployment.jsx + _playground.jsx engines (no model code there).
|
||||||
|
//
|
||||||
|
// Build: M.1 needs SGLang PR #28400 (softplus per-element output gating, MERGED) AND PR #28604
|
||||||
|
// (global-attention SWA fix — M.1 is sliding_window=0 / all-global; without it M.1 crashes ~1s
|
||||||
|
// into any concurrent batch with AssertionError: ... swa_lock_ref=0). Both are merged on main
|
||||||
|
// (verified on a 3f668733 build). The shipped recipe carries NO workaround flag, but the pinned
|
||||||
|
// build MUST contain BOTH; the #28400-merge wheel 0.5.14.dev20260618+g343aeeef39 is #28400-ONLY
|
||||||
|
// and crashes under load. Pin dockerImages + benchmarks.sglang_version to a build at a commit
|
||||||
|
// ≥ #28604. See /sgl-workspace/laguna-m1-day0-checklist.md (step 2) + laguna-m1-results.md.
|
||||||
|
//
|
||||||
|
// Model is now natively supported (#28400) → NO --trust-remote-code needed.
|
||||||
|
//
|
||||||
|
// Hardware: H200 (Hopper) + B200/B300/GB200/GB300 (Blackwell).
|
||||||
|
// - BF16 runs everywhere.
|
||||||
|
// - FP8 is HOPPER-ONLY — not compatible with Blackwell (use NVFP4 for low-precision there).
|
||||||
|
// - NVFP4 is Blackwell-only.
|
||||||
|
// So the only quant×hw combos with a cell are: H200×{BF16,FP8} and each Blackwell×{BF16,NVFP4}.
|
||||||
|
// TP: 8-GPU HGX nodes (H200/B200/B300) → --tp 8 (the maintainer's baseline); GB200/GB300
|
||||||
|
// (Grace-Blackwell, typically 4-GPU single node) → --tp 4. Adjust --tp to your node size.
|
||||||
|
//
|
||||||
|
// Strategy: a SINGLE "Balanced" operating point (maintainer decision — no LL/HT split; an
|
||||||
|
// earlier TP=8+DP-Attention "high-throughput" idea was dropped: DP-Attention is ~15% SLOWER
|
||||||
|
// on this GQA model, see laguna-m1-results.md).
|
||||||
|
|
||||||
|
export const config = {
|
||||||
|
modelName: "Laguna-M.1",
|
||||||
|
|
||||||
|
supportedHardware: ["h200", "b200", "b300", "gb200", "gb300"],
|
||||||
|
|
||||||
|
variants: [
|
||||||
|
{ id: "default", label: "Default" },
|
||||||
|
],
|
||||||
|
|
||||||
|
quantizations: [
|
||||||
|
{ id: "bf16", label: "BF16" },
|
||||||
|
{ id: "fp8", label: "FP8" },
|
||||||
|
{ id: "nvfp4", label: "NVFP4" },
|
||||||
|
],
|
||||||
|
|
||||||
|
// Single balanced operating point (maintainer decision — no LL/HT split).
|
||||||
|
strategies: [
|
||||||
|
{ id: "balanced", label: "Balanced" },
|
||||||
|
],
|
||||||
|
|
||||||
|
nodesOptions: [
|
||||||
|
{ id: "single", label: "Single Node" },
|
||||||
|
],
|
||||||
|
|
||||||
|
modelNames: {
|
||||||
|
"default|bf16": "poolside/Laguna-M.1",
|
||||||
|
"default|fp8": "poolside/Laguna-M.1-FP8",
|
||||||
|
"default|nvfp4": "poolside/Laguna-M.1-NVFP4",
|
||||||
|
},
|
||||||
|
|
||||||
|
placeholders: {
|
||||||
|
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||||
|
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||||
|
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
|
||||||
|
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||||
|
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||||
|
},
|
||||||
|
|
||||||
|
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||||
|
-H 'Content-Type: application/json' \\
|
||||||
|
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
|
||||||
|
|
||||||
|
benchmarkCommands: {
|
||||||
|
speed:
|
||||||
|
`python3 -m sglang.bench_serving \\
|
||||||
|
--backend sglang \\
|
||||||
|
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
|
||||||
|
--model {{MODEL_NAME}} \\
|
||||||
|
--dataset-name {{DATASET}} \\
|
||||||
|
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
|
||||||
|
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
|
||||||
|
// GSM8K sanity is the REQUIRED minimum on every verified cell (cookbook_guide §3); AIME 25
|
||||||
|
// (thinking ON) is the harder accuracy check (model_support_guide). All via sgl-eval.
|
||||||
|
// NOTE: M.1 needs enable_thinking, not sgl-eval's --thinking key (which the template ignores)
|
||||||
|
// — thinking evals were run via the enable_thinking wrapper (laguna-m1-results.md).
|
||||||
|
accuracy: {
|
||||||
|
gsm8k_pct:
|
||||||
|
`# pip install git+https://github.com/sgl-project/sgl-eval
|
||||||
|
sgl-eval run gsm8k \\
|
||||||
|
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||||
|
--num-threads 128`,
|
||||||
|
aime25_pct:
|
||||||
|
`# pip install git+https://github.com/sgl-project/sgl-eval
|
||||||
|
sgl-eval run aime25 --thinking \\
|
||||||
|
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||||
|
--num-threads 32`,
|
||||||
|
},
|
||||||
|
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 128: 256, 256: 512, 1024: 2048, 4096: 4096 },
|
||||||
|
},
|
||||||
|
|
||||||
|
// Accuracy here varies by QUANT (BF16 vs NVFP4), so real numbers live per-cell in
|
||||||
|
// laguna-m1-benchmarks.jsx (which overrides this). Keep null = no variant-wide default.
|
||||||
|
defaultAccuracy: {
|
||||||
|
default: { gsm8k_pct: null, aime25_pct: null },
|
||||||
|
},
|
||||||
|
|
||||||
|
accuracyLabels: [
|
||||||
|
["gsm8k_pct", "GSM8K", "%"],
|
||||||
|
["aime25_pct", "AIME 25", "%"],
|
||||||
|
],
|
||||||
|
|
||||||
|
// Pinned nightly with the Laguna-M.1 build (PR #28400 + #28604 + #28649; cu13 covers H200 + all Blackwell).
|
||||||
|
// dev-cu13-618-nightly was generated after the FP8 g_proj fix (#28649) landed, so it serves H200 FP8 as
|
||||||
|
// well as BF16/NVFP4. (Equivalent pip nightly: 0.5.14.dev20260618+g97e3b8998d.)
|
||||||
|
dockerImages: {
|
||||||
|
h200: "lmsysorg/sglang:dev-cu13-618-nightly",
|
||||||
|
b200: "lmsysorg/sglang:dev-cu13-618-nightly",
|
||||||
|
b300: "lmsysorg/sglang:dev-cu13-618-nightly",
|
||||||
|
gb200: "lmsysorg/sglang:dev-cu13-618-nightly",
|
||||||
|
gb300: "lmsysorg/sglang:dev-cu13-618-nightly",
|
||||||
|
},
|
||||||
|
|
||||||
|
github: {
|
||||||
|
cookbookModel: "poolside/Laguna-M.1",
|
||||||
|
},
|
||||||
|
|
||||||
|
playgroundFeatures: {
|
||||||
|
|
||||||
|
// M.1 is global-attention (no SWA); expose TP + DP-Attention. No CP.
|
||||||
|
// DP-Attention is a Playground experiment only — ~15% slower than plain TP on this GQA
|
||||||
|
// model (8 KV heads), so it is NOT in the shipped Balanced recipe.
|
||||||
|
attention: {
|
||||||
|
knobs: [
|
||||||
|
{ id: "tp", label: "TP", values: [null, 1, 2, 4, 8] },
|
||||||
|
{ id: "dpAttn", label: "DP-Attention", values: [null, false, 1, 2, 4, 8],
|
||||||
|
labels: { "auto": "Auto", "false": "Off" } },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
// 256-expert top-16 MoE. DeepEP all-to-all + EP degree.
|
||||||
|
moe: {
|
||||||
|
backend: {
|
||||||
|
options: [
|
||||||
|
{ id: null, label: "Inherited" },
|
||||||
|
{ id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
ep: { label: "EP", values: [null, 1, 2, 4, 8] },
|
||||||
|
},
|
||||||
|
|
||||||
|
// Reasoning + tool-call parsers (poolside_v1, same family as Laguna-XS.2). ALSO baked into
|
||||||
|
// every Deploy cell below (the maintainer's baseline carries them).
|
||||||
|
parsers: {
|
||||||
|
items: [
|
||||||
|
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser poolside_v1" },
|
||||||
|
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser poolside_v1" },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
|
||||||
|
// One Balanced cell per valid (hw × quant): H200×{BF16,FP8}, each Blackwell×{BF16,NVFP4}.
|
||||||
|
// FP8 is Hopper-only (no Blackwell FP8 cell — those combos grey out). Baseline recipe
|
||||||
|
// (parsers poolside_v1, NO --trust-remote-code) baked into every cell.
|
||||||
|
// TP: H200/B200/B300 = --tp 8; GB200/GB300 = --tp 4 (4-GPU single node).
|
||||||
|
// verified:true = ran that exact command on that hardware and it served correctly + passed a
|
||||||
|
// GSM8K-class eval. Absent verified = yellow/unverified badge.
|
||||||
|
cells: [
|
||||||
|
// ===== NVIDIA Hopper (H200) — BF16 / FP8 (NVFP4 is Blackwell-only) =====
|
||||||
|
{
|
||||||
|
// VERIFIED on 8xH200 (BF16, tp8): GSM8K 93.02% + perf (laguna-m1 H200 results).
|
||||||
|
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// VERIFIED on 8xH200 (FP8, tp8): GSM8K 93.25 + AIME25 0.50; g_proj FP8 quant fix validated.
|
||||||
|
// FP8 needs the g_proj fix (PR #28649, MERGED) on top of #28400+#28604 — the pinned
|
||||||
|
// dev-cu13-618-nightly image includes all three.
|
||||||
|
// FP8 is Hopper-only — it is NOT compatible with Blackwell, so there is no Blackwell FP8 cell.
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
// ===== NVIDIA Blackwell B200 (8-GPU HGX) — BF16 / NVFP4 (FP8 is Hopper-only) =====
|
||||||
|
{
|
||||||
|
// VERIFIED on 8xB200 (BF16, tp8): served clean under batched shared-prefix load,
|
||||||
|
// GSM8K 91.88% + AIME25 66.88% (laguna-m1-results.md).
|
||||||
|
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// VERIFIED on 8xB200 (NVFP4, tp8): GSM8K 89.38% (laguna-m1-results.md). tp8 now matches
|
||||||
|
// the shipped recipe.
|
||||||
|
match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
// ===== NVIDIA Blackwell B300 (8-GPU HGX) — BF16 / NVFP4 (UNVERIFIED; FP8 is Hopper-only) =====
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
// ===== NVIDIA Grace-Blackwell GB200 (4-GPU single node) — BF16 / NVFP4 (UNVERIFIED; FP8 Hopper-only) =====
|
||||||
|
{
|
||||||
|
match: { hw: "gb200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
// ===== NVIDIA Grace-Blackwell GB300 (4-GPU single node) — BF16 / NVFP4 (UNVERIFIED; FP8 Hopper-only) =====
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user