[Cookbook] Add Laguna-S-2.1 (#31918)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Jimmy Shong
co-authored by
Zijie Xia
Claude Sonnet 4.6
zijiexia
Zijie Xia
Claude Sonnet 4.6
zijiexia
parent
57e5846b90
commit
9a6c96083f
@@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Laguna-M.1
|
title: Laguna-M.1
|
||||||
description: "Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4."
|
description: "Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4."
|
||||||
tag: NEW
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|||||||
@@ -0,0 +1,244 @@
|
|||||||
|
---
|
||||||
|
title: Laguna-S-2.1
|
||||||
|
description: "Deploy poolside's Laguna-S-2.1 — a 118B hybrid-SWA Mixture-of-Experts model (8B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||||
|
tag: NEW
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
<a id="install" />
|
||||||
|
|
||||||
|
<Accordion title="Install SGLang">
|
||||||
|
|
||||||
|
Laguna-S-2.1 uses the same `laguna` model architecture as [Laguna-XS-2.1](./Laguna-XS-2.1), which is fully supported in SGLang `main`. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab title="Python (pip / uv)">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
pip install -U uv
|
||||||
|
uv venv --python 3.12 && source .venv/bin/activate
|
||||||
|
|
||||||
|
git clone https://github.com/sgl-project/sglang.git
|
||||||
|
cd sglang
|
||||||
|
uv pip install -e python
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the **Python** output of the command panel below in that environment.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab title="Docker">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
docker pull lmsysorg/sglang:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:
|
||||||
|
|
||||||
|
- **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
|
||||||
|
- **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.
|
||||||
|
|
||||||
|
On the 8-GPU HGX platforms (H200 / B300) all quantizations run `--tp 8`. The 4-GPU GB300 node runs `--tp 4` throughout. NVFP4 is Blackwell-only (B300 / GB300 only).
|
||||||
|
|
||||||
|
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||||
|
import { config } from "/src/snippets/configs/poolside/laguna-s21.jsx";
|
||||||
|
import { benchmarks } from "/src/snippets/configs/poolside/laguna-s21-benchmarks.jsx";
|
||||||
|
|
||||||
|
<Deployment config={config} benchmarks={benchmarks} />
|
||||||
|
|
||||||
|
## Playground
|
||||||
|
|
||||||
|
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.
|
||||||
|
|
||||||
|
import { Playground } from "/src/snippets/_playground.jsx";
|
||||||
|
|
||||||
|
<Playground config={config} />
|
||||||
|
|
||||||
|
## 1. Model Introduction
|
||||||
|
|
||||||
|
[Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is an open-weight **118B-parameter** hybrid sliding-window-attention MoE model (**~8B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering. It sits between [Laguna XS 2.1](./Laguna-XS-2.1) (33B/3B active) and Laguna M.1 (222B/23B active) in the Laguna family.
|
||||||
|
|
||||||
|
**Key Features:**
|
||||||
|
|
||||||
|
- **Sparse MoE**: 48 layers, 256 routed experts, top-10 routing, plus 1 shared expert.
|
||||||
|
- **Hybrid attention**: 36 sliding-window layers (window 512) interleaved with 12 full-attention layers (1:3 global-to-SWA ratio); 8 KV heads, head dim 128; per-head sigmoid output gating with per-layer-type rotary scales.
|
||||||
|
- **Long context**: 1,048,576 tokens.
|
||||||
|
- **DFlash drafts**: matched draft models ship per quantization for low-latency serving.
|
||||||
|
- **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.
|
||||||
|
|
||||||
|
**Available quantizations:**
|
||||||
|
|
||||||
|
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||||
|
<colgroup>
|
||||||
|
<col style={{width: "14%"}} />
|
||||||
|
<col style={{width: "43%"}} />
|
||||||
|
<col style={{width: "43%"}} />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Target model</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Draft model</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1`](https://huggingface.co/poolside/Laguna-S-2.1)</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-FP8)</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4)</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>INT4</strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-INT4)</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4)</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
The drafts are small BF16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).
|
||||||
|
|
||||||
|
**License:** [OpenMDW-1.1](https://openmdw.ai/)
|
||||||
|
|
||||||
|
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-S-2.1) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai)
|
||||||
|
|
||||||
|
## 2. Configuration Tips
|
||||||
|
|
||||||
|
**Attention backend**
|
||||||
|
|
||||||
|
Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (reproduced on Laguna-XS-2.1, greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Other attention backend choices have not been fully validated on Laguna; keep the default.
|
||||||
|
|
||||||
|
**BF16 memory on H200**
|
||||||
|
|
||||||
|
BF16 on H200 leaves less headroom for CUDA-graph capture and NCCL allocations than FP8/INT4. The High-throughput BF16 command carries `--mem-fraction-static 0.80`. FP8, INT4, and all B300/GB300 cells use the default heuristic.
|
||||||
|
|
||||||
|
**FP8 shared expert**
|
||||||
|
|
||||||
|
`SGLANG_SHARED_EXPERT_TP1=1` is required for FP8 cells on **all hardware** — confirmed on both H200 (TP=8) and GB300 (TP=4). The FP8 checkpoint block-quantizes the shared expert (128×128 scales), which cannot TP-shard cleanly at either TP degree on S-2.1. This env var replicates the shared expert instead of sharding it. INT4 keeps the shared expert in BF16 (no flag needed); BF16 is unquantized. Note: this differs from Laguna-XS-2.1 where TP=4 does not require the flag — the constraint is architecture-specific.
|
||||||
|
|
||||||
|
**FP8 and NVFP4 DFlash drafts**
|
||||||
|
|
||||||
|
Fixed upstream on 2026-07-21: all DFlash draft configs now use a flat top-level `rope_theta` (the `rope_parameters` block was removed). If a server crashes at draft-model load with `KeyError: 'rope_theta'`, you are serving a draft checkpoint cached before 2026-07-21 — re-download it (e.g. `hf download poolside/Laguna-S-2.1-DFlash-FP8`) to pick up the corrected config.
|
||||||
|
|
||||||
|
**DFlash memory**
|
||||||
|
|
||||||
|
Low-latency cells carry `--mem-fraction-static 0.7` (sufficient even for BF16 on H200). Dense cells use the default heuristic (except BF16 on H200 — see above).
|
||||||
|
|
||||||
|
**BF16 reasoning length**
|
||||||
|
|
||||||
|
BF16 reasons approximately 2× longer than FP8/INT4 on AIME25 (median 34.8 k vs 16.9 k tokens), consistently truncating at `max_tokens=64000`. FP8/INT4 truncate at ≈ 2%. For a valid BF16 AIME25 score, serve with `max_tokens ≥ 131072` (the model supports a 1 M context window).
|
||||||
|
|
||||||
|
**Chat template**
|
||||||
|
|
||||||
|
On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ ~5.8) pass `--chat-template <model-dir>/chat_template.jinja` explicitly.
|
||||||
|
|
||||||
|
**Thinking**
|
||||||
|
|
||||||
|
Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.
|
||||||
|
|
||||||
|
**Served model id**
|
||||||
|
|
||||||
|
The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-S-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).
|
||||||
|
|
||||||
|
## 3. Advanced Usage
|
||||||
|
|
||||||
|
### 3.1 DFlash Speculative Decoding
|
||||||
|
|
||||||
|
DFlash is a block-wise speculative decoder: the draft proposes a block of tokens and the target verifies the whole block in one forward pass — output quality is the target's by construction. The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step.
|
||||||
|
|
||||||
|
Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. The generated commands always pair the draft calibrated for the selected target precision.
|
||||||
|
|
||||||
|
### 3.2 Reasoning
|
||||||
|
|
||||||
|
Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.
|
||||||
|
|
||||||
|
<Accordion title="Reasoning Example (Python)">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="poolside/Laguna-S-2.1",
|
||||||
|
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||||
|
max_tokens=4096,
|
||||||
|
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||||
|
)
|
||||||
|
|
||||||
|
message = response.choices[0].message
|
||||||
|
print("=============== Reasoning ===============")
|
||||||
|
print(message.reasoning_content)
|
||||||
|
print("=============== Answer ==================")
|
||||||
|
print(message.content)
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
Give generous `max_tokens` when thinking is enabled — hard problems regularly reason
|
||||||
|
for thousands of tokens. Keep thinking off for short-form tasks.
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
### 3.3 Tool Calling
|
||||||
|
|
||||||
|
Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).
|
||||||
|
|
||||||
|
<Accordion title="Tool Calling Example (Python)">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
|
||||||
|
tools = [
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"description": "Get the current weather for a location",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"location": {"type": "string", "description": "The city name"},
|
||||||
|
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||||
|
},
|
||||||
|
"required": ["location"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="poolside/Laguna-S-2.1",
|
||||||
|
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||||
|
tools=tools,
|
||||||
|
)
|
||||||
|
|
||||||
|
message = response.choices[0].message
|
||||||
|
if message.tool_calls:
|
||||||
|
for call in message.tool_calls:
|
||||||
|
print(f"Tool: {call.function.name}")
|
||||||
|
print(f"Args: {call.function.arguments}")
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
@@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Laguna-XS-2.1
|
title: Laguna-XS-2.1
|
||||||
description: "Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
description: "Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||||
tag: NEW
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|||||||
@@ -160,7 +160,7 @@ metatags:
|
|||||||
<Card
|
<Card
|
||||||
title="Poolside"
|
title="Poolside"
|
||||||
mode="card"
|
mode="card"
|
||||||
href="/cookbook/autoregressive/Poolside/Laguna-XS-2.1"
|
href="/cookbook/autoregressive/Poolside/Laguna-S-2.1"
|
||||||
img="/cards/logos/poolside.png"
|
img="/cards/logos/poolside.png"
|
||||||
/>
|
/>
|
||||||
</CardGroup>
|
</CardGroup>
|
||||||
|
|||||||
@@ -1172,6 +1172,7 @@
|
|||||||
{
|
{
|
||||||
"group": "Poolside",
|
"group": "Poolside",
|
||||||
"pages": [
|
"pages": [
|
||||||
|
"cookbook/autoregressive/Poolside/Laguna-S-2.1",
|
||||||
"cookbook/autoregressive/Poolside/Laguna-XS-2.1",
|
"cookbook/autoregressive/Poolside/Laguna-XS-2.1",
|
||||||
"cookbook/autoregressive/Poolside/Laguna-M.1",
|
"cookbook/autoregressive/Poolside/Laguna-M.1",
|
||||||
"cookbook/autoregressive/Poolside/Laguna-XS.2"
|
"cookbook/autoregressive/Poolside/Laguna-XS.2"
|
||||||
|
|||||||
@@ -0,0 +1,218 @@
|
|||||||
|
export const benchmarks = [
|
||||||
|
|
||||||
|
// ── H200 (8×H200, tp 8, sglang 0.5.15.post1, random ISL=8192/OSL=1024) ──
|
||||||
|
// tokens_per_sec_per_gpu = (input+output) tok/s/GPU = output_tok_s / 8 * (8192+1024)/1024
|
||||||
|
// TTFT/TPOT are mean values from bench_serving.
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 23742, tpot_ms: 256.8, tokens_per_sec_per_gpu: 5264 },
|
||||||
|
],
|
||||||
|
// BF16 reasons ~2× longer than FP8/INT4 (median 34.8k vs 16.9k tokens);
|
||||||
|
// truncation at max_tokens=64000 invalidates the result. Needs max_tokens ≥ 131072.
|
||||||
|
accuracy: { gsm8k_pct: 93.18, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 102.3, tpot_ms: 3.48, tokens_per_sec_per_gpu: 305 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 97.7, tpot_ms: 6.45, tokens_per_sec_per_gpu: 2094 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 93.33, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 16199, tpot_ms: 293.2, tokens_per_sec_per_gpu: 5175 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.24, aime25_pct: 0.654 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 97.7, tpot_ms: 4.43, tokens_per_sec_per_gpu: 242 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 99.4, tpot_ms: 7.16, tokens_per_sec_per_gpu: 1956 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.47, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "int4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 19886, tpot_ms: 318.5, tokens_per_sec_per_gpu: 5055 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 95.00, aime25_pct: 0.690 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "int4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 113.1, tpot_ms: 3.83, tokens_per_sec_per_gpu: 276 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 94.1, tpot_ms: 6.40, tokens_per_sec_per_gpu: 2125 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.24, aime25_pct: null },
|
||||||
|
},
|
||||||
|
|
||||||
|
// ── B300 (8×B300, tp 8, sglang 0.5.15.post1) ──
|
||||||
|
// AIME25 (high-throughput cells): sgl-eval run aime25, 30 problems × 16 repeats,
|
||||||
|
// temp 1.0, top_p 0.95, max_tokens 64000 (BF16: 131072), 128 threads, thinking ON
|
||||||
|
// via enable_thinking-patched sgl-eval. 2026-07-21.
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 93.71, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 93.63, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 94.39, aime25_pct: 66.88 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 94.69, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 94.54, aime25_pct: 66.46 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 95.30, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "int4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 94.62, aime25_pct: 68.33 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "int4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 94.69, aime25_pct: null },
|
||||||
|
},
|
||||||
|
|
||||||
|
// ── GB300 (4×GB300, tp 4, sglang 0.5.15.post1, random ISL=8192/OSL=1024) ──
|
||||||
|
// tokens_per_sec_per_gpu = output_tok_s / 4 * (8192+1024)/1024
|
||||||
|
// TTFT/TPOT are median values from bench_serving.
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
accuracy: { gsm8k_pct: 93.33, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 114, tpot_ms: 4.2, tokens_per_sec_per_gpu: 562 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 141, tpot_ms: 8.4, tokens_per_sec_per_gpu: 3413 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 93.33, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 10062, tpot_ms: 216, tokens_per_sec_per_gpu: 9536 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.31, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 109, tpot_ms: 6.1, tokens_per_sec_per_gpu: 369 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 135, tpot_ms: 10.7, tokens_per_sec_per_gpu: 2578 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.54, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 7186, tpot_ms: 211, tokens_per_sec_per_gpu: 9658 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.47, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 115, tpot_ms: 11.2, tokens_per_sec_per_gpu: 204 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 125, tpot_ms: 18.8, tokens_per_sec_per_gpu: 1549 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.77, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "int4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 10428, tpot_ms: 216, tokens_per_sec_per_gpu: 9184 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 94.69, aime25_pct: null },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "int4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
sglang_version: "0.5.15.post1",
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 134, tpot_ms: 5.4, tokens_per_sec_per_gpu: 410 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 127, tpot_ms: 10.1, tokens_per_sec_per_gpu: 2880 },
|
||||||
|
],
|
||||||
|
accuracy: { gsm8k_pct: 95.00, aime25_pct: null },
|
||||||
|
},
|
||||||
|
];
|
||||||
@@ -0,0 +1,506 @@
|
|||||||
|
// Laguna-S-2.1 (poolside) — 118B MoE (8B active), 1M context, laguna arch (SGLang main).
|
||||||
|
// --trust-remote-code required (custom config code on Hub).
|
||||||
|
//
|
||||||
|
// Attention backend: leave unset for High-Throughput (auto-selects fa3/trtllm_mha).
|
||||||
|
// With DFlash active, auto falls back to flashinfer which breaks hybrid-SWA at tp≥4
|
||||||
|
// on Blackwell — Low-Latency cells pin the target backend explicitly.
|
||||||
|
// Never use --attention-backend triton on Laguna (broken SWA handling).
|
||||||
|
//
|
||||||
|
// BF16 on H200 HT: --mem-fraction-static 0.80 required — BF16 leaves less headroom
|
||||||
|
// for CUDA-graph capture and NCCL allocs than FP8/INT4 on 141 GB/GPU.
|
||||||
|
// B300/GB300 (288 GB/GPU) unaffected.
|
||||||
|
//
|
||||||
|
// SGLANG_SHARED_EXPERT_TP1=1 (FP8 cells, all hardware): FP8 block-quantizes the shared
|
||||||
|
// expert; required at both TP=4 (GB300) and TP=8 (H200/B300). INT4 shared expert stays
|
||||||
|
// bf16 — no flag needed. Differs from Laguna-XS-2.1 where TP=4 works without this flag.
|
||||||
|
//
|
||||||
|
// NVFP4 is Blackwell-only → no h200×nvfp4 cells.
|
||||||
|
// DFlash cells carry --mem-fraction-static 0.7.
|
||||||
|
|
||||||
|
export const config = {
|
||||||
|
modelName: "Laguna-S-2.1",
|
||||||
|
|
||||||
|
supportedHardware: ["h200", "b300", "gb300"],
|
||||||
|
|
||||||
|
variants: [
|
||||||
|
{ id: "default", label: "Default" },
|
||||||
|
],
|
||||||
|
|
||||||
|
quantizations: [
|
||||||
|
{ id: "bf16", label: "BF16" },
|
||||||
|
{ id: "fp8", label: "FP8" },
|
||||||
|
{ id: "nvfp4", label: "NVFP4" },
|
||||||
|
{ id: "int4", label: "INT4" },
|
||||||
|
],
|
||||||
|
|
||||||
|
strategies: [
|
||||||
|
{ id: "low-latency", label: "Low-latency" },
|
||||||
|
{ id: "high-throughput", label: "High-throughput" },
|
||||||
|
],
|
||||||
|
|
||||||
|
nodesOptions: [
|
||||||
|
{ id: "single", label: "Single Node" },
|
||||||
|
],
|
||||||
|
|
||||||
|
modelNames: {
|
||||||
|
"default|bf16": "poolside/Laguna-S-2.1",
|
||||||
|
"default|fp8": "poolside/Laguna-S-2.1-FP8",
|
||||||
|
"default|nvfp4": "poolside/Laguna-S-2.1-NVFP4",
|
||||||
|
"default|int4": "poolside/Laguna-S-2.1-INT4",
|
||||||
|
},
|
||||||
|
|
||||||
|
placeholders: {
|
||||||
|
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||||
|
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||||
|
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
|
||||||
|
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||||
|
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||||
|
},
|
||||||
|
|
||||||
|
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||||
|
-H 'Content-Type: application/json' \\
|
||||||
|
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
|
||||||
|
|
||||||
|
benchmarkCommands: {
|
||||||
|
speed:
|
||||||
|
`python3 -m sglang.bench_serving \\
|
||||||
|
--backend sglang \\
|
||||||
|
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
|
||||||
|
--model {{MODEL_NAME}} \\
|
||||||
|
--dataset-name {{DATASET}} \\
|
||||||
|
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
|
||||||
|
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
|
||||||
|
accuracy: {
|
||||||
|
gsm8k_pct:
|
||||||
|
`# pip install git+https://github.com/sgl-project/sgl-eval
|
||||||
|
sgl-eval run gsm8k \\
|
||||||
|
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||||
|
--num-threads 128`,
|
||||||
|
// Laguna's template gates on enable_thinking, not the generic 'thinking' key.
|
||||||
|
// Serve with a copy of the model's chat template whose enable_thinking default
|
||||||
|
// is flipped to true. For BF16: use --max-tokens 131072 (see Configuration Tips).
|
||||||
|
aime25_pct:
|
||||||
|
`# pip install git+https://github.com/sgl-project/sgl-eval
|
||||||
|
# Serve with an enable_thinking=true chat template (see Configuration Tips: Thinking).
|
||||||
|
# For BF16: use --max-tokens 131072 (see Configuration Tips: BF16 reasoning length).
|
||||||
|
sgl-eval run aime25 \\
|
||||||
|
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||||
|
--n-repeats 16 --max-tokens 64000 \\
|
||||||
|
--temperature 1.0 --top-p 0.95 --thinking \\
|
||||||
|
--num-threads 128`,
|
||||||
|
},
|
||||||
|
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 128: 256, 256: 512, 1024: 2048, 4096: 4096 },
|
||||||
|
},
|
||||||
|
|
||||||
|
defaultAccuracy: {
|
||||||
|
default: { gsm8k_pct: null, aime25_pct: null },
|
||||||
|
},
|
||||||
|
|
||||||
|
accuracyLabels: [
|
||||||
|
["gsm8k_pct", "GSM8K", "%"],
|
||||||
|
["aime25_pct", "AIME25", "%"],
|
||||||
|
],
|
||||||
|
|
||||||
|
dockerImages: {
|
||||||
|
h200: "lmsysorg/sglang:latest",
|
||||||
|
b300: "lmsysorg/sglang:dev",
|
||||||
|
gb300: "lmsysorg/sglang:dev",
|
||||||
|
},
|
||||||
|
|
||||||
|
github: {
|
||||||
|
cookbookModel: "poolside/Laguna-S-2.1",
|
||||||
|
},
|
||||||
|
|
||||||
|
playgroundFeatures: {
|
||||||
|
attention: {
|
||||||
|
knobs: [
|
||||||
|
{ id: "tp", label: "TP", values: [null, 1, 2, 4, 8] },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
parsers: {
|
||||||
|
items: [
|
||||||
|
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser poolside_v1" },
|
||||||
|
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser poolside_v1" },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
|
||||||
|
cells: [
|
||||||
|
|
||||||
|
// ══════════════ B300 FP8 low-latency — default (cells[0]) ══════════════
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: ["SGLANG_SHARED_EXPERT_TP1=1"],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-FP8",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
// ══════════════ H200 (8-GPU HGX, tp 8) ══════════════
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--mem-fraction-static 0.80",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend fa3",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: ["SGLANG_SHARED_EXPERT_TP1=1"],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: ["SGLANG_SHARED_EXPERT_TP1=1"],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend fa3",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-FP8",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "int4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "int4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend fa3",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-INT4",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
// ══════════════ B300 (8-GPU HGX, tp 8) ══════════════
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: ["SGLANG_SHARED_EXPERT_TP1=1"],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-NVFP4",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "int4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "int4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 8",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-INT4",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
// ══════════════ GB300 (4-GPU single node, tp 4) ══════════════
|
||||||
|
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: ["SGLANG_SHARED_EXPERT_TP1=1"],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: ["SGLANG_SHARED_EXPERT_TP1=1"],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-FP8",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-NVFP4",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "int4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "int4", strategy: "low-latency", nodes: "single" },
|
||||||
|
verified: true,
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path {{MODEL_NAME}}",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--reasoning-parser poolside_v1",
|
||||||
|
"--tool-call-parser poolside_v1",
|
||||||
|
"--tp 4",
|
||||||
|
"--attention-backend trtllm_mha",
|
||||||
|
"--speculative-algorithm DFLASH",
|
||||||
|
"--speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-INT4",
|
||||||
|
"--page-size 1",
|
||||||
|
"--mem-fraction-static 0.7",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user