[Doc] Cookbook: Laguna-XS-2.1 (DFlash low-latency + high-throughput) (#29884)
This commit is contained in:
Jimmy Shong
@@ -0,0 +1,241 @@
|
||||
---
|
||||
title: Laguna-XS-2.1
|
||||
description: "Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-XS-2.1 support is fully merged to SGLang `main` ([PR #29446](https://github.com/sgl-project/sglang/pull/29446): DFlash speculative decoding + shared-expert fix; [PR #29761](https://github.com/sgl-project/sglang/pull/29761): INT4 loader fix). Any build at or past their merge covers every cell below.
|
||||
|
||||
The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-cu13-laguna-xs-2-1
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:
|
||||
|
||||
- **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
|
||||
- **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.
|
||||
|
||||
On the 8-GPU HGX platforms (H200 / B300), BF16 and NVFP4 run plain `--tp 8`; FP8 and INT4 run `--tp 8 --ep-size 8` because their quantization scales cannot shard the MoE 8-way (see [Configuration Tips](#2-configuration-tips)). The 4-GPU GB300 node runs plain `--tp 4` throughout.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-xs21.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-xs21-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is an open-weight **33B-parameter** hybrid sliding-window-attention MoE model (**~3B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering — the extra-small sibling of [Laguna-M.1](./Laguna-M.1).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Sparse MoE**: 40 layers, 256 routed experts, top-8 routing.
|
||||
- **Hybrid attention**: 30 sliding-window layers (window 512) interleaved with 10 full-attention layers; 48 Q / 8 KV heads.
|
||||
- **Long context**: 262,144 tokens (RoPE + YaRN on the full-attention layers).
|
||||
- **DFlash drafts**: matched draft models (5-layer, ~0.9 GB) ship per quantization for low-latency serving.
|
||||
- **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.
|
||||
|
||||
**Available quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Target model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Draft model</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1`](https://huggingface.co/poolside/Laguna-XS-2.1)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-FP8)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>INT4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-INT4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The drafts themselves are small bf16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-XS-2.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [API platform](https://platform.poolside.ai).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Attention backend**
|
||||
|
||||
Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Never use `triton` attention with Laguna (GSM8K 13%).
|
||||
|
||||
**Quantized checkpoints cap plain TP at 4**
|
||||
|
||||
`moe_intermediate_size=512` with FP8 block `[128,128]` / INT4 `group_size=128` scales cannot shard 8-way (512/8 = 64 < 128 granularity): FP8 fails at weight creation, INT4 crashes in the Marlin kernel, on any hardware. The generated 8-GPU FP8/INT4 commands therefore use `--tp 8 --ep-size 8` — expert parallelism keeps whole experts per rank, using all 8 GPUs on one instance. FP8 additionally needs `SGLANG_SHARED_EXPERT_TP1=1` (its shared expert is also block-quantized; INT4's stays bf16). Alternatives: plain `--tp 4`, or `--tp 4 --dp-size 2`. Accuracy is parallelism-independent within eval noise (verified tp1 ≡ tp4 on GB300 and tp4 ≡ tp8+ep8 on H200).
|
||||
|
||||
**DFlash memory**
|
||||
|
||||
Low-latency cells carry `--mem-fraction-static 0.7`: the default fraction OOMs in the draft vocab all-gather at `tp 4` on GB300. Dense cells use the default heuristic.
|
||||
|
||||
**INT4 is mixed-precision**
|
||||
|
||||
The INT4 checkpoint quantizes MoE layers in mixed 4-bit / 8-bit config groups. Builds older than [PR #29761](https://github.com/sgl-project/sglang/pull/29761) crash at load with `KeyError: 'Linear'`.
|
||||
|
||||
**Chat template**
|
||||
|
||||
On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ ~5.8) the `{% include %}` stub in `tokenizer_config.json` cannot resolve and the server silently falls back to a generic template — pass `--chat-template <model-dir>/chat_template.jinja` explicitly there.
|
||||
|
||||
**Thinking**
|
||||
|
||||
Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.
|
||||
|
||||
**Served model id**
|
||||
|
||||
The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-XS-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 DFlash Speculative Decoding
|
||||
|
||||
DFlash is a block-wise speculative decoder: the 5-layer draft proposes a block of tokens and the target verifies the whole block in one forward pass, so only target-approved tokens are emitted — output quality is the target's by construction (GSM8K matches dense within noise on every quantization). The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step:
|
||||
|
||||
- Measured ~6 tokens/step at `tp 1`, ~4 at `tp 4` (greedy GSM8K, matched-precision pairs; ~3 under mixed reasoning-heavy traffic; FP8 reached 6.75 at `tp 8 + ep 8` on H200) — versus 1 token/step dense.
|
||||
- Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput.
|
||||
- The generated commands always pair the draft calibrated for the selected target precision.
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
XS-2.1 is an extra-small model — give it generous `max_tokens` when thinking is enabled
|
||||
(hard problems regularly reason for thousands of tokens), and keep thinking off for
|
||||
short-form tasks.
|
||||
</Note>
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -142,7 +142,7 @@ metatags:
|
||||
<Card
|
||||
title="Poolside"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Poolside/Laguna-M.1"
|
||||
href="/cookbook/autoregressive/Poolside/Laguna-XS-2.1"
|
||||
img="/cards/logos/poolside.png"
|
||||
/>
|
||||
</CardGroup>
|
||||
|
||||
Reference in New Issue
Block a user