[Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c949e91f18
commit
b819d2fb5b
@@ -0,0 +1,322 @@
|
||||
---
|
||||
title: Laguna-M.1
|
||||
description: "Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4."
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-M.1 support is already on SGLang `main` — **softplus per-element attention-output gating** ([PR #28400](https://github.com/sgl-project/sglang/pull/28400)) and a **global-attention fix** ([PR #28604](https://github.com/sgl-project/sglang/pull/28604), since M.1 is full-attention `sliding_window: 0`) — but not yet in a tagged release. The two paths below match the **Python / Docker** toggle in the command panel: install from `main` (Python tab), or use the **Docker** image, which bundles the same build (CUDA 13, covers H200 + all Blackwell). The model ships custom config code on the Hub, so `--trust-remote-code` is required (it is included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
# Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a
|
||||
# tagged release — install from main. The serving runtime is in the base dependencies, no extra needed:
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — `lmsysorg/sglang:latest` bundles the CUDA-13 runtime and the M.1 code.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# CUDA 13 — covers H200 + all Blackwell:
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization to generate the launch command. Laguna-M.1 ships a single **Balanced** recipe per cell — poolside's recommended operating point, a good speed/throughput trade-off for typical multi-user serving. The 8-GPU HGX platforms (H200 / B200 / B300) use `--tp 8`; the 4-GPU Grace-Blackwell single nodes (GB200 / GB300) use `--tp 4`.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-m1.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-m1-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs (parsers, DP-Attention, DeepEP / EP) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-M.1](https://huggingface.co/poolside/Laguna-M.1) is an open-weight, **225B-parameter** Mixture-of-Experts model (**23B activated per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software-engineering work. It is released under Apache 2.0.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts, top-16 routing** (+1 shared expert) and auxiliary-loss-free load balancing.
|
||||
- **Global attention with output gating**: global attention across all layers, 64 Q-heads / 8 KV-heads (head dim 128), with **softplus attention output gating** (requires [PR #28400](https://github.com/sgl-project/sglang/pull/28400)).
|
||||
- **Long context**: 262,144 tokens, RoPE with YaRN.
|
||||
- **Agentic coding**: competitive on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0.
|
||||
- **Native reasoning**: interleaved thinking between tool calls, toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.
|
||||
|
||||
**Available Quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "80%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Quantization</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Hugging Face path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1`](https://huggingface.co/poolside/Laguna-M.1)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1-FP8`](https://huggingface.co/poolside/Laguna-M.1-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-M.1-NVFP4`](https://huggingface.co/poolside/Laguna-M.1-NVFP4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-M.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Trust remote code** (`--trust-remote-code`): Laguna-M.1 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model.
|
||||
- **Long-context memory**: M.1 is global-attention (no sliding-window), so the 262,144-token KV cache is large. If you hit OOM at full context, lower `--mem-fraction-static` or cap `--context-length`.
|
||||
- **FP8**: On **Blackwell** the recipe adds `--fp8-gemm-backend triton` — the compressed-tensors block-FP8 weight scales aren't UE8M0-packed, so the default DeepGEMM path emits garbage on Blackwell (sm_100); the Triton backend is correct (~19% slower). Temporary workaround pending [PR #28662](https://github.com/sgl-project/sglang/pull/28662) (which fixes the scales and restores the DeepGEMM fast path). On **Hopper (H200)** FP8 uses DeepGEMM with no extra flag — pre-warm its multi-session JIT with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-M.1-FP8` to avoid paying it on each restart.
|
||||
- **Parsers** (`poolside_v1`): for agentic / tool-using deployments enable the **Reasoning Parser** and **Tool Call Parser** in the Playground above — they emit `--reasoning-parser poolside_v1` (thinking → `reasoning_content`) and `--tool-call-parser poolside_v1` (structured `tool_calls`).
|
||||
- **Thinking default**: thinking is **off by default**; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
- **Served model id**: the server registers the model under whatever you pass to `--model-path`, so a client's `model` field must match it — `poolside/Laguna-M.1` (BF16) or `poolside/Laguna-M.1-FP8` / `-NVFP4` for the quantized cells. The §3 examples use the BF16 id; swap in the id you launched.
|
||||
- **Recommended sampling**: poolside benchmarks M.1 at `temperature=1.0`, `top_k=20` with thinking enabled. These are per-request sampling params (not launch flags) — e.g. `temperature=1.0, extra_body={"top_k": 20}` on the OpenAI client.
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (or toggle **Reasoning Parser** in the **Parsers** card of the Playground above). Reasoning is **opt-in**: the Laguna chat template gates it on `enable_thinking=True` (passed via `chat_template_kwargs`) — the generic `thinking` key is ignored. The `<think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-M.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
=============== Reasoning ===============
|
||||
Okay, so I need to find out what 15% of 240 is. Hmm, percentages can sometimes be
|
||||
tricky, but let me think. I remember that "percent" means per hundred, right? So 15%
|
||||
is the same as 15 per 100 or 15/100. Maybe I can convert that percentage into a decimal
|
||||
first? ... 15 divided by 100 is 0.15. ... Now, to find 15% of 240, I just need to
|
||||
multiply 240 by 0.15. ... 240 times 0.1 is 24 (10% of 240), and 240 times 0.05 is 12
|
||||
(half of that), so 24 + 12 = 36.
|
||||
[… verifies the same result several more ways: 15/100 × 240, 240 × 15 ÷ 100,
|
||||
1% × 15, and the fraction 3/20 × 240 — all give 36 …]
|
||||
So ... all methods are pointing to 36. I'm pretty confident that 15% of 240 is 36.
|
||||
=============== Answer ==================
|
||||
To find 15% of 240, convert the percentage to a decimal (0.15) and multiply by 240:
|
||||
**240 × 0.15 = 36**.
|
||||
|
||||
**Step-by-Step Explanation:**
|
||||
1. **Convert 15% to a decimal:** 15% = 15/100 = 0.15.
|
||||
2. **Multiply by 240:**
|
||||
- Break it down:
|
||||
- 10% of 240 = 24 (since 240 × 0.1 = 24).
|
||||
- 5% of 240 = 12 (half of 24).
|
||||
- Add them: 24 + 12 = **36**.
|
||||
|
||||
**Answer:** 15% of 240 is **36**.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
Laguna-M.1's reasoning traces are long — the model explores and re-verifies an answer
|
||||
multiple ways. Give it a generous `max_tokens` for harder problems (reasoning regularly
|
||||
exceeds 3k tokens). The trace above is abbreviated; the model emits it in full.
|
||||
</Note>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (or toggle **Tool Call Parser** in the **Parsers** card of the Playground above). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (`enable_thinking=False`, the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-M.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 Prefill-Decode (PD) Disaggregation
|
||||
|
||||
[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. Laguna-M.1 is **global-attention with a standard KV cache** (no sliding window, no sparse "index" side-buffer), so its KV pages transfer with **no model-specific flags** — just the `--disaggregation-*` knobs. Both roles auto-select the same attention backend (FlashAttention-3) and page size because they share the model and flags, so the KV layout lines up for transfer.
|
||||
|
||||
**Supported / validated topology:**
|
||||
|
||||
- **Equal tensor parallelism** — prefill and decode run the same `--tp`.
|
||||
- **Single pipeline stage** — PP = 1 (the default).
|
||||
- **mooncake or NIXL** transfer backend over RDMA / InfiniBand.
|
||||
- Validated on **2 × 8×H200** (TP8 prefill + TP8 decode, BF16), one node each, over an 8× 400 Gb/s NDR InfiniBand fabric.
|
||||
|
||||
Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Point `--disaggregation-ib-device` at your RDMA NIC(s).
|
||||
|
||||
```bash Prefill server (node A)
|
||||
sglang serve \
|
||||
--model-path poolside/Laguna-M.1 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--tp 8 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--disaggregation-bootstrap-port 8998
|
||||
```
|
||||
|
||||
```bash Decode server (node B)
|
||||
sglang serve \
|
||||
--model-path poolside/Laguna-M.1 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser poolside_v1 \
|
||||
--tool-call-parser poolside_v1 \
|
||||
--tp 8 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:
|
||||
|
||||
```bash PD router
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://<prefill-host>:30000 8998 \
|
||||
--decode http://<decode-host>:30001 \
|
||||
--policy round_robin \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Clients hit the router exactly like a single server — it splits each request across the two stages transparently:
|
||||
|
||||
<Accordion title="PD Client Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-M.1",
|
||||
messages=[{"role": "user", "content": "What is 2 + 2?"}],
|
||||
max_tokens=64,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
2 + 2 = 4
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
**Transfer backend — mooncake (recommended).** mooncake honors `--disaggregation-ib-device` and establishes its RDMA connection at registration, so the **first request is already fast** (no cold start). It works with a single NIC or all eight; using **all 8 NICs lowers TTFT** (more aggregate bandwidth for the KV payload — the gap widens at longer context). On 8×H200 (random isl=512 / osl=256, 16 concurrent) it served ≈ **717 tok/s** output (≈ 2.2k tok/s total), mean **TTFT 244 ms**, mean **TPOT 17.7 ms**; with a single `mlx5_0` NIC, ≈ 697 tok/s and TTFT 287 ms (TPOT unchanged — decode is compute-bound).
|
||||
|
||||
**Transfer backend — NIXL (works, with two caveats).**
|
||||
|
||||
<Warning>
|
||||
The NIXL path **ignores `--disaggregation-ib-device`** — that flag is mooncake-only. NIXL uses its UCX backend, whose NIC is selected by the **`UCX_NET_DEVICES`** environment variable. **Set it** (e.g. `export UCX_NET_DEVICES=mlx5_0:1`) on both servers; without it UCX cannot establish a working cross-node path and every KV transfer hangs until it hits the 300 s timeout (`Request … timed out … in KVPoll.WaitingForInput`) and returns a 500.
|
||||
</Warning>
|
||||
|
||||
With `UCX_NET_DEVICES` pinned, NIXL matches mooncake on quality and steady-state speed (≈ 720 tok/s, TTFT 230 ms, TPOT 17.7 ms). One difference: the **first request after launch pays a ~38 s one-time UCX connection cold-start** (a single port or all eight behave the same). Warm the path with one throwaway request after startup, or raise `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` (default 300 s) so the first real request isn't dropped while UCX connects.
|
||||
|
||||
**Validation.** PD disaggregation preserves output quality — disaggregated output matches non-disaggregated serving, and GSM8K (no-thinking, 200-question subset via the router) scored **0.945** (mooncake, 8 NICs) / **0.940** (NIXL) / **0.950** (mooncake, 1 NIC), all with 100% stop-rate and 0% errors — in line with single-node BF16 (≈ 0.93 on the full split). Logs confirm the split: the prefill node logs `Prefill batch` (CUDA graph off), the decode node logs `Decode batch` (CUDA graph on).
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
title: Laguna-S-2.1
|
||||
description: "Deploy poolside's Laguna-S-2.1 — a 118B hybrid-SWA Mixture-of-Experts model (8B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-S-2.1 uses the same `laguna` model architecture as [Laguna-XS-2.1](./Laguna-XS-2.1), which is fully supported in SGLang `main`. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:
|
||||
|
||||
- **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
|
||||
- **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.
|
||||
|
||||
On the 8-GPU HGX platforms (H200 / B300) all quantizations run `--tp 8`. The 4-GPU GB300 node runs `--tp 4` throughout. NVFP4 is Blackwell-only (B300 / GB300 only).
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-s21.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-s21-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is an open-weight **118B-parameter** hybrid sliding-window-attention MoE model (**~8B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering. It sits between [Laguna XS 2.1](./Laguna-XS-2.1) (33B/3B active) and Laguna M.1 (222B/23B active) in the Laguna family.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Sparse MoE**: 48 layers, 256 routed experts, top-10 routing, plus 1 shared expert.
|
||||
- **Hybrid attention**: 36 sliding-window layers (window 512) interleaved with 12 full-attention layers (1:3 global-to-SWA ratio); 8 KV heads, head dim 128; per-head sigmoid output gating with per-layer-type rotary scales.
|
||||
- **Long context**: 1,048,576 tokens.
|
||||
- **DFlash drafts**: matched draft models ship per quantization for low-latency serving.
|
||||
- **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.
|
||||
|
||||
**Available quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Target model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Draft model</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1`](https://huggingface.co/poolside/Laguna-S-2.1)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-FP8)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>INT4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-INT4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-S-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The drafts are small BF16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).
|
||||
|
||||
**License:** [OpenMDW-1.1](https://openmdw.ai/)
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-S-2.1) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Attention backend**
|
||||
|
||||
Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (reproduced on Laguna-XS-2.1, greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Other attention backend choices have not been fully validated on Laguna; keep the default.
|
||||
|
||||
**BF16 memory on H200**
|
||||
|
||||
BF16 on H200 leaves less headroom for CUDA-graph capture and NCCL allocations than FP8/INT4. The High-throughput BF16 command carries `--mem-fraction-static 0.80`. FP8, INT4, and all B300/GB300 cells use the default heuristic.
|
||||
|
||||
**FP8 shared expert**
|
||||
|
||||
`SGLANG_SHARED_EXPERT_TP1=1` is required for FP8 cells on **all hardware** — confirmed on both H200 (TP=8) and GB300 (TP=4). The FP8 checkpoint block-quantizes the shared expert (128×128 scales), which cannot TP-shard cleanly at either TP degree on S-2.1. This env var replicates the shared expert instead of sharding it. INT4 keeps the shared expert in BF16 (no flag needed); BF16 is unquantized. Note: this differs from Laguna-XS-2.1 where TP=4 does not require the flag — the constraint is architecture-specific.
|
||||
|
||||
**FP8 and NVFP4 DFlash drafts**
|
||||
|
||||
Fixed upstream on 2026-07-21: all DFlash draft configs now use a flat top-level `rope_theta` (the `rope_parameters` block was removed). If a server crashes at draft-model load with `KeyError: 'rope_theta'`, you are serving a draft checkpoint cached before 2026-07-21 — re-download it (e.g. `hf download poolside/Laguna-S-2.1-DFlash-FP8`) to pick up the corrected config.
|
||||
|
||||
**DFlash memory**
|
||||
|
||||
Low-latency cells carry `--mem-fraction-static 0.7` (sufficient even for BF16 on H200). Dense cells use the default heuristic (except BF16 on H200 — see above).
|
||||
|
||||
**BF16 reasoning length**
|
||||
|
||||
BF16 reasons approximately 2× longer than FP8/INT4 on AIME25 (median 34.8 k vs 16.9 k tokens), consistently truncating at `max_tokens=64000`. FP8/INT4 truncate at ≈ 2%. For a valid BF16 AIME25 score, serve with `max_tokens ≥ 131072` (the model supports a 1 M context window).
|
||||
|
||||
**Chat template**
|
||||
|
||||
On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ ~5.8) pass `--chat-template <model-dir>/chat_template.jinja` explicitly.
|
||||
|
||||
**Thinking**
|
||||
|
||||
Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.
|
||||
|
||||
**Served model id**
|
||||
|
||||
The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-S-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 DFlash Speculative Decoding
|
||||
|
||||
DFlash is a block-wise speculative decoder: the draft proposes a block of tokens and the target verifies the whole block in one forward pass — output quality is the target's by construction. The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step.
|
||||
|
||||
Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. The generated commands always pair the draft calibrated for the selected target precision.
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-S-2.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=4096,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
Give generous `max_tokens` when thinking is enabled — hard problems regularly reason
|
||||
for thousands of tokens. Keep thinking off for short-form tasks.
|
||||
</Note>
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-S-2.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
title: Laguna-XS-2.1
|
||||
description: "Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4."
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Laguna-XS-2.1 support is fully merged to SGLang `main` ([PR #29446](https://github.com/sgl-project/sglang/pull/29446): DFlash speculative decoding + shared-expert fix; [PR #29761](https://github.com/sgl-project/sglang/pull/29761): INT4 loader fix). Any build at or past their merge covers every cell below.
|
||||
|
||||
The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:
|
||||
|
||||
- **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
|
||||
- **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.
|
||||
|
||||
On the 8-GPU HGX platforms (H200 / B300), BF16 and NVFP4 run plain `--tp 8`; FP8 and INT4 run `--tp 8 --ep-size 8` because their quantization scales cannot shard the MoE 8-way (see [Configuration Tips](#2-configuration-tips)). The 4-GPU GB300 node runs plain `--tp 4` throughout.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/poolside/laguna-xs21.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/poolside/laguna-xs21-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is an open-weight **33B-parameter** hybrid sliding-window-attention MoE model (**~3B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering — the extra-small sibling of [Laguna-M.1](./Laguna-M.1).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Sparse MoE**: 40 layers, 256 routed experts, top-8 routing.
|
||||
- **Hybrid attention**: 30 sliding-window layers (window 512) interleaved with 10 full-attention layers; 48 Q / 8 KV heads.
|
||||
- **Long context**: 262,144 tokens (RoPE + YaRN on the full-attention layers).
|
||||
- **DFlash drafts**: matched draft models (5-layer, ~0.9 GB) ship per quantization for low-latency serving.
|
||||
- **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.
|
||||
|
||||
**Available quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
<col style={{width: "43%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Target model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Draft model</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1`](https://huggingface.co/poolside/Laguna-XS-2.1)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-FP8)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><strong>INT4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-INT4)</td>
|
||||
<td style={{padding: "9px 12px"}}>[`poolside/Laguna-XS-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The drafts themselves are small bf16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-XS-2.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [API platform](https://platform.poolside.ai).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Attention backend**
|
||||
|
||||
Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Never use `triton` attention with Laguna (GSM8K 13%).
|
||||
|
||||
**Quantized checkpoints cap plain TP at 4**
|
||||
|
||||
`moe_intermediate_size=512` with FP8 block `[128,128]` / INT4 `group_size=128` scales cannot shard 8-way (512/8 = 64 < 128 granularity): FP8 fails at weight creation, INT4 crashes in the Marlin kernel, on any hardware. The generated 8-GPU FP8/INT4 commands therefore use `--tp 8 --ep-size 8` — expert parallelism keeps whole experts per rank, using all 8 GPUs on one instance. FP8 additionally needs `SGLANG_SHARED_EXPERT_TP1=1` (its shared expert is also block-quantized; INT4's stays bf16). Alternatives: plain `--tp 4`, or `--tp 4 --dp-size 2`. Accuracy is parallelism-independent within eval noise (verified tp1 ≡ tp4 on GB300 and tp4 ≡ tp8+ep8 on H200).
|
||||
|
||||
**DFlash memory**
|
||||
|
||||
Low-latency cells carry `--mem-fraction-static 0.7`: the default fraction OOMs in the draft vocab all-gather at `tp 4` on GB300. Dense cells use the default heuristic.
|
||||
|
||||
**INT4 is mixed-precision**
|
||||
|
||||
The INT4 checkpoint quantizes MoE layers in mixed 4-bit / 8-bit config groups. Builds older than [PR #29761](https://github.com/sgl-project/sglang/pull/29761) crash at load with `KeyError: 'Linear'`.
|
||||
|
||||
**Chat template**
|
||||
|
||||
On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ ~5.8) the `{% include %}` stub in `tokenizer_config.json` cannot resolve and the server silently falls back to a generic template — pass `--chat-template <model-dir>/chat_template.jinja` explicitly there.
|
||||
|
||||
**Thinking**
|
||||
|
||||
Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.
|
||||
|
||||
**Served model id**
|
||||
|
||||
The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-XS-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 DFlash Speculative Decoding
|
||||
|
||||
DFlash is a block-wise speculative decoder: the 5-layer draft proposes a block of tokens and the target verifies the whole block in one forward pass, so only target-approved tokens are emitted — output quality is the target's by construction (GSM8K matches dense within noise on every quantization). The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step:
|
||||
|
||||
- Measured ~6 tokens/step at `tp 1`, ~4 at `tp 4` (greedy GSM8K, matched-precision pairs; ~3 under mixed reasoning-heavy traffic; FP8 reached 6.75 at `tp 8 + ep 8` on H200) — versus 1 token/step dense.
|
||||
- Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput.
|
||||
- The generated commands always pair the draft calibrated for the selected target precision.
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
XS-2.1 is an extra-small model — give it generous `max_tokens` when thinking is enabled
|
||||
(hard problems regularly reason for thousands of tokens), and keep thinking off for
|
||||
short-form tasks.
|
||||
</Note>
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS-2.1",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,323 @@
|
||||
---
|
||||
title: Laguna-XS.2
|
||||
metatags:
|
||||
description: "Deploy Poolside's Laguna-XS.2 hybrid SWA + MoE model with SGLang on NVIDIA H200 / B200 — agentic coding with hybrid reasoning and tool calling."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) is an open-source hybrid sliding-window-attention MoE model from [Poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering work.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **MoE**: 33.4B total parameters, 3.0B active per token, 256 routed experts (top-8) plus 1 shared.
|
||||
- **Long context**: 131,072 tokens.
|
||||
- **Agentic coding**: Tuned for tool-using software engineering agents and long-horizon execution.
|
||||
- **Hybrid reasoning**: `<think>...</think>` segments toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.
|
||||
|
||||
**Available Quantizations:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "80%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Hugging Face path</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[`poolside/Laguna-XS.2`](https://huggingface.co/poolside/Laguna-XS.2)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[`poolside/Laguna-XS.2-FP8`](https://huggingface.co/poolside/Laguna-XS.2-FP8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[`poolside/Laguna-XS.2-NVFP4`](https://huggingface.co/poolside/Laguna-XS.2-NVFP4)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** Apache 2.0
|
||||
|
||||
For details, see the [Hugging Face model card](https://huggingface.co/poolside/Laguna-XS.2) and the [Laguna deeper-dive blog post](https://poolside.ai/blog/laguna-a-deeper-dive).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Laguna-XS.2 support is on `main` but not yet in a tagged release; install from the SGLang nightly wheel index, or pull a pre-built Docker image:
|
||||
|
||||
```bash Command
|
||||
# Install SGLang via pip (CUDA 13) — requires Python 3.10 (nightly wheels are cp310 only)
|
||||
python3 -m pip install --upgrade pip
|
||||
python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu130 \
|
||||
"sglang[all]==0.5.12.dev20260509+g096ad02b0"
|
||||
|
||||
# CUDA 12: swap to the cu129 index
|
||||
python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu129 \
|
||||
"sglang[all]==0.5.12.dev20260509+g096ad02b0"
|
||||
|
||||
# Or use Docker (multi-arch amd64/arm64; CUDA 13, H200 / B200)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for your hardware.
|
||||
|
||||
import { LagunaXS2Deployment } from '/src/snippets/autoregressive/laguna-xs2-deployment.jsx';
|
||||
|
||||
<LagunaXS2Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Trust remote code** (`--trust-remote-code`): Laguna-XS.2 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model.
|
||||
- **Quantization**: NVFP4 requires Blackwell (B200 / B300); BF16 and FP8 run on either H200 or B200. FP8's first launch triggers a multi-session DeepGEMM JIT pre-compile (~10-20 min); pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-XS.2-FP8` to avoid that cost on every restart.
|
||||
- **Reasoning parser** (`--reasoning-parser poolside_v1`): Splits `<think>...</think>` segments into `reasoning_content` so `content` holds only the final answer. Disable only if you want the raw `<think>` tags in `content`.
|
||||
- **Tool call parser** (`--tool-call-parser poolside_v1`): Required for OpenAI-compatible tool-call streaming. Disable only for chat-only deployments.
|
||||
- **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--dp <N> --enable-dp-attention` with `--dp` matching `--tp` (tune independently if needed).
|
||||
- **Thinking default**: Thinking is **off by default** at the model level. Opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
The samples below assume the server is reachable at `http://localhost:30000/v1`.
|
||||
|
||||
### 4.1 Basic Chat
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is the difference between TCP and UDP?"}
|
||||
],
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are two core protocols of the Internet Protocol (IP) suite, both used for network communication but with key differences:
|
||||
|
||||
## Connection Handling
|
||||
- **TCP**: Connection-oriented protocol that establishes a connection before data transfer (like a phone call)
|
||||
- **UDP**: Connectionless protocol that sends data without establishing a connection (like sending a letter)
|
||||
|
||||
## Reliability
|
||||
- **TCP**: Guaranteed delivery with error checking, retransmission of lost packets, and flow control
|
||||
- **UDP**: No guarantee of delivery; packets may be lost, duplicated, or arrive out of order
|
||||
|
||||
## Speed & Overhead
|
||||
- **TCP**: Slower due to connection setup, acknowledgment overhead, and error correction mechanisms
|
||||
- **UDP**: Faster with minimal overhead since it doesn't wait for acknowledgments or retransmit lost data
|
||||
|
||||
## Use Cases
|
||||
- **TCP**: Web browsing (HTTP/HTTPS), email (SMTP), file transfers (FTP), database connections
|
||||
- **UDP**: Video streaming, online gaming, VoIP calls, DNS queries, live broadcasts
|
||||
|
||||
In essence, TCP prioritizes reliability over speed, while UDP prioritizes speed over reliability.
|
||||
```
|
||||
|
||||
### 4.2 Reasoning (Thinking Mode)
|
||||
|
||||
Laguna-XS.2 emits reasoning between `<think>...</think>` tags. The `--reasoning-parser poolside_v1` flag separates the thinking text into `reasoning_content` so `content` holds only the final answer. Thinking is opt-in per request:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[
|
||||
{"role": "user", "content": "If a train travels at 60 km/h for 2.5 hours, how far does it go?"}
|
||||
],
|
||||
max_tokens=4096,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
|
||||
print("====== Reasoning Content ======")
|
||||
print(resp.choices[0].message.reasoning_content)
|
||||
print("====== Answer ======")
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
====== Reasoning Content ======
|
||||
The user is asking a straightforward math problem about distance, speed, and time. I need to calculate the distance using the formula:
|
||||
|
||||
Distance = Speed × Time
|
||||
|
||||
Given:
|
||||
- Speed = 60 km/h
|
||||
- Time = 2.5 hours
|
||||
|
||||
So the calculation would be:
|
||||
Distance = 60 × 2.5 = 150 km
|
||||
|
||||
This is a simple multiplication problem. I should provide a clear, direct answer and maybe explain the calculation briefly.
|
||||
|
||||
====== Answer ======
|
||||
To find the distance, use the formula:
|
||||
|
||||
Distance = Speed × Time
|
||||
Distance = 60 km/h × 2.5 h = 150 km
|
||||
|
||||
The train travels **150 kilometers**.
|
||||
```
|
||||
|
||||
To disable thinking, omit `extra_body` (off by default) or pass `chat_template_kwargs={"enable_thinking": False}` explicitly.
|
||||
|
||||
### 4.3 Tool Calling
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="poolside/Laguna-XS.2",
|
||||
messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
msg = resp.choices[0].message
|
||||
print("====== Reasoning Content ======")
|
||||
print(msg.reasoning_content)
|
||||
print("====== Content ======")
|
||||
print(msg.content)
|
||||
print("====== Tool Calls ======")
|
||||
for tc in msg.tool_calls or []:
|
||||
print(f" Function: {tc.function.name}")
|
||||
print(f" Arguments: {tc.function.arguments}")
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
====== Reasoning Content ======
|
||||
None
|
||||
====== Content ======
|
||||
|
||||
I'll check the current weather in Tokyo for you.
|
||||
|
||||
====== Tool Calls ======
|
||||
Function: get_weather
|
||||
Arguments: {"location": "Tokyo"}
|
||||
```
|
||||
|
||||
`reasoning_content` is `None` because thinking is off by default; `content` carries the brief assistant message that precedes the tool call. Add `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` if you want interleaved reasoning before the tool call.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 (4×H200)
|
||||
- Model: `poolside/Laguna-XS.2` (BF16)
|
||||
- Tensor Parallelism: 4
|
||||
- SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs)
|
||||
- Reasoning Parser: `poolside_v1`
|
||||
- Tool Call Parser: `poolside_v1`
|
||||
- Sampling: `temperature=0.6`, `max_tokens=16384`, `chat_template_kwargs={"enable_thinking": true}`, `n_repeats=1`
|
||||
- Grader: NeMo-Skills `math_verify` (math) and `eval_mcq` (multichoice)
|
||||
|
||||
**Results (from [PR #24204](https://github.com/sgl-project/sglang/pull/24204)):**
|
||||
|
||||
| Eval | Accuracy |
|
||||
| --- | ---: |
|
||||
| GPQA Diamond | 0.5556 |
|
||||
| AIME 25 | 0.5667 |
|
||||
| MMLU | 0.836 |
|
||||
| SWE-Bench Verified | 0.6540 |
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 (1×H200 for TP=1, 4×H200 for TP=4)
|
||||
- Model: `poolside/Laguna-XS.2` (BF16)
|
||||
- SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs)
|
||||
- Workload: `sglang.bench_serving --backend sglang --dataset-name random` (defaults: `--random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.0`)
|
||||
- Server flags identical to the accuracy runs above.
|
||||
|
||||
#### 5.2.1 Latency Benchmark (10 prompts, concurrency = 1)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving --backend sglang \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--dataset-name random --num-prompts 10 --max-concurrency 1
|
||||
```
|
||||
|
||||
| Metric | TP=1 | TP=4 |
|
||||
| --- | ---: | ---: |
|
||||
| Successful requests | 10 | 10 |
|
||||
| Output token throughput (tok/s) | 193.10 | 238.88 |
|
||||
| Total token throughput (tok/s) | 471.82 | 583.68 |
|
||||
| Mean TTFT (ms) | 35.32 | 24.17 |
|
||||
| Mean TPOT (ms) | 5.10 | 4.13 |
|
||||
| Median ITL (ms) | 5.14 | 4.14 |
|
||||
|
||||
#### 5.2.2 Throughput Benchmark (1000 prompts, concurrency = 100)
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving --backend sglang \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--dataset-name random --num-prompts 1000 --max-concurrency 100
|
||||
```
|
||||
|
||||
| Metric | TP=1 | TP=4 |
|
||||
| --- | ---: | ---: |
|
||||
| Successful requests | 1000 | 1000 |
|
||||
| Request throughput (req/s) | 7.32 | 14.61 |
|
||||
| Output token throughput (tok/s) | 3739.30 | 7465.18 |
|
||||
| Peak output token throughput (tok/s) | 4718.00 | 10133.00 |
|
||||
| Total token throughput (tok/s) | 7485.82 | 14944.81 |
|
||||
| Mean TTFT (ms) | 115.17 | 68.36 |
|
||||
| Mean TPOT (ms) | 25.51 | 12.71 |
|
||||
| Median ITL (ms) | 21.31 | 10.64 |
|
||||
|
||||
TP=4 delivers roughly 2.0× total-token throughput and ~1.7× lower mean TTFT compared to TP=1 on the `cc=100` random workload.
|
||||
Reference in New Issue
Block a user