minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Doc] Add LongCat 2.0 FP8 cookbook (#30320)

Browse Source 
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
Xiaoyu Zhang
2026-07-07 11:48:13 -07:00
committed by GitHub
co-authored by Zijie Xia
parent 11cea29c90
commit ead1e490b5
6 changed files with 422 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/Meituan/LongCat-2.0.mdx
+197
View File
@@ -0,0 +1,197 @@
---
title: LongCat-2.0
description: "Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install --upgrade pip
pip install uv
# Choose the nightly wheel index for your CUDA runtime.
SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130 # B300 / CUDA 13
# SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129 # CUDA 12.9
uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]"
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
```bash Command
# Choose the rolling nightly image for your hardware.
SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13 # B300 / CUDA 13
# SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev # Other supported hardware
docker pull "${SGLANG_DOCKER_IMAGE}"
```
For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy:
- **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/meituan-longcat/longcat-2.0.jsx";
import { benchmarks } from "/src/snippets/configs/meituan-longcat/longcat-2.0-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
<Note>
The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles.
</Note>
## Playground
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model Introduction
[LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Serving precision</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/meituan-longcat/LongCat-2.0-FP8">LongCat-2.0-FP8</a></strong></td>
<td style={{padding: "9px 12px"}}>Sparse MoE · LongCat Sparse Attention · n-gram embedding</td>
<td style={{padding: "9px 12px"}}>FP8 weights, BF16 KV cache</td>
</tr>
</tbody>
</table>
**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8).
## 2. Configuration Tips
- **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint.
- **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you.
- **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path.
- **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster.
- **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time.
- **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout.
- **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe.
## 3. Advanced Usage
### 3.1 Test the deployment
<Accordion title="Chat completion example (cURL)">
```bash Command
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meituan-longcat/LongCat-2.0-FP8",
"messages": [
{"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."}
],
"max_tokens": 32,
"chat_template_kwargs": {"enable_thinking": false}
}'
```
</Accordion>
<Accordion title="Expected output">
```text Output
15
```
</Accordion>
<Accordion title="OpenAI-compatible client (Python)">
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
response = client.chat.completions.create(
model="meituan-longcat/LongCat-2.0-FP8",
messages=[
{
"role": "user",
"content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.",
}
],
max_tokens=32,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
```
</Accordion>
<Accordion title="Example output">
```text Output
15
```
</Accordion>
## 4. Validation
The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Evaluation</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Examples</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Accuracy</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px"}}>GSM8K</td>
<td style={{padding: "9px 12px"}}>200</td>
<td style={{padding: "9px 12px"}}>98.0%</td>
</tr>
<tr>
<td style={{padding: "9px 12px"}}>GSM8K</td>
<td style={{padding: "9px 12px"}}>1314</td>
<td style={{padding: "9px 12px"}}>95.8904109589041%</td>
</tr>
</tbody>
</table>
CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation.
docs_new/cookbook/autoregressive/intro.mdx
+6
View File
@@ -31,6 +31,12 @@ metatags:
href="/cookbook/autoregressive/GLM/GLM-5.2"
img="/cards/logos/glm.png"
/>
<Card
title="Meituan"
mode="card"
href="/cookbook/autoregressive/Meituan/LongCat-2.0"
img="/cards/logos/meituan.png"
/>
<Card
title="Google"
mode="card"