diff --git a/docs_new/cards/logos/meituan.png b/docs_new/cards/logos/meituan.png new file mode 100644 index 000000000..a7e1edd80 Binary files /dev/null and b/docs_new/cards/logos/meituan.png differ diff --git a/docs_new/cookbook/autoregressive/Meituan/LongCat-2.0.mdx b/docs_new/cookbook/autoregressive/Meituan/LongCat-2.0.mdx new file mode 100644 index 000000000..d1b8fd5a0 --- /dev/null +++ b/docs_new/cookbook/autoregressive/Meituan/LongCat-2.0.mdx @@ -0,0 +1,197 @@ +--- +title: LongCat-2.0 +description: "Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +```bash Command +pip install --upgrade pip +pip install uv + +# Choose the nightly wheel index for your CUDA runtime. +SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130 # B300 / CUDA 13 +# SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129 # CUDA 12.9 + +uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]" +``` + +Then run the **Python** output of the command panel below in that environment. + + + + + +```bash Command +# Choose the rolling nightly image for your hardware. +SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13 # B300 / CUDA 13 +# SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev # Other supported hardware + +docker pull "${SGLANG_DOCKER_IMAGE}" +``` + +For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy: + +- **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/meituan-longcat/longcat-2.0.jsx"; +import { benchmarks } from "/src/snippets/configs/meituan-longcat/longcat-2.0-benchmarks.jsx"; + + + + + The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles. + + +## Playground + +The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +[LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently. + + + + + + + + + + + + + + + + +
ModelArchitectureServing precision
LongCat-2.0-FP8Sparse MoE · LongCat Sparse Attention · n-gram embeddingFP8 weights, BF16 KV cache
+ +**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8). + +## 2. Configuration Tips + +- **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint. +- **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you. +- **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path. +- **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster. +- **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time. +- **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout. +- **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe. + +## 3. Advanced Usage + +### 3.1 Test the deployment + + + +```bash Command +curl http://localhost:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "meituan-longcat/LongCat-2.0-FP8", + "messages": [ + {"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."} + ], + "max_tokens": 32, + "chat_template_kwargs": {"enable_thinking": false} + }' +``` + + + + + +```text Output +15 +``` + + + + + +```python Example +from openai import OpenAI + +client = OpenAI( + base_url="http://localhost:30000/v1", + api_key="EMPTY", +) + +response = client.chat.completions.create( + model="meituan-longcat/LongCat-2.0-FP8", + messages=[ + { + "role": "user", + "content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.", + } + ], + max_tokens=32, + extra_body={"chat_template_kwargs": {"enable_thinking": False}}, +) + +print(response.choices[0].message.content) +``` + + + + + +```text Output +15 +``` + + + +## 4. Validation + +The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above. + + + + + + + + + + + + + + + + + + + + + +
EvaluationExamplesAccuracy
GSM8K20098.0%
GSM8K131495.8904109589041%
+ +CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation. diff --git a/docs_new/cookbook/autoregressive/intro.mdx b/docs_new/cookbook/autoregressive/intro.mdx index 03737967d..783331a60 100644 --- a/docs_new/cookbook/autoregressive/intro.mdx +++ b/docs_new/cookbook/autoregressive/intro.mdx @@ -31,6 +31,12 @@ metatags: href="/cookbook/autoregressive/GLM/GLM-5.2" img="/cards/logos/glm.png" /> + " }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + // Reproduce commands for the Benchmark card's "Reproduce" modal. + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --random-range-ratio 1.0 \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --warmup-requests 64 --flush-cache`, + accuracy: { + gsm8k_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gsm8k \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --num-threads 32`, + }, + numPromptsByConc: { 1: 8, 16: 64, 64: 128, 256: 512, 1024: 2048 }, + }, + + accuracyLabels: [ + ["gsm8k_pct", "GSM8K", "%"], + ], + + dockerImages: { + b300: "lmsysorg/sglang:dev-cu13", + b200: "lmsysorg/sglang:dev", + h200: "lmsysorg/sglang:dev", + h20: "lmsysorg/sglang:dev", + }, + + github: { + cookbookModel: "meituan-longcat/LongCat-2.0-FP8", + }, + + playgroundFeatures: { + attention: { + knobs: [ + { id: "tp", label: "TP", values: [ + null, + 8, + { value: 16, disable: { nodes: ["single"] }, + disableReason: "TP=16 requires 16 ranks - switch the Deploy panel's Nodes to Multi-Nodes first." }, + ]}, + ], + }, + + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + { id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] }, + ], + }, + ep: { label: "EP", values: [ + null, + 8, + { value: 16, disable: { nodes: ["single"] }, + disableReason: "EP=16 requires 16 ranks - switch the Deploy panel's Nodes to Multi-Nodes first." }, + ]}, + }, + + hicache: { + backends: [ + { id: null, label: "Auto" }, + { id: "file", label: "File" }, + { id: "mooncake", label: "Mooncake" }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + ], + }, + }, + + cells: [ + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--ep 8", + "--max-running-requests 64", + "--mem-fraction-static 0.92", + "--chunked-prefill-size 2048", + "--nsa-prefill-backend fa3", + "--kv-cache-dtype bfloat16", + "--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--ep 16", + "--max-running-requests 64", + "--mem-fraction-static 0.92", + "--chunked-prefill-size 2048", + "--nsa-prefill-backend fa3", + "--kv-cache-dtype bfloat16", + "--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--ep 16", + "--max-running-requests 64", + "--mem-fraction-static 0.92", + "--chunked-prefill-size 2048", + "--nsa-prefill-backend fa3", + "--kv-cache-dtype bfloat16", + "--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h20", variant: "default", quant: "fp8", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--ep 16", + "--max-running-requests 64", + "--mem-fraction-static 0.92", + "--chunked-prefill-size 2048", + "--nsa-prefill-backend fa3", + "--kv-cache-dtype bfloat16", + "--model-loader-extra-config '{\"enable_multithread_load\":true,\"num_threads\":12}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};