156 lines
5.8 KiB
Cheetah
156 lines
5.8 KiB
Cheetah
---
|
|
title: __MODEL_DISPLAY__
|
|
description: "__ONE_LINER__"
|
|
tag: NEW
|
|
---
|
|
|
|
{/* TEMPLATE — instantiate via the cookbook-add-model skill, then DELETE this banner.
|
|
(Frontmatter MUST stay the first thing in the file, so this note lives below it.)
|
|
Replace every __TOKEN__, fill the TODO prose, delete the §3 subsections your model
|
|
lacks. Tokens: __MODEL_DISPLAY__ __ONE_LINER__ __HF_ORG__ __MODEL_SLUG__ __HF_REPO__
|
|
__REASONING_PARSER__ __TOOLCALL_PARSER__. MDX rules (JSX tables, labeled fences, no
|
|
Docusaurus/@site/GitHub-alert/pipe-tables):
|
|
.claude/skills/cookbook-add-model/references/mintlify-authoring.md */}
|
|
|
|
## Deployment
|
|
|
|
<a id="install" />
|
|
|
|
<Accordion title="Install SGLang">
|
|
|
|
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
|
|
|
<Tabs>
|
|
|
|
<Tab title="Python (pip / uv)">
|
|
|
|
```bash Command
|
|
pip install --upgrade pip
|
|
pip install uv
|
|
uv pip install --prerelease=allow sglang
|
|
```
|
|
|
|
Then run the **Python** output of the command panel below in that environment.
|
|
|
|
</Tab>
|
|
|
|
<Tab title="Docker">
|
|
|
|
```bash Command
|
|
docker pull lmsysorg/sglang:latest
|
|
```
|
|
|
|
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
|
|
|
</Tab>
|
|
|
|
</Tabs>
|
|
|
|
</Accordion>
|
|
|
|
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
|
|
|
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
|
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
|
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
|
|
|
import { Deployment } from "/src/snippets/_deployment.jsx";
|
|
import { config } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx";
|
|
import { benchmarks } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__-benchmarks.jsx";
|
|
|
|
<Deployment config={config} benchmarks={benchmarks} />
|
|
|
|
## Playground
|
|
|
|
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
|
|
|
import { Playground } from "/src/snippets/_playground.jsx";
|
|
|
|
<Playground config={config} />
|
|
|
|
## 1. Model Introduction
|
|
|
|
{/* TODO: 1-2 paragraph intro from the HF card — what the model is, release date,
|
|
license, architecture highlights, context length. Keep it lean. */}
|
|
**__MODEL_DISPLAY__** is __ONE_LINER__.
|
|
|
|
{/* TODO: variants table (JSX, NOT a markdown pipe table). Drop the table if there's
|
|
a single variant and inline the HF link in the intro paragraph above instead. */}
|
|
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
|
<thead>
|
|
<tr style={{borderBottom: "2px solid #d55816"}}>
|
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Variant</th>
|
|
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Total params</th>
|
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Use</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/__HF_ORG__/__HF_REPO__">__MODEL_DISPLAY__</a></strong></td>
|
|
<td style={{padding: "9px 12px", textAlign: "right"}}>TODO</td>
|
|
<td style={{padding: "9px 12px"}}>TODO</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
**Recommended generation:** {/* TODO e.g. `temperature=1.0`, `top_p=1.0` (informational; do NOT hardcode in sample code) */}
|
|
|
|
**Resources:** [HuggingFace](https://huggingface.co/__HF_ORG__/__HF_REPO__).
|
|
|
|
## 2. Configuration Tips
|
|
|
|
{/* TODO: model/hardware-specific tuning notes, caveats, known issues. Delete if none. */}
|
|
|
|
## 3. Advanced Usage
|
|
|
|
{/* Keep only the subsections that apply. Commands and outputs in this section are
|
|
COLLAPSIBLE (required — match DeepSeek-V4 §3): each runnable example lives in an
|
|
<Accordion>, its REAL server output in a following <Accordion title="Example Output">. */}
|
|
|
|
### 3.1 Reasoning
|
|
|
|
Enable the `__REASONING_PARSER__` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer.
|
|
|
|
{/* This example assumes a SEPARATE-FIELD parser (thinking → `reasoning_content`,
|
|
answer → `content`). If your parser emits inline `<think>...</think>` tags inside
|
|
`content`, parse the tags from `content` instead. */}
|
|
|
|
<Accordion title="Reasoning Example (Python)">
|
|
|
|
```python Example
|
|
from openai import OpenAI
|
|
|
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
|
resp = client.chat.completions.create(
|
|
model="__HF_ORG__/__HF_REPO__",
|
|
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
|
extra_body={"chat_template_kwargs": {"thinking": True}},
|
|
)
|
|
msg = resp.choices[0].message
|
|
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
|
print("Answer:", msg.content)
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Example Output">
|
|
|
|
```text Output
|
|
TODO: paste real server output here.
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
### 3.2 Tool Calling
|
|
|
|
Enable the `__TOOLCALL_PARSER__` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
|
|
|
{/* TODO: tool-calling example in an <Accordion> + an <Accordion title="Example Output">.
|
|
On thinking-mode models the follow-up may put text in `reasoning_content`;
|
|
print both that and `content`. */}
|
|
|
|
### 3.3 HiCache (Hierarchical KV Caching)
|
|
|
|
{/* TODO: keep only if the model is large enough for hierarchical KV caching; link
|
|
the HiCache card in the Playground. Otherwise delete this subsection. */}
|