minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

docs(cookbook): add IBM Granite 4.2 cookbook (#36286)

Browse Source
This commit is contained in:
Xinyuan Tong
2026-08-25 22:54:55 +08:00
committed by GitHub
parent 46d9427b91
commit b760f7fb19
6 changed files with 545 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/IBM/Granite-4.2.mdx
+231
View File
@@ -0,0 +1,231 @@
---
title: Granite 4.2
description: "Deploy Granite 4.2 3B, 8B, and 30B dense models with SGLang on NVIDIA H200 and B200, including thinking modes and structured tool calling."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:dev
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick a Granite 4.2 checkpoint to generate the launch command. The verified matrix covers BF16 serving on one NVIDIA H200 or B200 with tensor parallelism 1.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/ibm-granite/granite-4.2.jsx";
import { benchmarks } from "/src/snippets/configs/ibm-granite/granite-4.2-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
<Note>
The H200 speed results use `lmsysorg/sglang:dev` at SGLang commit `d59c1ddf7` and the B200 results at commit `d10a656ad8`; the launch recipes were verified end to end on both GPUs against the release checkpoints. Each speed point uses 80 fixed-length random requests at 8,192 input and 1,024 output tokens, 8 warmup requests, a flushed cache, greedy sampling, and ignore-EOS.
</Note>
## Playground
The Playground layers SGLang features on top of the verified recipe. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model introduction
**Granite 4.2** is IBM's dense decoder-only language model family with 3B, 8B, and 30B checkpoints. Each checkpoint uses BF16 weights, has a configured context length of 131,072 tokens, and supports default thinking, non-thinking, low-effort thinking, and structured tool calls through its chat template. The repositories declare the Apache-2.0 license.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Variant</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Total params</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Position in family</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/ibm-granite/granite-4.2-3b">Granite 4.2 3B</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right"}}>3B</td>
<td style={{padding: "9px 12px"}}>Smallest checkpoint</td>
</tr>
<tr>
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/ibm-granite/granite-4.2-8b">Granite 4.2 8B</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right"}}>8B</td>
<td style={{padding: "9px 12px"}}>Mid-size checkpoint</td>
</tr>
<tr>
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/ibm-granite/granite-4.2-30b">Granite 4.2 30B</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right"}}>30B</td>
<td style={{padding: "9px 12px"}}>Largest checkpoint</td>
</tr>
</tbody>
</table>
**Recommended generation:** IBM recommends `temperature=1.0` and `top_p=0.95` for general chat, reasoning, and tool calling. The release checkpoints ship these values in `generation_config.json`; send them per request when you want to be explicit.
**Resources:** [Granite 4.2 3B](https://huggingface.co/ibm-granite/granite-4.2-3b) · [Granite 4.2 8B](https://huggingface.co/ibm-granite/granite-4.2-8b) · [Granite 4.2 30B](https://huggingface.co/ibm-granite/granite-4.2-30b).
## 2. Configuration tips
- **Thinking is enabled by default.** Set `chat_template_kwargs.enable_thinking` to `false` for a direct answer. Set `enable_thinking` and `low_effort` to `true` for a shorter reasoning trace.
- **Give thinking enough tokens.** At `temperature=1.0` the default thinking mode can run past 1,000 tokens on multi-step problems. Use a `max_tokens` of at least 2,048 for thinking requests so the answer is not cut off.
- **Reasoning parser.** Launch with `--reasoning-parser nemotron_3` so OpenAI-compatible responses separate the trace into `message.reasoning_content` and the answer into `message.content`. Without a parser flag the reasoning markup stays inline in `message.content`. `--reasoning-parser auto` also resolves to `nemotron_3` for these checkpoints.
- **Tool-call parser.** Launch with `--tool-call-parser qwen3_coder` so tool requests are returned through `message.tool_calls`. Without it, raw `<tool_call>` markup stays in `message.content`. `--tool-call-parser auto` also resolves to `qwen3_coder` for these checkpoints.
- **Single-GPU sizing.** All three BF16 checkpoints loaded and completed chat requests with `--tp 1 --mem-fraction-static 0.8` on one H200 and on one B200. Increase TP only after validating the new topology.
- **Image selection.** The validated path uses `lmsysorg/sglang:dev`. A stable image tested during validation had an incompatible dependency set before model loading, so use the recipe's image until a newer tagged release is confirmed.
## 3. Advanced usage
The outputs below are verbatim captures from Granite 4.2 3B on the verified server. Sampling is stochastic, so a repeated request can produce different wording.
### 3.1 Thinking modes
The `nemotron_3` reasoning parser keeps reasoning and final content in separate fields. Granite 4.2 accepts three chat-template modes: default thinking, non-thinking, and low-effort thinking.
<Accordion title="Thinking modes example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
model = "ibm-granite/granite-4.2-3b"
modes = {
"thinking": {"enable_thinking": True},
"non-thinking": {"enable_thinking": False},
"low-effort": {"enable_thinking": True, "low_effort": True},
}
for name, chat_template_kwargs in modes.items():
response = client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": "What is 17 * 23? Answer briefly."}
],
extra_body={"chat_template_kwargs": chat_template_kwargs},
)
message = response.choices[0].message
print(f"[{name}]")
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)
```
</Accordion>
<Accordion title="Example output">
```text Output
[thinking]
Reasoning: Okay, the user asked "What is 17 * 23? Answer briefly." I need to calculate 17 multiplied by 23.
Let me do the multiplication. 17 times 23.
I can break it down: 17 * 20 = 340, and 17 * 3 = 51. Then add them: 340 + 51 = 391.
Alternatively, 23 * 17: 23*10=230, 23*7=161, 230+161=391. Same result.
So the answer is 391.
The user wants a brief answer, so just state the number.
Answer:
391
[non-thinking]
Reasoning: None
Answer: 391
[low-effort]
Reasoning: Compute 17*23 = 17*20=340, plus 17*3=51 => 391.
Answer:
391
```
</Accordion>
### 3.2 Tool calling
The `qwen3_coder` parser converts the model's tool markup into OpenAI-compatible structured calls.
<Accordion title="Tool calling example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "The city name"},
},
"required": ["city"],
},
},
}]
response = client.chat.completions.create(
model="ibm-granite/granite-4.2-3b",
messages=[{"role": "user", "content": "What is the weather in Boston right now?"}],
tools=tools,
tool_choice="auto",
)
choice = response.choices[0]
message = choice.message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Content:", message.content)
for call in message.tool_calls or []:
print("Tool:", call.function.name)
print("Arguments:", call.function.arguments)
print("Finish reason:", choice.finish_reason)
```
</Accordion>
<Accordion title="Example output">
```text Output
Reasoning: Okay, the user is asking for the weather in Boston right now. I need to use the available tool called get_weather. The tool requires the city parameter. Since the user specified Boston, I'll call get_weather with city set to Boston.
Content: None
Tool: get_weather
Arguments: {"city": "Boston"}
Finish reason: tool_calls
```
</Accordion>
docs/cookbook/autoregressive/intro.mdx
+6
View File
@@ -61,6 +61,12 @@ metatags:
href="/cookbook/autoregressive/Google/Gemma4"
img="/cards/logos/google.png"
/>
<Card
title="IBM"
mode="card"
href="/cookbook/autoregressive/IBM/Granite-4.2"
img="/cards/logos/ibm.png"
/>
<Card
title="LiquidAI"
mode="card"