Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,701 @@
|
||||
---
|
||||
title: LLaDA 2.1
|
||||
metatags:
|
||||
description: "Deploy LLaDA 2.1 with SGLang - large-scale discrete diffusion language model with parallel token generation, iterative denoising, MoE architecture, and reinforcement learning for reasoning."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
import { LLaDA21Deployment } from '/src/snippets/autoregressive/llada-21-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight
|
||||
- **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores
|
||||
- **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling
|
||||
- **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following
|
||||
- **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFace</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-mini**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE (20 layers, 16 attention heads)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-flash**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:**
|
||||
|
||||
Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs.
|
||||
|
||||
<LLaDA21Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**dLLM-Specific Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`JointThreshold`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for LLaDA model loading</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.8`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-running-requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum concurrent requests</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1` (for best quality)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention computation backend</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Decoding Mode Comparison:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Threshold</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Speed</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Quality</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quality Mode (Q)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Conservative</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Moderate</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Higher benchmark scores</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Accuracy-critical tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Speed Mode (S)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Aggressive</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Very fast, relies on T2T editing</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Slightly lower</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput-critical tasks</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Hardware Requirements:**
|
||||
|
||||
- **LLaDA2.1-mini (16B)**: ~47 GB VRAM, runs on a single GPU (TP=1)
|
||||
- **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200)
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment
|
||||
|
||||
Start the server using the command generated above, for example:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-mini \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 1 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
**Simple Completion Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain what a diffusion language model is in simple terms."}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Sure! Let's break it down in simple terms.
|
||||
|
||||
A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text.
|
||||
|
||||
Here’s how it works, using a simple real-life analogy:
|
||||
|
||||
Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together.
|
||||
|
||||
Now, think of the process like this:
|
||||
|
||||
1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper).
|
||||
2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book.
|
||||
3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start.
|
||||
4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word.
|
||||
|
||||
So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story.
|
||||
|
||||
In short:
|
||||
A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned.
|
||||
```
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Streaming
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
Here are several ways to implement the Fibonacci sequence in Python:
|
||||
|
||||
## 1. Recursive Approach (Simple but Inefficient)
|
||||
|
||||
```python
|
||||
def fibonacci_recursive(n):
|
||||
"""
|
||||
Compute the nth Fibonacci number using recursion.
|
||||
|
||||
Args:
|
||||
n (int): The position in the Fibonacci sequence (0-indexed)
|
||||
|
||||
Returns:
|
||||
int: The nth Fibonacci number
|
||||
|
||||
Raises:
|
||||
ValueError: If n is negative
|
||||
"""
|
||||
if n < 0:
|
||||
raise ValueError("n must be non-negative")
|
||||
|
||||
if n <= 1:
|
||||
return n
|
||||
|
||||
return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
|
||||
|
||||
# Example usage
|
||||
print(fibonacci_recursive(10)) # Output: 55
|
||||
```
|
||||
|
||||
## 2. Iterative Approach (Efficient)
|
||||
...
|
||||
````
|
||||
|
||||
#### 4.3.2 Code Generation
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."}
|
||||
],
|
||||
max_tokens=2048
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
```python
|
||||
def is_palindrome(s):
|
||||
"""
|
||||
Check if a string is a palindrome.
|
||||
|
||||
A palindrome is a word, phrase, or sequence that reads the same backward as forward.
|
||||
This function ignores case, spaces, punctuation, and non characters characters.
|
||||
|
||||
Args:
|
||||
s (str): The string to check
|
||||
|
||||
Returns:
|
||||
bool: True if the string is a palindrome, False otherwise
|
||||
|
||||
Examples:
|
||||
>>> is_palindrome("racecar")
|
||||
True
|
||||
>>> is_palindrome("A man a plan a canal Panama")
|
||||
True
|
||||
>>> is_palindrome("race a car")
|
||||
False
|
||||
>>> is_palindrome("")
|
||||
True
|
||||
>>> is_palindrome("a")
|
||||
True
|
||||
"""
|
||||
# Remove non-alphanumeric characters and convert to lowercase
|
||||
cleaned = ''.join(char.lower() for char in s if char.isalnum())
|
||||
|
||||
# Check if the cleaned string reads the same forwards and backwards
|
||||
return cleaned == cleaned[::-1]
|
||||
|
||||
# Test cases
|
||||
def test_is_palindrome():
|
||||
"""Test the is_palindrome function with various inputs."""
|
||||
|
||||
# Test basic palindromes
|
||||
assert is_palindrome("racecar") == True
|
||||
assert is_palindrome("level") == True
|
||||
assert is_palindrome("madam") == True
|
||||
assert is_palindrome("radar") == True
|
||||
|
||||
# Test palindromes with spaces and punctuation
|
||||
assert is_palindrome("A man a plan a canal Panama") == True
|
||||
assert is_palindrome("race a car") == False
|
||||
assert is_palindrome("Was it a car or a cat I saw?") == True
|
||||
assert is_palindrome("Madam, I'm Adam") == True
|
||||
|
||||
# Test edge cases
|
||||
assert is_palindrome("") == True
|
||||
assert is_palindrome("a") == True
|
||||
assert is_palindrome("A") == True
|
||||
assert is_palindrome("Aa") == True
|
||||
|
||||
# Test non-palindromes
|
||||
assert is_palindrome("hello") == False
|
||||
assert is_palindrome("world") == False
|
||||
assert is_palindrome("python") == False
|
||||
|
||||
# Test single characters
|
||||
assert is_palindrome("1") == True
|
||||
assert is_palindrome("1") == True
|
||||
|
||||
print("All tests passed!")
|
||||
|
||||
# Run the tests
|
||||
if __name__ == "__main__":
|
||||
# Example usage
|
||||
print("Testing isalindrome function:")
|
||||
print(f"'racecar' {is_palindrome('racecar')}")
|
||||
print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}")
|
||||
print(f"'race a car': {is_palindrome('race a car')}")
|
||||
print(f"'hello': {is_palindrome('hello')}")
|
||||
|
||||
# Run tests
|
||||
test_is_palindrome()
|
||||
```
|
||||
|
||||
This implementation includes:
|
||||
|
||||
1. **Comprehensive function** `is_palindrome()` that:
|
||||
- Ignores case by converting to lowercase
|
||||
- Removes all non-alphanumeric characters (spaces, punctuation, etc.)
|
||||
- Uses string slicing (`[::-1]`) to reverse the string
|
||||
|
||||
2. **Detailed docstring** explaining:
|
||||
- What the function does
|
||||
- How it works
|
||||
- Return value
|
||||
- Examples of usage
|
||||
|
||||
3. **Extensive test cases** covering:
|
||||
- Basic palindromes
|
||||
- Palindromes with spaces and punctuation
|
||||
- Edge cases (empty string, single character)
|
||||
- Non-palindromes
|
||||
- Mixed case scenarios
|
||||
|
||||
4. **Test function** that uses assertions to verify the function works correctly
|
||||
|
||||
The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama".
|
||||
````
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 (4x)
|
||||
- SGLang Version: 0.5.8+
|
||||
|
||||
#### 5.1.1 LLaDA2.1-mini
|
||||
|
||||
**Model Deployment:**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-mini \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 1 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer
|
||||
```
|
||||
|
||||
- Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-mini \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Latency Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 9.90
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3433
|
||||
Request throughput (req/s): 1.01
|
||||
Input token throughput (tok/s): 616.26
|
||||
Output token throughput (tok/s): 426.26
|
||||
Peak output token throughput (tok/s): 1010.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 1042.53
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 988.87
|
||||
Median E2E Latency (ms): 655.27
|
||||
P90 E2E Latency (ms): 1952.50
|
||||
P99 E2E Latency (ms): 2932.19
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 152.74
|
||||
Median TTFT (ms): 150.37
|
||||
P99 TTFT (ms): 229.78
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.16
|
||||
Median TPOT (ms): 2.08
|
||||
P99 TPOT (ms): 3.72
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.10
|
||||
Median ITL (ms): 1.99
|
||||
P95 ITL (ms): 4.03
|
||||
P99 ITL (ms): 6.34
|
||||
Max ITL (ms): 26.59
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-mini \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Throughput Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 467.74
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 189717
|
||||
Request throughput (req/s): 1.07
|
||||
Input token throughput (tok/s): 534.12
|
||||
Output token throughput (tok/s): 540.17
|
||||
Peak output token throughput (tok/s): 1753.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 1074.30
|
||||
Concurrency: 90.77
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 84912.27
|
||||
Median E2E Latency (ms): 86564.26
|
||||
P90 E2E Latency (ms): 110567.26
|
||||
P99 E2E Latency (ms): 114303.38
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 83920.39
|
||||
Median TTFT (ms): 85669.54
|
||||
P99 TTFT (ms): 112969.91
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.67
|
||||
Median TPOT (ms): 1.65
|
||||
P99 TPOT (ms): 4.43
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.69
|
||||
Median ITL (ms): 1.46
|
||||
P95 ITL (ms): 3.96
|
||||
P99 ITL (ms): 4.84
|
||||
Max ITL (ms): 92.08
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 LLaDA2.1-flash
|
||||
|
||||
**Model Deployment:**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-flash \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer
|
||||
```
|
||||
|
||||
- Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Latency Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 14.46
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3276
|
||||
Request throughput (req/s): 0.69
|
||||
Input token throughput (tok/s): 421.79
|
||||
Output token throughput (tok/s): 291.75
|
||||
Peak output token throughput (tok/s): 676.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 713.53
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1445.16
|
||||
Median E2E Latency (ms): 968.06
|
||||
P90 E2E Latency (ms): 3101.86
|
||||
P99 E2E Latency (ms): 4208.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 231.63
|
||||
Median TTFT (ms): 242.67
|
||||
P99 TTFT (ms): 341.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.04
|
||||
Median TPOT (ms): 2.79
|
||||
P99 TPOT (ms): 5.33
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.05
|
||||
Median ITL (ms): 2.41
|
||||
P95 ITL (ms): 7.25
|
||||
P99 ITL (ms): 8.27
|
||||
Max ITL (ms): 29.27
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Throughput Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 671.85
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 177961
|
||||
Request throughput (req/s): 0.74
|
||||
Input token throughput (tok/s): 371.85
|
||||
Output token throughput (tok/s): 376.07
|
||||
Peak output token throughput (tok/s): 1521.00
|
||||
Peak concurrent requests: 103
|
||||
Total token throughput (tok/s): 747.92
|
||||
Concurrency: 91.28
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 122658.36
|
||||
Median E2E Latency (ms): 125265.55
|
||||
P90 E2E Latency (ms): 159554.07
|
||||
P99 E2E Latency (ms): 165174.88
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 121009.17
|
||||
Median TTFT (ms): 124437.80
|
||||
P99 TTFT (ms): 163579.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.73
|
||||
Median TPOT (ms): 2.16
|
||||
P99 TPOT (ms): 7.13
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.38
|
||||
Median ITL (ms): 1.40
|
||||
P95 ITL (ms): 6.89
|
||||
P99 ITL (ms): 8.60
|
||||
Max ITL (ms): 176.78
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.895
|
||||
Invalid: 0.000
|
||||
Latency: 100.552 s
|
||||
Output throughput: 262.094 token/s
|
||||
```
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
title: Ling-2.5-1T
|
||||
metatags:
|
||||
description: "Deploy Ling-2.5-1T with SGLang - 1T parameter MoE model with 63B active parameters, trillion-scale context length up to 1M tokens, and agentic tool calling capabilities."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens.
|
||||
- **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume ~4x the output tokens.
|
||||
- **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following.
|
||||
- **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4.
|
||||
- **Context Length**: 256K -> 1M (YaRN)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ling-2.5-1T requires a specific SGLang Docker image:
|
||||
|
||||
```bash Command
|
||||
# For H200/B200
|
||||
docker pull lmsysorg/sglang:nightly-dev-20260213-a0ebaa64
|
||||
|
||||
# For GB200/GB300
|
||||
docker pull lmsysorg/sglang:nightly-dev-cu13-20260213-a0ebaa64
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
|
||||
|
||||
Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/installation) guide for setup instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform.
|
||||
|
||||
import { Ling251TDeployment } from '/src/snippets/autoregressive/ling-25-1t-deployment.jsx'
|
||||
|
||||
<Ling251TDeployment />
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
|
||||
- `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes.
|
||||
- Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading.
|
||||
- On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For example, launch the server on 2 H200 nodes:
|
||||
|
||||
```bash Command
|
||||
export MASTER_IP=10.10.0.1 # The IP of Node 0
|
||||
export PORT=30000
|
||||
export DIST_PORT=50000
|
||||
|
||||
# Node 0:
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path inclusionAI/Ling-2.5-1T \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--pp-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
|
||||
--mem-frac 0.95
|
||||
|
||||
|
||||
# Node 1:
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path inclusionAI/Ling-2.5-1T \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--pp-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
|
||||
--mem-frac 0.95
|
||||
```
|
||||
|
||||
Once the server is running, send requests to the master node:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
|
||||
```
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "e82af153da844ee6aed7a27a3187f2f4",
|
||||
"object": "chat.completion",
|
||||
"created": 1771216764,
|
||||
"model": "auto",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "The capital of France is **Paris**.\n\n**Additional details:**\n* It is the largest city in France.\n* It is located in the north-central part of the country along the Seine River.\n* Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).",
|
||||
"reasoning_content": null,
|
||||
"tool_calls": null
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "stop",
|
||||
"matched_stop": 156895
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 25,
|
||||
"total_tokens": 93,
|
||||
"completion_tokens": 68,
|
||||
"prompt_tokens_details": null,
|
||||
"reasoning_tokens": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For more API usage examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "inclusionAI/Ling-2.5-1T",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"description": "Search for information on the internet",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "The search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto"
|
||||
}'
|
||||
```
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "b968e45c7d414f7482c8ffc0f9c6b688",
|
||||
"object": "chat.completion",
|
||||
"created": 1771216520,
|
||||
"model": "inclusionAI/Ling-2.5-1T",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"reasoning_content": null,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_e75f711d8ad840ed9d382c9e",
|
||||
"index": 0,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"arguments": "{\"query\": \"latest news about AI\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "tool_calls",
|
||||
"matched_stop": null
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 173,
|
||||
"total_tokens": 196,
|
||||
"completion_tokens": 23,
|
||||
"prompt_tokens_details": null,
|
||||
"reasoning_tokens": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 45.410 s
|
||||
Output throughput: 560.642 token/s
|
||||
```
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
title: Ring-2.5-1T
|
||||
metatags:
|
||||
description: "Deploy Ring-2.5-1T with SGLang - world's first open-source 1T parameter reasoning model with hybrid linear attention, deep reasoning, and agentic tool calling capabilities."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Model**: ~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention)
|
||||
- **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens
|
||||
- **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback
|
||||
- **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training
|
||||
- **Tool Calling**: Supports function calling with XML-style tool call format
|
||||
- **Context Length**: 128K -> 256K (YaRN)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ring-2.5-1T requires a specific SGLang Docker image:
|
||||
|
||||
```bash Command
|
||||
# For H200/B200
|
||||
docker pull lmsysorg/sglang:nightly-dev-20260213-a0ebaa64
|
||||
|
||||
# For GB200/GB300
|
||||
docker pull lmsysorg/sglang:nightly-dev-cu13-20260213-a0ebaa64
|
||||
|
||||
# For MI300X/325X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
|
||||
|
||||
# For MI355X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
import { Ring251TDeployment } from '/src/snippets/autoregressive/ring-25-1t-deployment.jsx'
|
||||
|
||||
<Ring251TDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
|
||||
- The model uses FP8 quantization (compressed-tensors format).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy Ring-2.5-1T with the following command (on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.5-1T",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Output Example</summary>
|
||||
|
||||
````text Output
|
||||
We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution.
|
||||
|
||||
We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36.
|
||||
|
||||
But we need to present step by step. Also ensure it's clear.
|
||||
|
||||
We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36.
|
||||
|
||||
But any method is fine.
|
||||
|
||||
We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)."
|
||||
|
||||
We'll show:
|
||||
|
||||
15% = 15/100 = 0.15
|
||||
|
||||
Then 0.15 × 240 = 36.
|
||||
|
||||
Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36.
|
||||
|
||||
Finally, answer: 36.
|
||||
|
||||
We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36."
|
||||
|
||||
We'll present as a final answer: \boxed{36}.
|
||||
|
||||
However, we need to provide step-by-step solution as per instructions. We'll write a full explanation.
|
||||
|
||||
We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36.
|
||||
|
||||
Alr.
|
||||
|
||||
I think that's it.
|
||||
|
||||
|
||||
**Step 1:** Write 15% as a fraction or decimal.
|
||||
\[ 15\% = \frac{15}{100} = 0.15\]
|
||||
|
||||
**Step 2:** Multiply the number (240) by this fraction/decimal.
|
||||
\[ 240 \times 0.15 = 36\]
|
||||
|
||||
Alternatively, using the fraction:
|
||||
\[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\]
|
||||
|
||||
**Conclusion:** 15% of 240 is 36.
|
||||
|
||||
\[ \boxed{36} \]
|
||||
````
|
||||
|
||||
</details>
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
To enable tool calling, add `--tool-call-parser qwen` when launching the server.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser qwen \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.5-1T",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
[ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K
|
||||
|
||||
- Deployment Command
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp-size 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.010
|
||||
Latency: 615.833 s
|
||||
Output throughput: 412.360 token/s
|
||||
```
|
||||
Reference in New Issue
Block a user