Muse Glimmer Cookbook (#34271)
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai> Co-authored-by: Jimmy Shong <jimmysh341@gmail.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
Brayden Zhong
co-authored by
Brayden Zhong
Jimmy Shong
Zijie Xia
Brayden Zhong
Jimmy Shong
Zijie Xia
parent
955569a2dc
commit
a6c34df044
@@ -0,0 +1,650 @@
|
||||
---
|
||||
title: Llama-3.1
|
||||
metatags:
|
||||
description: "Deploy Llama 3.1 (8B/70B/405B) with SGLang - 128K context, tool use, multilingual support, and speculative decoding optimization."
|
||||
---
|
||||
## 1. Model Introduction
|
||||
|
||||
Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time.
|
||||
|
||||
These models bring open intelligence to all, with several new features and improvements:
|
||||
|
||||
- **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities.
|
||||
- **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning.
|
||||
- **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools.
|
||||
- **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai.
|
||||
|
||||
For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models.
|
||||
|
||||
import { Llama31Deployment } from "/src/snippets/autoregressive/llama31-deployment.jsx";
|
||||
|
||||
<Llama31Deployment />
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Speculative Decoding (NVIDIA GPUs):**
|
||||
|
||||
- Using Speculative Decoding for latency-sensitive scenarios:
|
||||
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
|
||||
- `--speculative-num-steps 3`: Number of speculative verification rounds
|
||||
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
|
||||
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
|
||||
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B).
|
||||
|
||||
**AMD GPU Deployment:**
|
||||
|
||||
- **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB)
|
||||
- **Verified TP Configurations**:
|
||||
- MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
|
||||
- MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
|
||||
- **FP8 Model Variants**:
|
||||
- 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8`
|
||||
- 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV`
|
||||
- **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models
|
||||
|
||||
**Xeon CPU Deployment:**
|
||||
|
||||
- Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
SGLang exposes an OpenAI-compatible endpoint. First, start the server
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="Meta-Llama/Llama-3.1-405B-Instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful coding assistant."},
|
||||
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
|
||||
],
|
||||
temperature=0.2,
|
||||
max_tokens=512,
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
**Exponential Backoff Retry Function in Python**
|
||||
=====================================================
|
||||
|
||||
Below is a Python function that uses the `requests` library to retry a request with exponential backoff.
|
||||
|
||||
```python
|
||||
import requests
|
||||
import time
|
||||
import random
|
||||
|
||||
def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
|
||||
"""
|
||||
Retry a request with exponential backoff.
|
||||
|
||||
Args:
|
||||
url (str): The URL to make the request to.
|
||||
method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
|
||||
retries (int): The number of retries to attempt. Defaults to 3.
|
||||
backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
|
||||
max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.
|
||||
|
||||
Returns:
|
||||
The response object from the successful request.
|
||||
"""
|
||||
|
||||
delay = 1
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
response = requests.request(method, url)
|
||||
response.raise_for_status() # Raise an exception for HTTP errors
|
||||
return response
|
||||
except requests.RequestException as e:
|
||||
if attempt < retries:
|
||||
# Calculate the delay for this retry
|
||||
delay = min(delay * backoff_factor, max_delay)
|
||||
# Add a random jitter to the delay to prevent thundering herd problem
|
||||
delay += random.uniform(0, delay * 0.1)
|
||||
# Wait for the calculated delay before retrying
|
||||
time.sleep(delay)
|
||||
else:
|
||||
# If all retries have failed, raise the exception
|
||||
raise e
|
||||
...
|
||||
````
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Tool Calling
|
||||
|
||||
Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
|
||||
--tool-call-parser llama3 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Python Example**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.1-405B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What's the weather like in Boston today?",
|
||||
}
|
||||
],
|
||||
temperature=0.7,
|
||||
stream=True,
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
|
||||
arguments = []
|
||||
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#openai-compatible-api)
|
||||
|
||||
**Output Example**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"city": "Boston", "unit": "fahrenheit"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results**
|
||||
After getting the tool call, you can execute the function:
|
||||
|
||||
```python Example
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather like in Boston today?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Boston", "unit": "fahrenheit"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Boston", "fahrenheit")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="Meta-Llama/Llama-3.1-405B-Instruct",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA A100 GPU (8x)
|
||||
- Model: Meta-Llama/Llama-3.1-70B
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 79.81
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4208
|
||||
Request throughput (req/s): 0.13
|
||||
Input token throughput (tok/s): 76.44
|
||||
Output token throughput (tok/s): 52.88
|
||||
Peak output token throughput (tok/s): 54.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 129.32
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7977.81
|
||||
Median E2E Latency (ms): 6373.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 131.61
|
||||
Median TTFT (ms): 131.77
|
||||
P99 TTFT (ms): 163.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.63
|
||||
Median TPOT (ms): 18.63
|
||||
P99 TPOT (ms): 18.65
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.64
|
||||
Median ITL (ms): 18.64
|
||||
P95 ITL (ms): 18.69
|
||||
P99 ITL (ms): 18.74
|
||||
Max ITL (ms): 21.95
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 79.47
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 38450
|
||||
Request throughput (req/s): 1.01
|
||||
Input token throughput (tok/s): 499.17
|
||||
Output token throughput (tok/s): 513.48
|
||||
Peak output token throughput (tok/s): 674.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1012.65
|
||||
Concurrency: 13.47
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13376.67
|
||||
Median E2E Latency (ms): 14130.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 264.84
|
||||
Median TTFT (ms): 147.02
|
||||
P99 TTFT (ms): 791.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 26.09
|
||||
Median TPOT (ms): 26.08
|
||||
P99 TPOT (ms): 34.65
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.76
|
||||
Median ITL (ms): 23.95
|
||||
P95 ITL (ms): 24.72
|
||||
P99 ITL (ms): 98.32
|
||||
Max ITL (ms): 478.92
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 131.64
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 243641
|
||||
Request throughput (req/s): 3.80
|
||||
Input token throughput (tok/s): 1897.87
|
||||
Output token throughput (tok/s): 1919.38
|
||||
Peak output token throughput (tok/s): 3100.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3817.25
|
||||
Concurrency: 89.70
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 23616.71
|
||||
Median E2E Latency (ms): 22770.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 245.98
|
||||
Median TTFT (ms): 184.22
|
||||
P99 TTFT (ms): 1251.67
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 47.19
|
||||
Median TPOT (ms): 48.67
|
||||
P99 TPOT (ms): 56.37
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 46.34
|
||||
Median ITL (ms): 33.46
|
||||
P95 ITL (ms): 108.61
|
||||
P99 ITL (ms): 166.11
|
||||
Max ITL (ms): 1107.09
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B\
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 83.25
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.12
|
||||
Input token throughput (tok/s): 503.77
|
||||
Output token throughput (tok/s): 50.69
|
||||
Peak output token throughput (tok/s): 54.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 554.46
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8322.45
|
||||
Median E2E Latency (ms): 6873.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 395.25
|
||||
Median TTFT (ms): 318.02
|
||||
P99 TTFT (ms): 850.80
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.80
|
||||
Median TPOT (ms): 18.81
|
||||
P99 TPOT (ms): 19.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.83
|
||||
Median ITL (ms): 18.81
|
||||
P95 ITL (ms): 19.06
|
||||
P99 ITL (ms): 19.08
|
||||
Max ITL (ms): 23.08
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 107.12
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41603
|
||||
Request throughput (req/s): 0.75
|
||||
Input token throughput (tok/s): 2800.81
|
||||
Output token throughput (tok/s): 389.00
|
||||
Peak output token throughput (tok/s): 624.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3189.81
|
||||
Concurrency: 14.18
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18988.30
|
||||
Median E2E Latency (ms): 20290.66
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 603.42
|
||||
Median TTFT (ms): 531.82
|
||||
P99 TTFT (ms): 2607.95
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 36.94
|
||||
Median TPOT (ms): 36.73
|
||||
P99 TPOT (ms): 79.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.36
|
||||
Median ITL (ms): 25.72
|
||||
P95 ITL (ms): 27.07
|
||||
P99 ITL (ms): 439.74
|
||||
Max ITL (ms): 2529.51
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 215.66
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169035
|
||||
Request throughput (req/s): 1.48
|
||||
Input token throughput (tok/s): 5906.92
|
||||
Output token throughput (tok/s): 788.27
|
||||
Peak output token throughput (tok/s): 1920.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 6695.19
|
||||
Concurrency: 60.01
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 40443.85
|
||||
Median E2E Latency (ms): 39813.12
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 633.32
|
||||
Median TTFT (ms): 616.38
|
||||
P99 TTFT (ms): 1912.97
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 74.95
|
||||
Median TPOT (ms): 82.85
|
||||
P99 TPOT (ms): 118.46
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 75.08
|
||||
Median ITL (ms): 34.12
|
||||
P95 ITL (ms): 261.18
|
||||
P99 ITL (ms): 828.12
|
||||
Max ITL (ms): 1970.03
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.830
|
||||
Invalid: 0.000
|
||||
Latency: 11.794 s
|
||||
Output throughput: 1406.961 token/s
|
||||
```
|
||||
@@ -0,0 +1,235 @@
|
||||
---
|
||||
title: Llama-3.3-70B
|
||||
metatags:
|
||||
description: "Deploy Llama-3.3-70B-Instruct with SGLang on AMD GPUs - 128K context, enhanced reasoning, tool calling, and multilingual support."
|
||||
---
|
||||
## 1. Model Introduction
|
||||
|
||||
[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
|
||||
- **Tool Calling**: Native support for function calling and tool use scenarios
|
||||
- **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
|
||||
- **Extended Context**: 128K token context window for processing long documents and complex tasks
|
||||
- **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X
|
||||
|
||||
**License:**
|
||||
Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs.
|
||||
|
||||
### 3.1 Interactive Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
|
||||
|
||||
import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
|
||||
|
||||
<Llama33Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**AMD GPU Deployment:**
|
||||
|
||||
- All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
|
||||
- **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
|
||||
- **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
|
||||
- **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput
|
||||
|
||||
**Xeon CPU Deployment:**
|
||||
|
||||
Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Tool Calling
|
||||
|
||||
Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.3-70B-Instruct \
|
||||
--tool-call-parser llama3 \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Tokyo?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
# Check for tool calls
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
tool_call = message.tool_calls[0]
|
||||
print(f"Function: {tool_call.function.name}")
|
||||
print(f"Arguments: {tool_call.function.arguments}")
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After executing the function, send the result back
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Build conversation with tool result
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Tokyo?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Tokyo", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Tokyo", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
|
||||
```
|
||||
|
||||
#### 4.2.2 Long Context Processing
|
||||
|
||||
Leverage the 128K context window for processing long documents:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Example with long document
|
||||
long_document = "..." * 10000 # Your long document here
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=1000
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## 5. Benchmarking
|
||||
|
||||
Use the SGLang benchmarking suite to test model performance with different workload patterns:
|
||||
|
||||
### 5.1 Basic Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 1000 \
|
||||
--random-input 1024 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
### 5.2 Adjusting Benchmark Parameters
|
||||
|
||||
**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:
|
||||
|
||||
- Short conversations: `--random-input 1024 --random-output 1024`
|
||||
- Long outputs: `--random-input 1024 --random-output 8192`
|
||||
- Long inputs: `--random-input 8192 --random-output 1024`
|
||||
|
||||
**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:
|
||||
|
||||
- Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
|
||||
- Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
|
||||
- High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
|
||||
- [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
|
||||
- [SGLang Documentation](/)
|
||||
- [AMD ROCm Documentation](https://rocm.docs.amd.com/)
|
||||
@@ -0,0 +1,572 @@
|
||||
---
|
||||
title: Llama 4
|
||||
metatags:
|
||||
description: "Deploy Llama 4 Scout and Maverick with SGLang - Meta's latest generation open-source LLMs with industry-leading performance."
|
||||
---
|
||||
|
||||
import { Llama4ScoutDeployment } from '/src/snippets/autoregressive/llama4-scout-deployment.jsx';
|
||||
import { Llama4MaverickDeployment } from '/src/snippets/autoregressive/llama4-maverick-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance.
|
||||
|
||||
SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
|
||||
|
||||
Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
The highly capable Llama 4 Maverick with 17B active parameters out of ~400B total, with 128 experts.
|
||||
The efficient Llama 4 Scout also has 17B active parameters out of ~109B total, using just 16 experts.
|
||||
Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi).
|
||||
|
||||
For more details, please refer to the official llama4 Repository:https://www.llama.com/models/llama-4/
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Llama4ScoutDeployment />
|
||||
|
||||
<Llama4MaverickDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **OOM Mitigation:** Reduce `--context-length` to avoid GPU out-of-memory. Recommended: Scout up to 1M on 8×H100, up to 2.5M on 8×H200; Maverick doesn't need context-length set on 8×H200. With hybrid KV cache enabled, Scout can reach 5M on 8×H100 and 10M on 8×H200.
|
||||
- **Attention Backend Auto-Selection:** SGLang automatically picks the optimal backend. Manual override with `--attention-backend`:
|
||||
- Blackwell (B200/GB200): `trtllm_mha`
|
||||
- Hopper (H100/H200): `fa3`
|
||||
- AMD GPUs: `aiter`
|
||||
- Intel XPU: `intel_xpu`
|
||||
- Other: `triton`
|
||||
- **Chat Template:** Add `--chat-template llama-4` for chat completion tasks.
|
||||
- **Multi-Modal:** Add `--enable-multimodal` to enable image input support.
|
||||
- **Hybrid KV Cache:** Set `--swa-full-tokens-ratio` to control the ratio of SWA (local attention) KV tokens to full-attention KV tokens (default: 0.8, range: 0–1).
|
||||
- **EAGLE Speculative Decoding:** Supported for Llama 4 Scout and Maverick via EAGLE3. Enable with the interactive command generator above.
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Llama4 \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 Launch the server
|
||||
|
||||
### Llama-4-Scout
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### Llama-4-Maverick
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--tp 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
#### 4.2.3 EAGLE Speculative Decoding
|
||||
|
||||
SGLang supports Llama 4 Maverick (400B) with [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding). Enable with the EAGLE3 algorithm and the SGLang EAGLE3 draft model:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
### 5.1 Speed Benchmark (Scout)
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300x GPU
|
||||
|
||||
Model: Llama-4-Scout
|
||||
|
||||
Tensor Parallelism: 8
|
||||
|
||||
sglang version: 0.5.9
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 74.62
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4211
|
||||
Request throughput (req/s): 0.14
|
||||
Input token throughput (tok/s): 82.88
|
||||
Output token throughput (tok/s): 57.42
|
||||
Peak output token throughput (tok/s): 146.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 140.20
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7459.48
|
||||
Median E2E Latency (ms): 4489.77
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4246.98
|
||||
Median TTFT (ms): 68.57
|
||||
P99 TTFT (ms): 48091.05
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.49
|
||||
Median TPOT (ms): 7.40
|
||||
P99 TPOT (ms): 7.40
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.49
|
||||
Median ITL (ms): 7.49
|
||||
P95 ITL (ms): 7.47
|
||||
P99 ITL (ms): 7.52
|
||||
Max ITL (ms): 10.44
|
||||
==================================================
|
||||
```
|
||||
### 5.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 45.41
|
||||
Total input tokens: 49668
|
||||
Total input text tokens: 49668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40516
|
||||
Request throughput (req/s): 2.26
|
||||
Input token throughput (tok/s): 1120.46
|
||||
Output token throughput (tok/s): 1152.47
|
||||
Peak output token throughput (tok/s): 1520.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2272.84
|
||||
Concurrency: 14.76
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6089.22
|
||||
Median E2E Latency (ms): 6568.80
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 124.44
|
||||
Median TTFT (ms): 87.42
|
||||
P99 TTFT (ms): 268.72
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.88
|
||||
Median TPOT (ms): 12.00
|
||||
P99 TPOT (ms): 15.49
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.72
|
||||
Median ITL (ms): 10.54
|
||||
P95 ITL (ms): 11.22
|
||||
P99 ITL (ms): 67.88
|
||||
Max ITL (ms): 74.05
|
||||
==================================================
|
||||
```
|
||||
### 5.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 85.84
|
||||
Total input tokens: 249841
|
||||
Total input text tokens: 249841
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250498
|
||||
Request throughput (req/s): 5.84
|
||||
Input token throughput (tok/s): 2910.84
|
||||
Output token throughput (tok/s): 2944.82
|
||||
Peak output token throughput (tok/s): 4100.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5854.65
|
||||
Concurrency: 92.24
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 15844.00
|
||||
Median E2E Latency (ms): 15262.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 204.46
|
||||
Median TTFT (ms): 129.96
|
||||
P99 TTFT (ms): 528.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 41.56
|
||||
Median TPOT (ms): 42.90
|
||||
P99 TPOT (ms): 47.48
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 40.99
|
||||
Median ITL (ms): 24.46
|
||||
P95 ITL (ms): 84.46
|
||||
P99 ITL (ms): 87.64
|
||||
Max ITL (ms): 226.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark (Maverick)
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300x GPU
|
||||
|
||||
Model: Llama-4-Maverick
|
||||
|
||||
Tensor Parallelism: 8
|
||||
|
||||
sglang version: 0.5.9
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.2.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 68.08
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4202
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 89.62
|
||||
Output token throughput (tok/s): 61.99
|
||||
Peak output token throughput (tok/s): 168.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 151.61
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6805.62
|
||||
Median E2E Latency (ms): 2733.91
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4296.56
|
||||
Median TTFT (ms): 57.45
|
||||
P99 TTFT (ms): 38633.95
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.95
|
||||
Median TPOT (ms): 5.96
|
||||
P99 TPOT (ms): 5.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.96
|
||||
Median ITL (ms): 5.96
|
||||
P95 ITL (ms): 6.02
|
||||
P99 ITL (ms): 6.08
|
||||
Max ITL (ms): 7.02
|
||||
==================================================
|
||||
```
|
||||
### 5.2.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 30.72
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40923
|
||||
Request throughput (req/s): 2.60
|
||||
Input token throughput (tok/s): 1291.39
|
||||
Output token throughput (tok/s): 1328.41
|
||||
Peak output token throughput (tok/s): 1760.00
|
||||
Peak concurrent requests: 22
|
||||
Total token throughput (tok/s): 2619.80
|
||||
Concurrency: 13.92
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5345.15
|
||||
Median E2E Latency (ms): 5679.73
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 259.30
|
||||
Median TTFT (ms): 72.60
|
||||
P99 TTFT (ms): 1063.45
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.53
|
||||
Median TPOT (ms): 10.22
|
||||
P99 TPOT (ms): 20.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.99
|
||||
Median ITL (ms): 9.10
|
||||
P95 ITL (ms): 9.87
|
||||
P99 ITL (ms): 55.62
|
||||
Max ITL (ms): 868.54
|
||||
==================================================
|
||||
```
|
||||
### 5.2.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 90.95
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 251625
|
||||
Request throughput (req/s): 5.50
|
||||
Input token throughput (tok/s): 2746.77
|
||||
Output token throughput (tok/s): 2777.90
|
||||
Peak output token throughput (tok/s): 3700.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 5524.67
|
||||
Concurrency: 93.04
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16924.17
|
||||
Median E2E Latency (ms): 16294.85
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 188.19
|
||||
Median TTFT (ms): 128.96
|
||||
P99 TTFT (ms): 534.81
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.63
|
||||
Median TPOT (ms): 35.37
|
||||
P99 TPOT (ms): 38.26
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.19
|
||||
Median ITL (ms): 27.66
|
||||
P95 ITL (ms): 76.91
|
||||
P99 ITL (ms): 78.82
|
||||
Max ITL (ms): 268.17
|
||||
==================================================
|
||||
```
|
||||
### 5.3 Accuracy Benchmark
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
- Llama-4-Scout-17B-16E-Instruct
|
||||
```text Output
|
||||
Accuracy: 0.945
|
||||
Invalid: 0.000
|
||||
Latency: 12.731 s
|
||||
Output throughput: 1595.418 token/s
|
||||
```
|
||||
- Llama-4-Maverick-17B-128E-Instruct
|
||||
```text Output
|
||||
Accuracy: 0.895
|
||||
Invalid: 0.000
|
||||
Latency: 9.739 s
|
||||
Output throughput: 2405.505 token/s
|
||||
```
|
||||
|
||||
#### 5.3.2 MMLU Pro with lm-eval
|
||||
|
||||
Accuracy on MMLU Pro matches [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) on 8×H100 (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)):
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Official</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Scout-17B-16E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>74.3</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>75.2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Maverick-17B-128E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>80.5</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>80.7</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Scout:**
|
||||
|
||||
```bash Command
|
||||
# Start the server
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
|
||||
# Run lm_eval
|
||||
lm_eval --model local-chat-completions \
|
||||
--model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
|
||||
--tasks mmlu_pro \
|
||||
--batch_size 128 \
|
||||
--apply_chat_template \
|
||||
--num_fewshot 0
|
||||
```
|
||||
|
||||
**Maverick:**
|
||||
|
||||
```bash Command
|
||||
# Start the server
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
|
||||
# Run lm_eval
|
||||
lm_eval --model local-chat-completions \
|
||||
--model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
|
||||
--tasks mmlu_pro \
|
||||
--batch_size 128 \
|
||||
--apply_chat_template \
|
||||
--num_fewshot 0
|
||||
```
|
||||
@@ -0,0 +1,205 @@
|
||||
---
|
||||
title: Muse Glimmer
|
||||
description: "A multimodal reasoning model served from a BF16, NVFP4 + MXFP8, vendor GGUF, or MLX checkpoint."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
See the [official SGLang installation guide](../../../docs/get-started/install) for all installation methods and hardware platforms. The steps below match the **Python** and **Docker** options in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-muse-glimmer
|
||||
```
|
||||
|
||||
See [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker) to start the image. Replace the inner `sglang serve ...` command with the command from the panel below.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Select a checkpoint format. Select whether to use speculative decoding:
|
||||
|
||||
- **Standard**: Use normal autoregressive decoding.
|
||||
- **DFlash**: Use speculative decoding with the DFlash draft model. The draft serves as published, with no conversion step. See [§2](#2-configuration-tips).
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/meta-models/muse-glimmer.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/meta-models/muse-glimmer-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
Use the Playground to test SGLang features that are not in the verified matrix. The Deploy panel above shows only combinations that the SGLang team has verified. The Playground lets you add more options to the command from the Deploy panel.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
Muse Glimmer is a multimodal reasoning model. You can serve Muse Glimmer in four formats:
|
||||
|
||||
- A BF16 checkpoint (`MuseGlimmerForConditionalGeneration`).
|
||||
- A set of vendor GGUF files.
|
||||
- A ready-to-serve NVFP4 + MXFP8 checkpoint.
|
||||
- Three MLX repacks for Apple Silicon.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Form</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Source</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong>BF16</strong></td>
|
||||
<td style={{padding: "9px 12px"}}><code>meta-models/Muse-Glimmer-30B</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Supports image input.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong>GGUF Q4_K_M</strong></td>
|
||||
<td style={{padding: "9px 12px"}}><code>meta-models/Muse-Glimmer-30B-GGUF</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Text only. This path is not optimized. SGLang shows a warning at startup.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong>NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}><code>RadixArk/Muse-Glimmer-NVFP4</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Text only. Ready to serve, no conversion needed.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong>MLX Q4</strong></td>
|
||||
<td style={{padding: "9px 12px"}}><code>RadixArk/Muse-Glimmer-q4-MLX</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong>MLX Q4_K_M (gs128)</strong></td>
|
||||
<td style={{padding: "9px 12px"}}><code>RadixArk/Muse-Glimmer-q4km-gs128-MLX</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Text only. Apple Silicon (MLX backend). Carries the vendor GGUF's exact quantization codes in MLX format. The measured MLX artifact. See §3.4.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong>MLX Q4_K (dynamic)</strong></td>
|
||||
<td style={{padding: "9px 12px"}}><code>RadixArk/Muse-Glimmer-q4k-dynamic-MLX</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Resources:** [Muse-Glimmer-30B (BF16)](https://huggingface.co/meta-models/Muse-Glimmer-30B) · [Muse-Glimmer-30B-assistant (DFlash draft)](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) · [Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) · [Muse-Glimmer-NVFP4](https://huggingface.co/RadixArk/Muse-Glimmer-NVFP4) · MLX · [q4](https://huggingface.co/RadixArk/Muse-Glimmer-q4-MLX) · [q4km-gs128](https://huggingface.co/RadixArk/Muse-Glimmer-q4km-gs128-MLX) · [q4k-dynamic](https://huggingface.co/RadixArk/Muse-Glimmer-q4k-dynamic-MLX).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**The GGUF format is text only.** SGLang has no `mmproj` path. You cannot use the vision GGUF files. Use the BF16 checkpoint for multimodal input.
|
||||
|
||||
**The NVFP4 checkpoint.** `RadixArk/Muse-Glimmer-NVFP4` is a ready-to-serve NVFP4 + MXFP8 checkpoint. No conversion needed — point `--model-path` straight at it.
|
||||
|
||||
**The DFlash draft.** `meta-models/Muse-Glimmer-30B-assistant` is the vendor's native draft export and serves directly. No conversion needed.
|
||||
|
||||
**DFlash with a GGUF target model** needs `--speculative-draft-load-format auto`. Without this flag, the draft model uses the `gguf` load format from the target model. The loader then rejects the draft directory.
|
||||
|
||||
**Apple Silicon uses an MLX checkpoint, not the GGUF files.** The MLX backend has no GGUF path. Serve one of the three `RadixArk/Muse-Glimmer-*-MLX` artifacts with `SGLANG_USE_MLX=1` (see the Apple Silicon cells in the command panel). All three take the same flags; `q4km-gs128` is the one with a measured round. Keep `--disable-radix-cache` — the windowed KV storage for the sliding-window layers requires it — and set `SGLANG_MLX_CACHE_LIMIT_GB=8` so the MLX buffer cache does not grow the footprint under concurrent load. Speculative decoding is not available on the MLX backend.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Muse Glimmer enables the `muse` reasoning parser by default. This parser separates the reasoning text from the final answer.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="meta-models/Muse-Glimmer-30B",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Muse Glimmer enables the `muse` tool-call parser by default. This parser sends structured tool calls in `message.tool_calls`.
|
||||
|
||||
### 3.3 Multimodal
|
||||
|
||||
The BF16 checkpoint supports image input. It defaults to text only. To switch, select **Modality** in the command panel above.
|
||||
|
||||
**Text only** adds `--language-model-only`. This flag turns off the vision tower. SGLang does not build or load the vision weights. This frees memory for the KV cache. SGLang rejects image requests in this mode.
|
||||
|
||||
Select **Image + text** to turn on image input.
|
||||
|
||||
NVFP4, GGUF, and the MLX artifacts are text only. The Modality option does not appear for GGUF or MLX; NVFP4 only offers **Text only**.
|
||||
|
||||
### 3.4 Apple Silicon (MLX)
|
||||
|
||||
The MLX backend serves three Muse Glimmer artifacts on Apple Silicon Macs (48 GB unified memory or more). All three are text only — the MLX backend has no vision path — and all three take the same flags, so pick one in the command panel:
|
||||
|
||||
- `RadixArk/Muse-Glimmer-q4-MLX` — no measured round yet.
|
||||
- `RadixArk/Muse-Glimmer-q4km-gs128-MLX` — a lossless repack of the vendor's Q4_K_M (gs128) GGUF: every weight keeps the GGUF's exact quantization code, with the group scales re-expressed in MLX affine bf16 (≤2⁻⁸ relative rounding). The numbers below are for this artifact.
|
||||
- `RadixArk/Muse-Glimmer-q4k-dynamic-MLX` — no measured round yet.
|
||||
|
||||
Choose along the speed-versus-accuracy axis: footprint and expected accuracy both grow `q4` → `q4km-gs128` → `q4k-dynamic`, and decode speed moves the other way. Decode on Apple Silicon is memory-bandwidth-bound, so a smaller artifact reads fewer weight bytes per token — more tokens per second, and more unified memory left over for the KV cache. Take `q4` for the fastest responses on the smallest machine, `q4k-dynamic` to stay closest to BF16, and `q4km-gs128` for the middle ground — it is also the only one of the three with a measured round, below.
|
||||
|
||||
This table shows accuracy for the gs128 checkpoint, with the vendor llama.cpp fork serving the source GGUF on the same machine as the reference. GSM8K: 200 questions, no-thinking chat template, temperature 0, max 2048 new tokens. CIMemories: 1 profile, full combo, single trial, DeepSeek-R1-0528 judge.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Benchmark</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>SGLang MLX</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>llama.cpp (same GGUF)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>GSM8K (200q, no-thinking, greedy)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>0.970</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>0.970</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>CIMemories — violation rate (lower is better)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>0.00%</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>8.27%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>CIMemories — coverage (higher is better)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>76.0%</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>68.4%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
CIMemories is a single-trial benchmark with a nondeterministic judge; treat the SGLang-vs-llama.cpp gap on that row as run noise, not a runtime effect. GSM8K parity is exact.
|
||||
|
||||
Decode throughput for gs128 on an M5 Pro (64 GB), 1k-in/1k-out greedy: 15.3 tok/s at batch 1, rising to 52.6 tok/s aggregate at batch 8 — ahead of llama.cpp on the same GGUF codes at every batch size above 1.
|
||||
Reference in New Issue
Block a user