Muse Glimmer Cookbook (#34271)

Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai> Co-authored-by: Jimmy Shong <jimmysh341@gmail.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-08-10 10:21:13 +00:00
parent 955569a2dc
commit a6c34df044
11 changed files with 761 additions and 10 deletions
@@ -0,0 +1,235 @@
+---
+title: Llama-3.3-70B
+metatags:
+    description: "Deploy Llama-3.3-70B-Instruct with SGLang on AMD GPUs - 128K context, enhanced reasoning, tool calling, and multilingual support."
+---
+## 1. Model Introduction
+
+[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.
+
+**Key Features:**
+
+- **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
+- **Tool Calling**: Native support for function calling and tool use scenarios
+- **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
+- **Extended Context**: 128K token context window for processing long documents and complex tasks
+- **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X
+
+**License:**
+Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.
+
+For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
+
+For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs.
+
+### 3.1 Interactive Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
+
+import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
+
+<Llama33Deployment />
+
+### 3.2 Configuration Tips
+
+**AMD GPU Deployment:**
+
+- All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
+- **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
+- **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
+- **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput
+
+**Xeon CPU Deployment:**
+
+Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Tool Calling
+
+Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.3-70B-Instruct \
+  --tool-call-parser llama3 \
+  --tp 1 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+**Python Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request
+response = client.chat.completions.create(
+    model="meta-llama/Llama-3.3-70B-Instruct",
+    messages=[
+        {"role": "user", "content": "What's the weather in Tokyo?"}
+    ],
+    tools=tools,
+    temperature=0.7
+)
+
+# Check for tool calls
+message = response.choices[0].message
+if message.tool_calls:
+    tool_call = message.tool_calls[0]
+    print(f"Function: {tool_call.function.name}")
+    print(f"Arguments: {tool_call.function.arguments}")
+```
+
+**Handling Tool Call Results:**
+
+```python Example
+# After executing the function, send the result back
+def get_weather(location, unit="celsius"):
+    # Your weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Build conversation with tool result
+messages = [
+    {"role": "user", "content": "What's the weather in Tokyo?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Tokyo", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Tokyo", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="meta-llama/Llama-3.3-70B-Instruct",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
+```
+
+#### 4.2.2 Long Context Processing
+
+Leverage the 128K context window for processing long documents:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Example with long document
+long_document = "..." * 10000  # Your long document here
+
+response = client.chat.completions.create(
+    model="meta-llama/Llama-3.3-70B-Instruct",
+    messages=[
+        {"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
+    ],
+    temperature=0.7,
+    max_tokens=1000
+)
+
+print(response.choices[0].message.content)
+```
+
+## 5. Benchmarking
+
+Use the SGLang benchmarking suite to test model performance with different workload patterns:
+
+### 5.1 Basic Benchmark Command
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --dataset-name random \
+  --num-prompts 1000 \
+  --random-input 1024 \
+  --random-output 1024 \
+  --max-concurrency 16
+```
+
+### 5.2 Adjusting Benchmark Parameters
+
+**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:
+
+- Short conversations: `--random-input 1024 --random-output 1024`
+- Long outputs: `--random-input 1024 --random-output 8192`
+- Long inputs: `--random-input 8192 --random-output 1024`
+
+**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:
+
+- Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
+- Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
+- High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`
+
+---
+
+## 📚 Additional Resources
+
+- [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
+- [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
+- [SGLang Documentation](/)
+- [AMD ROCm Documentation](https://rocm.docs.amd.com/)