Files
sglang/docs_new/docs/hardware-platforms/apple_metal.mdx
T

99 lines
2.9 KiB
Plaintext

---
title: "Apple Silicon with Metal"
metatags:
description: "Run SGLang on Apple Silicon using the Metal backend."
---
This document describes how run SGLang on Apple Silicon using [Metal (MLX)](https://opensource.apple.com/projects/mlx/). If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## Prerequisites
Building the native Metal kernels in `sgl-kernel` requires the Apple
toolchain (`clang++`, the Metal framework headers, and `xcrun`). These ship
with the **Xcode Command Line Tools**, which cannot be installed via `pip`:
```bash
xcode-select --install
```
If you have the full Xcode app installed, the Command Line Tools are already
available. You can verify with `xcode-select -p && xcrun --find metal`.
## Install SGLang
You can install SGLang using one of the methods below.
### Install from Source
```bash
# Use the default branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
# Create and activate a virtual environment
uv venv -p 3.12 sglang-metal
source sglang-metal/bin/activate
# (Optional) Compile sgl-kernel
uv pip install --upgrade pip
uv run sgl-kernel/setup_metal.py install
# Install sglang python package along with diffusion support
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
uv pip install -e "python[all_mps]"
```
## Launch of the Serving Engine
Launch the server with:
```bash
SGLANG_USE_MLX=1 python -m sglang.launch_server \
--model <MODEL_ID_OR_PATH> \
--disable-cuda-graph \
--host 0.0.0.0
```
**Key Parameters Explained:**
1. `SGLANG_USE_MLX=1` - Enables the use of MLX as the SGLang runtime backend (if disabled, SGLang will fall back to `torch.mps`, which has less support)
2. `--disable-cuda-graph` - Disables usage of CUDA graph, which is not relevant for Apple Metal.
3. `--disable-overlap-schedule` - Disables overlap scheduling (enabled/not present by default) achieved using MLX's `async_eval()`
## Benchmarking with Requests
`sglang.benchmark_one_batch` calls the synchronous prefill/decode methods directly without going through the scheduler and the overlap code path.
`sglang.benchmark_offline_throughput` can toggle overlap scheduling as it uses the scheduler and the overlap code path by using the flag `--disable-overlap-schedule`.
### Throughput Testing
Basic synchronous one batch throughput:
```bash
SGLANG_USE_MLX=1 python -m sglang.bench_one_batch \
--model-path <MODEL_ID_OR_PATH> \
--disable-cuda-graph \
--tp-size 1 \
--batch-size 1 \
--input-len 60 \
--output-len 10
```
Synchronous offline throughput:
```bash
SGLANG_USE_MLX=1 python -m sglang.bench_offline_throughput \
--model-path <MODEL_ID_OR_PATH> \
--disable-cuda-graph \
--num-prompts 1 \
--disable-overlap-schedule
```
Asynchronous offline throughput:
```bash
SGLANG_USE_MLX=1 python -m sglang.bench_offline_throughput \
--model-path <MODEL_ID_OR_PATH> \
--disable-cuda-graph \
--num-prompts 1
```