sglang/docs_new/docs/hardware-platforms/amd-gpus.mdx

---
title: "AMD GPUs"
---
This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).

## System Configuration

When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:

* [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
* [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
* [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
* [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
* [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)

<Note>
We strongly recommend reading these docs and guides entirely to fully utilize your system.
</Note>

Below are a few key settings to confirm or enable for SGLang:

### Update GRUB Settings

In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:

<CodeGroup>
```text GRUB Configuration
pci=realloc=off iommu=pt
```
</CodeGroup>

Afterward, run `sudo update-grub` (or your distro's equivalent) and reboot.

### Disable NUMA Auto-Balancing

<CodeGroup>
```bash Disable NUMA
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
```
</CodeGroup>

You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).

Again, please go through the entire documentation to confirm your system is using the recommended configuration.

## Install SGLang

<Tabs>
  <Tab title="Docker (Recommended)">
    The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).

    1. **Build the docker image**
        If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.

        <CodeGroup>
        ```bash Build Image
        docker build -t sglang_image -f rocm.Dockerfile .
        ```
        </CodeGroup>

    2. **Create a convenient alias**
        <CodeGroup>
        ```bash Create Alias
        alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
            --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
            --security-opt seccomp=unconfined \
            -v $HOME/dockerx:/dockerx \
            -v /data:/data'
        ```
        </CodeGroup>

        If you are using RDMA, please note that:

        * `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
        * You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.

    3. **Launch the server**
        <Note>
        Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
        </Note>

        <CodeGroup>
        ```bash Launch Server
        drun -p 30000:30000 \
            -v ~/.cache/huggingface:/root/.cache/huggingface \
            --env "HF_TOKEN=<secret>" \
            sglang_image \
            python3 -m sglang.launch_server \
            --model-path NousResearch/Meta-Llama-3.1-8B \
            --host 0.0.0.0 \
            --port 30000
        ```
        </CodeGroup>

    4. **Verify the installation**
        You can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.

        <CodeGroup>
                    ```bash Run Benchmark
        drun sglang_image \
            python3 -m sglang.bench_serving \
            --backend sglang \
            --dataset-name random \
            --num-prompts 4000 \
            --random-input 128 \
            --random-output 128
        ```
        </CodeGroup>

    With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang's machine learning capabilities.
  </Tab>

  <Tab title="From Source">
    1. **Clone the repository**
        Clone the SGLang repository.

        <CodeGroup>
        ```bash
        git clone https://github.com/sgl-project/sglang.git
        cd sglang
        ```
        </CodeGroup>

    2. **Compile sgl-kernel**
        Upgrade pip and compile the sgl-kernel for ROCm support.

        <CodeGroup>
        ```bash
        pip install --upgrade pip
        cd sgl-kernel
        python setup_rocm.py install
        ```
        </CodeGroup>

    3. **Install sglang package**
        Install the SGLang Python package with HIP and diffusion support.

        <CodeGroup>
        ```bash
        cd ..
        rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
        pip install -e "python[all_hip]"
        ```
        </CodeGroup>
  </Tab>
</Tabs>

## Examples

### Running DeepSeek-V3

The only difference when running DeepSeek-V3 is in how you start the server.

<CodeGroup>
```bash DeepSeek-V3
drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    --env "HF_TOKEN=<secret>" \
    sglang_image \
    python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000
```
</CodeGroup>

[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.

### Running Llama3.1

Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server.

<CodeGroup>
```bash Llama3.1
drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    --env "HF_TOKEN=<secret>" \
    sglang_image \
    python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tp 8 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000
```
</CodeGroup>

### Warmup Step

When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.