Add new Mintlify documentation site (docs_new/) (#23001)

Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-04-20 15:10:22 -07:00
parent 575fdc2c4c
commit a3291b5654
330 changed files with 100371 additions and 0 deletions
@@ -0,0 +1,194 @@
+---
+title: "AMD GPUs"
+---
+This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
+
+## System Configuration
+
+When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
+
+* [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
+* [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
+* [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
+* [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
+* [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
+
+<Note>
+We strongly recommend reading these docs and guides entirely to fully utilize your system.
+</Note>
+
+Below are a few key settings to confirm or enable for SGLang:
+
+### Update GRUB Settings
+
+In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
+
+<CodeGroup>
+```text GRUB Configuration
+pci=realloc=off iommu=pt
+```
+</CodeGroup>
+
+Afterward, run `sudo update-grub` (or your distro's equivalent) and reboot.
+
+### Disable NUMA Auto-Balancing
+
+<CodeGroup>
+```bash Disable NUMA
+sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
+```
+</CodeGroup>
+
+You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
+
+Again, please go through the entire documentation to confirm your system is using the recommended configuration.
+
+## Install SGLang
+
+<Tabs>
+  <Tab title="Docker (Recommended)">
+    The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
+
+    1. **Build the docker image**
+        If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
+
+        <CodeGroup>
+        ```bash Build Image
+        docker build -t sglang_image -f rocm.Dockerfile .
+        ```
+        </CodeGroup>
+
+    2. **Create a convenient alias**
+        <CodeGroup>
+        ```bash Create Alias
+        alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
+            --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
+            --security-opt seccomp=unconfined \
+            -v $HOME/dockerx:/dockerx \
+            -v /data:/data'
+        ```
+        </CodeGroup>
+
+        If you are using RDMA, please note that:
+
+        * `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
+        * You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
+
+    3. **Launch the server**
+        <Note>
+        Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
+        </Note>
+
+        <CodeGroup>
+        ```bash Launch Server
+        drun -p 30000:30000 \
+            -v ~/.cache/huggingface:/root/.cache/huggingface \
+            --env "HF_TOKEN=<secret>" \
+            sglang_image \
+            python3 -m sglang.launch_server \
+            --model-path NousResearch/Meta-Llama-3.1-8B \
+            --host 0.0.0.0 \
+            --port 30000
+        ```
+        </CodeGroup>
+
+    4. **Verify the installation**
+        You can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.
+
+        <CodeGroup>
+                    ```bash Run Benchmark
+        drun sglang_image \
+            python3 -m sglang.bench_serving \
+            --backend sglang \
+            --dataset-name random \
+            --num-prompts 4000 \
+            --random-input 128 \
+            --random-output 128
+        ```
+        </CodeGroup>
+
+    With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang's machine learning capabilities.
+  </Tab>
+
+  <Tab title="From Source">
+    1. **Clone the repository**
+        Clone the SGLang repository.
+
+        <CodeGroup>
+        ```bash
+        git clone https://github.com/sgl-project/sglang.git
+        cd sglang
+        ```
+        </CodeGroup>
+
+    2. **Compile sgl-kernel**
+        Upgrade pip and compile the sgl-kernel for ROCm support.
+
+        <CodeGroup>
+        ```bash
+        pip install --upgrade pip
+        cd sgl-kernel
+        python setup_rocm.py install
+        ```
+        </CodeGroup>
+
+    3. **Install sglang package**
+        Install the SGLang Python package with HIP and diffusion support.
+
+        <CodeGroup>
+        ```bash
+        cd ..
+        rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
+        pip install -e "python[all_hip]"
+        ```
+        </CodeGroup>
+  </Tab>
+</Tabs>
+
+## Examples
+
+### Running DeepSeek-V3
+
+The only difference when running DeepSeek-V3 is in how you start the server.
+
+<CodeGroup>
+```bash DeepSeek-V3
+drun -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --ipc=host \
+    --env "HF_TOKEN=<secret>" \
+    sglang_image \
+    python3 -m sglang.launch_server \
+    --model-path deepseek-ai/DeepSeek-V3 \
+    --tp 8 \
+    --trust-remote-code \
+    --host 0.0.0.0 \
+    --port 30000
+```
+</CodeGroup>
+
+[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
+
+### Running Llama3.1
+
+Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server.
+
+<CodeGroup>
+```bash Llama3.1
+drun -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --ipc=host \
+    --env "HF_TOKEN=<secret>" \
+    sglang_image \
+    python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --tp 8 \
+    --trust-remote-code \
+    --host 0.0.0.0 \
+    --port 30000
+```
+</CodeGroup>
+
+### Warmup Step
+
+When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.