sglang/docs/start/send_request.ipynb

Quick Start: Sending Requests

This notebook provides a quick-start guide for using SGLang after installation.

Launch a server

This code block is equivalent to executing

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0

in your terminal and wait for the server to be ready. Once the server is running, you can send test requests using curl or requests. The server implements the OpenAI-compatible API.

In [ ]:

from sglang.utils import (
    execute_shell_command,
    wait_for_server,
    terminate_process,
    print_highlight,
)

server_process = execute_shell_command(
"""
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0
"""
)

wait_for_server("http://localhost:30000")

Using cURL

In [ ]:

import subprocess, json

curl_command = """
curl -s http://localhost:30000/v1/chat/completions \
  -d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is a LLM?"}]}'
"""

response = json.loads(subprocess.check_output(curl_command, shell=True))
print_highlight(response)

Using OpenAI Compatible API w/ Requests

In [ ]:

import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is a LLM?"}
    ]
}

response = requests.post(url, json=data)
print_highlight(response.json())

Using OpenAI Python Client

You can also use the OpenAI Python API library to send requests.

In [ ]:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print_highlight(response)

Using Native Generation APIs

You can also use the native /generate endpoint with requests, which provides more flexiblity. An API reference is available at Sampling Parameters.

In [ ]:

import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
    },
)

print_highlight(response.json())

In [6]:

terminate_process(server_process)