diff --git a/docs_new/cookbook/autoregressive/GLM/GLM-5.2.mdx b/docs_new/cookbook/autoregressive/GLM/GLM-5.2.mdx index 1ea7240bd..8275a54b9 100644 --- a/docs_new/cookbook/autoregressive/GLM/GLM-5.2.mdx +++ b/docs_new/cookbook/autoregressive/GLM/GLM-5.2.mdx @@ -201,3 +201,27 @@ Tool calls: [ ### 3.3 HiCache (Hierarchical KV Caching) For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern. + +### 3.4 Claude Code Integration + +GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then: + +```bash Command +export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" +export ANTHROPIC_AUTH_TOKEN="dummy" +export API_TIMEOUT_MS="3000000" +export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" +export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 +export CLAUDE_CODE_ATTRIBUTION_HEADER=0 +export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" +export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" +export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" +claude +``` + +Two of these matter specifically for GLM-5.2: + +- **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse. +- **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side. + +For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api). diff --git a/docs_new/docs.json b/docs_new/docs.json index 3a11f6353..f5e049a66 100644 --- a/docs_new/docs.json +++ b/docs_new/docs.json @@ -707,6 +707,7 @@ "docs/basic_usage/openai_api_embeddings" ] }, + "docs/basic_usage/anthropic_api", "docs/basic_usage/ollama_api", "docs/basic_usage/offline_engine_api", "docs/basic_usage/native_api", diff --git a/docs_new/docs/basic_usage/anthropic_api.mdx b/docs_new/docs/basic_usage/anthropic_api.mdx new file mode 100644 index 000000000..5b42f2103 --- /dev/null +++ b/docs_new/docs/basic_usage/anthropic_api.mdx @@ -0,0 +1,287 @@ +--- +title: "Anthropic-Compatible API" +description: "Use the Anthropic Messages API (/v1/messages) with SGLang, including Claude Code integration and prefix-cache tuning." +--- +SGLang ships an Anthropic-compatible `/v1/messages` endpoint so any client built for the Anthropic +Messages API — including the Anthropic SDKs and agentic CLIs such as Claude Code — can talk to a +self-hosted SGLang server without changes. A complete reference for the API is available in the +[Anthropic API Reference](https://docs.anthropic.com/en/api/messages). + +The endpoint is registered automatically on every SGLang server; no extra flag is required to enable it. +It reuses the same model, chat template, and reasoning / tool-call parsers as the OpenAI-compatible +endpoint, and supports both non-streaming and streaming responses, tool use, and a `count_tokens` route. + +This tutorial covers: + +- `POST /v1/messages` (non-streaming and streaming) +- `POST /v1/messages/count_tokens` +- Pointing **Claude Code** at the server, including the `CLAUDE_CODE_ATTRIBUTION_HEADER` setting that is + required for good prefix-cache reuse. + +## Launch A Server + +Launch the server in your terminal and wait for it to initialize. The Anthropic `/v1/messages` endpoint +is registered automatically — no extra flag is required beyond the usual server launch. The example below +is a single-node GLM-5.2-FP8 config; see the +[GLM-5.2 cookbook](/cookbook/autoregressive/GLM/GLM-5.2) for verified commands +across hardware and quantizations. + +```bash Command +sglang serve \ + --model-path zai-org/GLM-5.2-FP8 \ + --tp 8 \ + --speculative-algorithm EAGLE \ + --speculative-num-steps 5 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 6 \ + --reasoning-parser glm45 \ + --tool-call-parser glm47 \ + --host 0.0.0.0 \ + --port 30000 +``` + + +- **The endpoint is model-agnostic.** The `/v1/messages` route is on by default for any model; GLM-5.2 is + used here because its reasoning + tool-use output is where Claude Code integration shines, but any model + works. +- **Model name and `[1m]`.** SGLang does not validate the request `model` field, so Claude Code can send + any name. The `[1m]` suffix is a **client-side hint**: Claude Code only enables its 1M-context beta when + the model name ends in `[1m]` — without it, context is capped. Set the same `glm-5.2[1m]` in the + `ANTHROPIC_DEFAULT_*_MODEL` env vars below. +- **`--reasoning-parser` / `--tool-call-parser` are optional.** Add them when the model emits reasoning + content (GLM-5.2, Qwen3, DeepSeek-R1, …) or when you want tool calls parsed into structured `tool_use` + blocks. Without a tool-call parser, tool schemas are still accepted but the model's tool calls come back + as raw text, and Claude Code cannot execute them. +- **Context length** defaults to the model's own (1M for GLM-5.2); pass `--context-length` only to cap it. + + +## Send A Message + +### Non-Streaming + +Use the Anthropic Python SDK pointed at the server. Unlike the OpenAI SDK, the Anthropic SDK appends +`/v1/messages` itself, so `base_url` is the server root **without** a `/v1` suffix. + +```python Example +from anthropic import Anthropic + +client = Anthropic( + base_url="http://127.0.0.1:30000", + api_key="EMPTY", # SGLang does not require a real key by default +) + +message = client.messages.create( + model="zai-org/GLM-5.2-FP8", + max_tokens=512, + messages=[{"role": "user", "content": "List 3 countries and their capitals."}], +) +# A reasoning model may emit a `thinking` block before the `text` block — +# pick the text block rather than assuming content[0]. +print(next(b.text for b in message.content if b.type == "text")) +``` + +**Example Output:** +```text Output +Here are 3 countries and their capitals: + +1. **France** - Paris +2. **Japan** - Tokyo +3. **Brazil** - Brasília +``` + +### Streaming + +Set `stream=True` to receive Server-Sent Events as they are produced. + +```python Example +with client.messages.stream( + model="zai-org/GLM-5.2-FP8", + max_tokens=512, + messages=[{"role": "user", "content": "Say this is a test"}], +) as stream: + for text in stream.text_stream: + print(text, end="", flush=True) +``` + +**Example Output:** +```text Output +This is a test. +``` + +### System Prompt + +The top-level `system` field is accepted as a string or as a list of text blocks, matching the Anthropic +API shape: + +```python Example +message = client.messages.create( + model="zai-org/GLM-5.2-FP8", + max_tokens=512, + system="You are a helpful assistant that answers concisely.", + messages=[{"role": "user", "content": "What is the capital of France?"}], +) +print(next(b.text for b in message.content if b.type == "text")) +``` + +**Example Output:** +```text Output +The capital of France is Paris. +``` + +### Tool Use + +Tool definitions follow the Anthropic `tools` schema. When the server is launched with a +`--tool-call-parser`, the model's tool calls are returned as `tool_use` content blocks: + +```python Example +message = client.messages.create( + model="zai-org/GLM-5.2-FP8", + max_tokens=512, + tools=[ + { + "name": "get_weather", + "description": "Get the weather for a city", + "input_schema": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + } + ], + messages=[{"role": "user", "content": "What is the weather in Paris?"}], +) +print(message.stop_reason) +print([b for b in message.content if b.type == "tool_use"]) +``` + +**Example Output:** +```text Output +tool_use +[ToolUseBlock(type='tool_use', id='toolu_01XXXX', name='get_weather', input={'city': 'Paris'})] +``` + +### Counting Tokens + +`POST /v1/messages/count_tokens` returns the tokenized length of a request without generating a +response. It reuses the same request conversion as `/v1/messages`, so system prompts, tools, and +multi-turn history are all accounted for. + +```python Example +resp = client.messages.count_tokens( + model="zai-org/GLM-5.2-FP8", + messages=[{"role": "user", "content": "Hello, world"}], +) +print(resp.input_tokens) +``` + +**Example Output:** +```text Output +15 +``` + +## Using Claude Code + +Claude Code can be pointed at an SGLang server by setting a few env vars in the shell that starts it. +With the server already running on `:30000`, export the full set and launch `claude`: + +```bash Command +export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" +export ANTHROPIC_AUTH_TOKEN="dummy" # required by Claude Code; any non-empty string works +export API_TIMEOUT_MS="3000000" # long timeout — reasoning + 1M-context turns are slow +export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" # let auto-compact use the full 1M window +export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 # drop autoupdater/telemetry/error-reporting noise +export CLAUDE_CODE_ATTRIBUTION_HEADER=0 # required for prefix-cache reuse — see below +export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta +export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta +export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta +claude +``` + +Each var matters: + +- **`ANTHROPIC_BASE_URL`** — points Claude Code at your SGLang server instead of the Anthropic API. +- **`ANTHROPIC_AUTH_TOKEN`** — Claude Code requires a non-empty auth token; SGLang accepts any value + when launched without `--api-key`. +- **`API_TIMEOUT_MS`** — raise it; reasoning models with long outputs and 1M-context turns routinely + exceed the default timeout. +- **`ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL`** — the model name Claude Code sends for each tier. + SGLang does not validate this field, so any name works. Use `glm-5.2[1m]`: the `[1m]` suffix is a + client-side hint that enables Claude Code's 1M-context beta (without it, context is capped). +- **`CLAUDE_CODE_AUTO_COMPACT_WINDOW`** — set to `1000000` so auto-compaction uses the full 1M window + instead of the default, keeping long sessions alive. + + +Instead of exporting these in every shell, persist them in `~/.claude/settings.json` under the `env` key +— they apply to all Claude Code sessions: + +```json +{ + "env": { + "ANTHROPIC_BASE_URL": "http://127.0.0.1:30000", + "ANTHROPIC_AUTH_TOKEN": "dummy", + "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", + "CLAUDE_CODE_ATTRIBUTION_HEADER": "0", + "ANTHROPIC_DEFAULT_HAIKU_MODEL": "glm-5.2[1m]", + "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-5.2[1m]", + "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-5.2[1m]" + } +} +``` + + +### Required: `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for prefix-cache reuse + + +**Set this whenever Claude Code routes through SGLang (or any non-Anthropic gateway).** Without it, +multi-turn conversations re-prefill the whole history every turn. + + +Claude Code prepends a per-request attribution block to the start of the system prompt, of the form +`x-anthropic-billing-header: cc_version=.; cc_entrypoint=...; cch=;`. The +per-request hash is the **first token to differ between turns**, so the radix prefix cache can only reuse +the short prefix before that hash and re-prefills the system prompt plus the entire conversation history +on every turn. + +Setting `CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes the whole attribution line from the system prompt. +This is a documented Claude Code env var whose explicit purpose is to "improve prompt-cache hit rates when +routing through an [LLM gateway](https://code.claude.com/docs/en/llm-gateway)" (see the [Claude Code env-vars reference](https://code.claude.com/docs/en/env-vars)). + + +`CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** remove the attribution block — it only covers +autoupdater/telemetry/error reporting. The attribution header is a separate code path; use +`CLAUDE_CODE_ATTRIBUTION_HEADER=0` for it. + + +## Troubleshooting + +**Connection refused / `fetch failed`** — Ensure the server is up and the port in `ANTHROPIC_BASE_URL` +matches `--port` (default 30000). If you set `ANTHROPIC_BASE_URL` to a remote host, confirm it's reachable +and not behind a proxy that blocks the connection. + +**`Model not found` / 404 from the server** — SGLang does not validate the request `model` field and +serves whatever model was loaded at startup, so a 404 usually means the request did not reach the +`/v1/messages` route at all. Confirm `ANTHROPIC_BASE_URL` points at the server (not missing the port) and +that the server finished loading. + +**Tool calls not working / returned as raw text** — Launch the server with the correct +`--tool-call-parser` for your model (e.g. `glm47`, `qwen3`). Without it the `tools` field is still accepted +but the model's tool calls come back as text instead of `tool_use` blocks, and Claude Code cannot execute +them. + +**Slow / re-prefills the whole history every turn** — You are missing +`CLAUDE_CODE_ATTRIBUTION_HEADER=0`. Claude Code's per-request attribution hash in the system prompt +defeats radix prefix-cache reuse; see the section above. + +**Context capped below 1M** — The model name must end in `[1m]` for Claude Code to enable its 1M-context +beta. Verify `ANTHROPIC_DEFAULT_*_MODEL` uses the `[1m]` suffix, and that the loaded model's native context +is 1M (GLM-5.2 is 1048576; pass `--context-length` only to cap it, not to extend). + +## Parameters + +The `/v1/messages` endpoint accepts the standard Anthropic Messages API parameters. Refer to the +[Anthropic Messages API reference](https://docs.anthropic.com/en/api/messages) for the full list. + +Reasoning models are supported through the same `--reasoning-parser` mechanism as the OpenAI-compatible +endpoint; pass the model's reasoning kwarg via the request (e.g. `thinking` for DeepSeek-V3-style models, +`enable_thinking` for Qwen3-style models). See [OpenAI APIs - Completions](./openai_api_completions) for +the reasoning-parser / chat-template mapping. diff --git a/docs_new/docs/basic_usage/openai_api.mdx b/docs_new/docs/basic_usage/openai_api.mdx index 523fa2db6..9951c0399 100644 --- a/docs_new/docs/basic_usage/openai_api.mdx +++ b/docs_new/docs/basic_usage/openai_api.mdx @@ -5,3 +5,5 @@ description: "Documentation for OpenAI-Compatible APIs" - [Openai Api Completions](./openai_api_completions) - [Openai Api Vision](./openai_api_vision) - [Openai Api Embeddings](./openai_api_embeddings) + +For the Anthropic-compatible `/v1/messages` endpoint (including Claude Code integration), see [Anthropic-Compatible API](./anthropic_api). diff --git a/docs_new/docs/basic_usage/overview.mdx b/docs_new/docs/basic_usage/overview.mdx index 5004cfd3c..f50f74342 100644 --- a/docs_new/docs/basic_usage/overview.mdx +++ b/docs_new/docs/basic_usage/overview.mdx @@ -4,6 +4,7 @@ description: Core APIs and common usage patterns for SGLang. --- - [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings +- [Anthropic-Compatible API](./anthropic_api) — `/v1/messages`, including Claude Code integration - [Ollama API](./ollama_api) - [Offline Engine API](./offline_engine_api) - [Native API](./native_api)