minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

[PD] Add /v1/responses support to the HTTP PD router (#36141)

Browse Source 
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
This commit is contained in:
lz
2026-09-13 22:48:13 +08:00
committed by GitHub
co-authored by Shangming Cai Xinyuan Tong
parent 7f1f8c706a
commit 6220f45d8e
3 changed files with 564 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/advanced_features/pd_disaggregation.mdx
+19
View File
@@ -28,6 +28,25 @@ When you need to profile prefill or decode workers in PD disaggregation mode, pl
For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation).
### Responses API under HTTP PD
HTTP PD supports foreground generation through `POST /v1/responses`, including streaming generation with `stream=true` through the HTTP PD router. This requires both the router's Responses dispatch support and serving workers that accept Responses PD routing metadata.
Store-backed and stateful Responses features are **not supported under PD**:
- Response persistence (`store`) and response history.
- Conversation chaining with `previous_response_id`.
- Background Responses (`background=true`), including background streams.
- Retrieval by response ID (`GET /v1/responses/{id}`).
- Cancel-by-ID (`POST /v1/responses/{id}/cancel`).
- Other workflows requiring shared process-local Responses history.
Prefill and decode run in separate serving processes. Their process-local Responses stores cannot provide coherent shared state: both legs need the same conversation history to reconstruct the prompt, and prefill's stored response is not available to decode. Send explicit conversation history in foreground requests instead of relying on stored response IDs.
Built-in tools (`web_search`, `code_interpreter`) are also unsupported under PD: each tool call requires another generation, but the router dispatches exactly one prefill/decode pair per HTTP request.
Serving-side capability enforcement is tracked in [#39122](https://github.com/sgl-project/sglang/pull/39122). That change introduces opt-in standalone storage with `--enable-response-store`, disallows it under PD, and rejects stateful requests with HTTP 400 before generation. These checks require a serving version containing that change; older versions must not be assumed to reject every unsupported workflow at admission. The HTTP PD router also rejects detached background requests because it does not implement their retrieval/cancel lifecycle.
## Mooncake
### Requirements