responses-shim v0.2.3: DeepSeek Responses -> chat completions shim
- /v1/responses 双向翻译(含流式 response.* 事件、tools、usage 全字段) - /v1/chat/completions 字节级透传 - STOP_SCOPE=content 内容侧 stop(保思维链 + matched_stop 回填) - b300 docker 构建 + 孔明 kaniko 构建 + 两版 cw manifest
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
shim.exe
|
||||
shim-linux
|
||||
@@ -0,0 +1,6 @@
|
||||
# responses-shim: 静态二进制,scratch 镜像(离线构建无需拉取任何镜像)
|
||||
# b300 构建: docker build -t ymkymx/responses-shim:v0.1.0 .
|
||||
FROM scratch
|
||||
COPY shim-linux /shim
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["/shim"]
|
||||
@@ -0,0 +1,15 @@
|
||||
# 孔明 kmctl kaniko 构建(dockerfile 模式,无构建上下文)
|
||||
# 不 COPY 本地文件,从 Gitee 匿名 clone 源码编译(与 sglang 的 build-kmctl/Dockerfile 同套路)。
|
||||
# 用法: kmctl --cluster <集群> create image responses-shim --build-mode dockerfile \
|
||||
# --dockerfile Dockerfile.kaniko --node-group <节点组> --wait
|
||||
FROM uhub.service.ucloud.cn/umirror/golang:1.23 AS build
|
||||
|
||||
RUN git clone --depth 1 https://gitee.com/ymkymkymkymx/responses-shim.git /src \
|
||||
&& cd /src \
|
||||
&& CGO_ENABLED=0 go build -ldflags="-s -w" -o /shim . \
|
||||
&& git rev-parse HEAD > /etc/shim-commit || true
|
||||
|
||||
FROM scratch
|
||||
COPY --from=build /shim /shim
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["/shim"]
|
||||
@@ -0,0 +1,43 @@
|
||||
# responses-shim
|
||||
|
||||
DeepSeek Responses API(`/v1/responses`)→ OpenAI Chat Completions 的协议转换 shim。
|
||||
后置部署在 new-api 与 sglang 之间:客户端打 new-api 的 `/v1/responses`,new-api 按原路径透传给
|
||||
本服务(渠道上游),本服务翻译后直连 sglang router。
|
||||
|
||||
## 功能
|
||||
|
||||
- `POST /v1/responses` → 翻译为 `/v1/chat/completions`:
|
||||
- `input`(纯字符串 / item 列表:message、function_call、function_call_output、custom_tool_call*、reasoning)→ `messages`
|
||||
- `instructions` → 首条 system 消息
|
||||
- `reasoning.effort`(none/low/high/max,别名 minimal→low、medium/xhigh→high、ultra→max)→ `chat_template_kwargs`
|
||||
- `max_output_tokens` → `max_tokens`;`text.format` → `response_format`;`tools`/`tool_choice` 拍平;
|
||||
`top_logprobs` / `user` / `stop` 透传
|
||||
- 响应:`choices` → `output[]`(reasoning / message / function_call item);usage 全字段
|
||||
(含 `cached_tokens` 缓存命中、`reasoning_tokens`)
|
||||
- 流式:chat SSE → `response.*` 事件序列(created/in_progress/output_item/delta/completed/incomplete/failed)
|
||||
- `POST /v1/chat/completions` → 字节级透传
|
||||
- `GET /healthz`
|
||||
|
||||
## STOP_SCOPE:内容侧 stop
|
||||
|
||||
`STOP_SCOPE=content`(生产配置)时,chat 与 responses 两条路径的 `stop` 都**不下发上游**,
|
||||
由 shim 在 content 流上跨 chunk 匹配截断:思维链(reasoning_content)完整生成不被误杀,
|
||||
命中后 `finish_reason=stop` + `matched_stop` 回填,流式命中即断开上游触发 sglang abort。
|
||||
`STOP_SCOPE=all`(默认)为原生语义:stop 透传上游,token 级截断(含思维链)。
|
||||
|
||||
## 环境变量
|
||||
|
||||
| 变量 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `UPSTREAM_BASE` | `http://127.0.0.1:50002` | sglang router 地址 |
|
||||
| `LISTEN` | `:8080` | 监听地址 |
|
||||
| `STOP_SCOPE` | `all` | `all` / `content` |
|
||||
|
||||
## 构建与部署
|
||||
|
||||
- 本地/b300 docker 构建(scratch,需先交叉编译):`GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -o shim-linux . && docker build -t ymkymx/responses-shim:vX .`
|
||||
- 孔明 kaniko 构建(无上下文,从 Gitee clone 源码):
|
||||
`kmctl create image responses-shim --build-mode dockerfile --dockerfile Dockerfile.kaniko --node-group <ng> --wait`
|
||||
- 部署清单:`deploy/`(b300 版 + h20 版),`kmctl create cw <name> -f <yaml> --node-group <ng>`
|
||||
|
||||
仅标准库,零依赖。源码 source of truth 在本仓库;b300 构建目录 `/data/ymk/build/responses-shim/` 是镜像构建缓存。
|
||||
@@ -0,0 +1,71 @@
|
||||
# responses-shim — DeepSeek Responses API (/v1/responses) → chat completions 协议转换
|
||||
# 架构: 客户端 → new-api(:31000, /v1/responses 原路径透传) → 本服务(渠道上游) → ds41-3p1d:50002
|
||||
# 部署: wsl kmctl create cw ds41-resp-shim -f /mnt/d/B300/build-kmctl/responses-shim-cw.yaml --node-group B300
|
||||
# 镜像: ymkymx/responses-shim:v0.1.0(scratch+静态 Go 二进制,5.6MB,
|
||||
# 已 docker save | ctr -n k8s.io images import 侧载到 b300-01;Kyverno 对未映射镜像保持原名)
|
||||
# 网络: 普通 Pod 网络 + ClusterIP Service(同 jaeger 之外的常规姿势);new-api 渠道 base URL 填
|
||||
# http://ds41-resp-shim:8080 (同 namespace 短名可达,与 ds41-3p1d 同理)
|
||||
# 透传: /v1/chat/completions 字节级透传(渠道测试/健康检查/误路由的 chat 请求都正常)
|
||||
# 源码: D:\B300\responses-shim\main.go(另存 b300 /data/ymk/build/responses-shim/)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: responses-shim
|
||||
labels:
|
||||
app: responses-shim
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # 无状态但无需滚动,重建越快越好
|
||||
selector:
|
||||
matchLabels:
|
||||
app: responses-shim
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: responses-shim
|
||||
spec:
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: b300-01
|
||||
containers:
|
||||
- name: shim
|
||||
image: ymkymx/responses-shim:v0.2.3
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: UPSTREAM_BASE
|
||||
value: "http://ds41-3p1d:50002"
|
||||
- name: LISTEN
|
||||
value: ":8080"
|
||||
- name: STOP_SCOPE
|
||||
value: "content" # stop 只在 content 上截断,思维链不被误杀(Claude stop_sequences 语义)
|
||||
ports:
|
||||
- {containerPort: 8080}
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 512Mi
|
||||
readinessProbe:
|
||||
httpGet: {path: /healthz, port: 8080}
|
||||
initialDelaySeconds: 2
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet: {path: /healthz, port: 8080}
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 20
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: ds41-resp-shim
|
||||
labels:
|
||||
app: responses-shim
|
||||
spec:
|
||||
selector:
|
||||
app: responses-shim
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
@@ -0,0 +1,4 @@
|
||||
<h3>2026-09-30 | new-api 参数覆盖链 + responses-shim 上线(v0.1.0→v0.2.3)</h3>
|
||||
<p><b>背景</b>:sglang 不支持官方 reasoning_effort / thinking 开关语义与 /v1/responses 协议,new-api 只能透传。</p>
|
||||
<p><b>改动</b>:①new-api 渠道参数覆盖链——thinking⇄chat_template_kwargs 映射、reasoning_effort 归一(含 minimal/medium/xhigh 别名)、temperature 思考模式钉 1(关思考跟随用户)、top_p 钳 0.95、max_tokens 缺省 65536 keep_origin;preflight 验收 13/13。②新建 responses-shim(Go stdlib 单文件,scratch 镜像 5.6MB),孔明 cw ds41-resp-shim(ClusterIP ds41-resp-shim:8080,钉 b300-01,上游 ds41-3p1d:50002);new-api 渠道上游改指 shim。当天迭代至 v0.2.3:内容侧 stop 截断(STOP_SCOPE=content)、usage details 透传、reasoning_tokens 收编、responses 路径 stop 修复。</p>
|
||||
<p><b>边界</b>:chat 路径响应 stop_sequence=null 属 new-api Claude 转换器限制救不了;previous_response_id 不支持;上游 sglang PR #5209 合入后 shim 可退役。详见六、new-api 网关层。</p>
|
||||
@@ -0,0 +1,52 @@
|
||||
<h1>六、new-api 网关层:参数覆盖链 + responses-shim(2026-09-30 上线)</h1>
|
||||
<p>对外入口统一走 new-api 网关(<code>http://125.93.200.207:31000/</code>,模型名 <code>deepseek-v4.1-flash</code>)。sglang 原生不支持的官方 API 参数(reasoning_effort / thinking 开关 / responses 协议)在网关层补齐:<b>new-api 渠道"参数覆盖"</b>负责采样参数归一,<b>responses-shim</b>(孔明 cw <code>ds41-resp-shim</code>)负责 /v1/responses ⇄ /v1/chat/completions 双向翻译。new-api 渠道上游已改为 shim(<code>http://ds41-resp-shim:8080</code>),shim 再转发 ds41-3p1d router(ClusterIP <code>ds41-3p1d:50002</code>)。</p>
|
||||
<whiteboard type="mermaid">flowchart LR
|
||||
C[客户端] --> N[new-api :31000 参数覆盖链]
|
||||
N --> S[responses-shim :8080 cw ds41-resp-shim]
|
||||
S --> R[sgl-router ds41-3p1d:50002]
|
||||
R --> P[P x3 prefill cp4+dspark]
|
||||
R --> D[D decode dp2]
|
||||
</whiteboard>
|
||||
<h2>6.1 new-api 参数覆盖链(渠道高级设置 → 参数覆盖)</h2>
|
||||
<p>目标:对上暴露 DeepSeek 官方 API 语义,对下翻译成 sglang 认识的参数。覆盖 ops 按顺序执行,要点:</p>
|
||||
<ul>
|
||||
<li><b>thinking 映射</b>:Claude 风格 <code>thinking.type=enabled/disabled</code> → <code>chat_template_kwargs.thinking=true/false</code>(sglang 用 chat template kwargs 控制思维链)。</li>
|
||||
<li><b>reasoning_effort 归一</b>:官方取值 none/low/high/max,别名 minimal→low、medium/xhigh→high;落到 <code>chat_template_kwargs</code> 的 thinking 开关与 reasoning_effort。responses 请求另有一条豁免:<code>reasoning.effort</code> 由 shim 翻译,不在覆盖链重复处理。</li>
|
||||
<li><b>temperature 条件钉 1</b>:思考模式下官方语义=不生效,强制 set 1;<b>关闭思考(none / thinking disabled)时跟随用户设置</b>——条件带 invert + pass_missing_key,缺省不补。</li>
|
||||
<li><b>top_p 钳制</b>:思考模式有效范围 0.95–1.0,低于 0.95 按 0.95;非思考模式恒 1.0(传入忽略)。</li>
|
||||
<li><b>max_tokens 缺省 65536</b>:<code>keep_origin=true</code>,用户传了不覆盖;responses 侧的 <code>max_output_tokens</code> 同理 keep_origin。</li>
|
||||
</ul>
|
||||
<p>验收:llm-api-preflight 13/13 通过(采样参数生效性、思考开关、默认值、/v1/messages 兼容路径),基线存档 <code>D:\B300\preflight\newapi-thinking-baseline-20260930.md</code>。</p>
|
||||
<h2>6.2 responses-shim(/v1/responses 协议适配)</h2>
|
||||
<p>sglang 不支持 <a href="https://api-docs.deepseek.com/zh-cn/api/create-response">官方 /v1/responses 协议</a>,new-api 只能透传不能翻译,所以在中间加了一层 shim:</p>
|
||||
<ul>
|
||||
<li><b>实现</b>:Go 单文件、纯 stdlib 零依赖;源码 <code>D:\B300\responses-shim\main.go</code>,b300 构建目录 <code>/data/ymk/build/responses-shim/</code>。</li>
|
||||
<li><b>镜像</b>:<code>ymkymx/responses-shim:v0.2.3</code>(scratch 基础,~5.6MB,本机交叉编译 GOOS=linux GOARCH=amd64 CGO_ENABLED=0 → scp → b300 docker build → <code>ctr -n k8s.io images import</code> 侧载)。</li>
|
||||
<li><b>部署</b>:孔明 cw <code>ds41-resp-shim</code>(manifest <code>D:\B300\build-kmctl\responses-shim-cw.yaml</code>),ClusterIP <code>ds41-resp-shim:8080</code>,钉 b300-01,资源 1 核 / 512Mi;env:<code>UPSTREAM_BASE=http://ds41-3p1d:50002</code>、<code>STOP_SCOPE=content</code>。</li>
|
||||
<li><b>功能</b>:/v1/responses 双向翻译——入参 input / instructions / reasoning.effort / max_output_tokens / text.format / tools / tool_choice / top_logprobs / user / stop;流式输出完整 response.* 事件序列;usage 全字段透传(含 cached_tokens、reasoning_tokens——sglang 把 reasoning_tokens 放在顶层,v0.2.2 起收编进 output_tokens_details)。/v1/chat/completions 字节级透传不解析。</li>
|
||||
<li><b>版本履历</b>:v0.2.0 内容侧 stop 截断(STOP_SCOPE=content);v0.2.1 usage details 透传;v0.2.2 reasoning_tokens 收编;v0.2.3 responses 路径 stop 参数修复。</li>
|
||||
</ul>
|
||||
<h2>6.3 已知坑与边界</h2>
|
||||
<callout emoji="⚠️" background-color="light-yellow" border-color="yellow">
|
||||
<ul>
|
||||
<li><b>stop_sequence=null 救不了</b>:chat 路径 new-api 的 Claude 转换器不读上游 matched_stop 字段,响应里 stop_sequence 恒 null——shim 无法补齐,属网关转换器限制。</li>
|
||||
<li><b>STOP_SCOPE=content 语义偏离</b>:stop 只截正文不截思维链(官方行为是连 reasoning 一起截)。如需严格对齐官方,改 STOP_SCOPE=all 重启。</li>
|
||||
<li><b>previous_response_id 不支持</b>:responses 链式会话 ID 不生效(多轮需客户端自带历史)。</li>
|
||||
<li><b>shim 单副本承接全量流量</b>:chat 透传也过 shim,1 核 / 512Mi 实测 1000 RPM 余量充足;若日后流量翻倍再扩副本(无状态可直接加)。</li>
|
||||
<li><b>Windows scp 丢执行位</b>:二进制传到 b300 后必须 <code>chmod +x</code> 再 docker build,否则容器启动报 permission denied。</li>
|
||||
<li><b>上游 PR #5209 若合入</b>(sglang 原生 responses 支持)shim 可退役,届时回滚 new-api 渠道上游直指 ds41-3p1d:50002 即可。</li>
|
||||
</ul>
|
||||
</callout>
|
||||
<h2>6.4 运维速查</h2>
|
||||
<pre lang="bash" caption="ds41-resp-shim 日常操作"><code># 查状态 / 日志 / 启停
|
||||
wsl kmctl logs cw ds41-resp-shim
|
||||
wsl kmctl {events,stop,start} cw ds41-resp-shim
|
||||
|
||||
# 改代码重建(本机绿色 Go:D:\B300\tools\go\bin\go.exe)
|
||||
cd D:\B300\responses-shim
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o responses-shim .
|
||||
scp responses-shim 24.125.93.200.207:/data/ymk/build/responses-shim/
|
||||
# b300 上:chmod +x → docker build -t ymkymx/responses-shim:vX.Y.Z .
|
||||
# → docker save | ctr -n k8s.io images import -
|
||||
# 改 manifest tag 后:wsl kmctl update cw ds41-resp-shim</code></pre>
|
||||
<hr/>
|
||||
@@ -0,0 +1,11 @@
|
||||
<pre lang="text" caption="网关链路拓扑"><code>客户端(/v1/chat/completions · /v1/messages · /v1/responses)
|
||||
│
|
||||
▼
|
||||
new-api 网关 :31000 ── 参数覆盖链(thinking / reasoning_effort / temperature / top_p / max_tokens)
|
||||
│ 渠道上游
|
||||
▼
|
||||
responses-shim :8080(孔明 cw ds41-resp-shim;responses ⇄ chat 翻译,chat 字节透传)
|
||||
│ UPSTREAM_BASE=http://ds41-3p1d:50002
|
||||
▼
|
||||
sgl-router ds41-3p1d:50002 ──┬── P ×3(prefill,cp4 + dspark)
|
||||
└── D(decode,dp2)</code></pre>
|
||||
Reference in New Issue
Block a user