[Cookbook] Kimi-K3: keep DCP under HiCache L1+L2 with DSPARK (#39190)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
55e5e21c88
commit
0d08668821
@@ -283,7 +283,8 @@ Pending update...
|
||||
|
||||
K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads.
|
||||
|
||||
- On the DCP recipes (Blackwell Balanced / High-Throughput, in both the `Unified` and `Decode` roles), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 with Spec Decode off keeps DCP. Only DCP goes: the MLA KV reverts to TP-replicated, but the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP while B200 Unified keeps its `--pp-size 2` / `--ep-size`.
|
||||
- On the DCP recipes (Blackwell Balanced / High-Throughput, `Unified` and `Decode` roles), **L1+L2 keeps DCP**, with Spec Decode off or on DSPARK. On B200 the DSPARK pipeline collapse scales it too (PP2 × DCPEP8 → DCPEP16).
|
||||
- **L3 drops the DCP flags** on those recipes (storage keys are not dcp_rank-aware yet; the command hints call it out, and per-request KV capacity shrinks). Only DCP goes: the MLA KV reverts to TP-replicated while the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP and B200 Unified keeps its `--pp-size 2` / `--ep-size`.
|
||||
- Low-Latency and the Hopper recipes take all tiers unchanged.
|
||||
|
||||
<a id="pd-disaggregation" />
|
||||
|
||||
@@ -81,7 +81,7 @@ export const config = {
|
||||
const [hw, pdMode] = k.split("|");
|
||||
return {
|
||||
when: { hw: [hw], pdMode: [pdMode], strategy: [...strategies],
|
||||
hicache: ["l2"], spec: ["none"] },
|
||||
hicache: ["l2"], spec: ["none", "dspark"] },
|
||||
reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2.",
|
||||
};
|
||||
});
|
||||
@@ -115,10 +115,8 @@ export const config = {
|
||||
const cell = config.cellFor(s);
|
||||
const pp = config.sizeOf(cell, "--pp-size");
|
||||
const out = [`--tp-size ${config.sizeOf(cell, "--tp-size") * pp}`];
|
||||
// HiCache under DCP rejects speculative decoding, so when a host tier is on
|
||||
// the DCP half is dropped instead of scaled (the HiCache options' own
|
||||
// stripPrefixes reach cell flags only, never these overlay flags).
|
||||
if (config.flagOf(cell, "--dcp-size") && s.hicache !== "l2" && s.hicache !== "l3") {
|
||||
// L3 rejects DCP at startup, so under L3 the DCP half is dropped, not scaled.
|
||||
if (config.flagOf(cell, "--dcp-size") && s.hicache !== "l3") {
|
||||
out.push(`--dcp-size ${config.sizeOf(cell, "--dcp-size") * pp}`);
|
||||
}
|
||||
if (config.flagOf(cell, "--ep-size")) {
|
||||
@@ -355,26 +353,17 @@ export const config = {
|
||||
flags: [
|
||||
"--enable-hierarchical-cache",
|
||||
],
|
||||
// L1/L2 host tiering IS supported under DCP, so DCP stays — except with
|
||||
// speculative decoding, which HiCache under DCP rejects at startup (the
|
||||
// draft host pool has no DCP index translation). There the DCP operating
|
||||
// point is dropped instead of blocking the option, same as L3 does. The
|
||||
// ratio calculator reads --dcp-size off this command, so dropping it also
|
||||
// re-solves --mamba-full-memory-ratio for the plain-TP shape.
|
||||
//
|
||||
// Which cells carry DCP is read off the cells themselves, so adding or
|
||||
// reshaping a DCP recipe needs no edit here.
|
||||
stripPrefixes: (s) =>
|
||||
s.spec !== "none" && config.hasDcp(s)
|
||||
!["none", "dspark"].includes(s.spec) && config.hasDcp(s)
|
||||
? ["--dcp-size", "--dcp-comm-backend"]
|
||||
: [],
|
||||
hints: (s) =>
|
||||
s.spec !== "none" && config.hasDcp(s)
|
||||
!["none", "dspark"].includes(s.spec) && config.hasDcp(s)
|
||||
? [
|
||||
"HiCache under DCP rejects speculative decoding, so this recipe drops DCP",
|
||||
"and serves the MLA KV TP-replicated (any PP/EP in the cell stays).",
|
||||
"Per-request context is far shorter than the DCP",
|
||||
"version — DCP is what buys KV capacity. Run the cell NOSPEC to keep DCP.",
|
||||
"HiCache under DCP only accepts DSPARK speculative decoding, so this",
|
||||
"recipe drops DCP and serves the MLA KV TP-replicated (any PP/EP in the",
|
||||
"cell stays). Per-request context is far shorter than the DCP version —",
|
||||
"DCP is what buys KV capacity. Run the cell NOSPEC or DSPARK to keep DCP.",
|
||||
]
|
||||
: [],
|
||||
},
|
||||
@@ -977,10 +966,6 @@ export const config = {
|
||||
when: { hicache: ["l3"] },
|
||||
reason: "L3 storage keys are not dcp_rank-aware, so DCP and L3 cannot run together. Use Peak Throughput, or switch HiCache to L1+L2.",
|
||||
},
|
||||
{
|
||||
when: { hicache: ["l2"], spec: ["dspark"] },
|
||||
reason: "HiCache under DCP rejects speculative decoding, so this preset cannot add DCP here. Use Peak Throughput, or run the cell NOSPEC.",
|
||||
},
|
||||
],
|
||||
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480"],
|
||||
flags: (v, b) => {
|
||||
|
||||
Reference in New Issue
Block a user