73 KiB
73 KiB
In [ ]:
%pip install --quiet --upgrade openai arxiv pypdf requestsIn [ ]:
import os
# Paste your token here if you haven't run `huggingface-cli login`
# os.environ["HF_TOKEN"] = "hf_..."In [ ]:
import requests, time
SGLANG_BASE_URL = "http://localhost:30000"
for attempt in range(30):
try:
if requests.get(f"{SGLANG_BASE_URL}/health", timeout=5).status_code == 200:
print("Server ready.")
break
except requests.exceptions.ConnectionError:
pass
print(f"Waiting... ({attempt + 1}/30)")
time.sleep(10)Waiting... (1/30) Server ready.
In [ ]:
from openai import OpenAI
client = OpenAI(base_url=f"{SGLANG_BASE_URL}/v1", api_key="not-needed")
MODEL = "deepseek-ai/DeepSeek-V4-Flash"
print([m.id for m in client.models.list().data])['deepseek-ai/DeepSeek-V4-Flash']
In [ ]:
import arxiv
import time
SEARCH_QUERY = (
'ti:"LLM inference" OR ti:"efficient inference" OR ti:"speculative decoding" '
'OR ti:"continuous batching" OR ti:"KV cache" OR ti:"paged attention"'
)
MAX_PAPERS = 20
MAX_ABSTRACT_CHARS = 2000
search = arxiv.Search(
query=SEARCH_QUERY,
max_results=MAX_PAPERS,
sort_by=arxiv.SortCriterion.SubmittedDate,
sort_order=arxiv.SortOrder.Descending,
)
papers = []
for result in arxiv.Client(delay_seconds=3, num_retries=5).results(search):
papers.append(
{
"id": result.entry_id.split("/")[-1],
"title": result.title,
"authors": ", ".join(a.name for a in result.authors[:5])
+ (" et al." if len(result.authors) > 5 else ""),
"published": result.published.strftime("%Y-%m-%d"),
"abstract": result.summary.replace("\n", " ")[:MAX_ABSTRACT_CHARS],
"categories": ", ".join(result.categories),
"pdf_url": result.pdf_url,
}
)
time.sleep(1) # extra courtesy delay between results
for i, p in enumerate(papers, 1):
print(f"{i:>2}. [{p['published']}] {p['title']}")
print(f" {p['authors']}") 1. [2026-06-22] Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse
Bole Ma, Jan Eitzinger, Harald Koestler, Gerhard Wellein
2. [2026-06-22] Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference
Yuhang Gan, Yiwei Yang, Yuyi Li, Xiangyu Gao, Yichen Wang et al.
3. [2026-06-22] EnerInfer: Energy-Aware On-Device LLM Inference
Bohua Zou, Nian Liu, Binqi Sun, Matteo Mascherin, Debayan Roy et al.
4. [2026-06-22] MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inference
Zichuan Wang, Huizheng Wang, Yuheng Xiao, Haonan Zuo, Taiquan Wei et al.
5. [2026-06-22] RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Haifeng Wu, Srinivasan Manoharan, Fangbo Tu, Junhua Zhao, Jian Wan
6. [2026-06-21] Evidence-Bound Gateway-Path Provenance for Third-Party LLM Inference
Fei Wang, Zebai Tian
7. [2026-06-20] The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference
Naihao Deng, Alissa Shen, Yiming Feng, Joan Nwatu, Jae-Won Chung et al.
8. [2026-06-20] Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAG
He Sun, Shinan Liu, Siyuan Ma, Junhao Li, Mingjun Xiao et al.
9. [2026-06-19] HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica et al.
10. [2026-06-19] Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Yang Shen, Meghana Madhyastha, Robert Underwood, Bogdan Nicolae, Randal Burns
11. [2026-06-19] Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL
Zhenting Zhu, Lucas Thai, Shan Yu, Yicheng Liu, Yifan Qiao et al.
12. [2026-06-18] SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation
Shilong Xiang, Zirui Zhang, Lijun Yu, Chengzhi Mao
13. [2026-06-18] UltraQuant: 4-bit KV Caching for Context-Heavy Agents
Inesh Chakrabarti, David Limpus, Aditi Ghai Rana, Bowen Bao, Spandan Tiwari et al.
14. [2026-06-18] Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference
Huang Peng, Jiuyang Tang, Weixin Zeng, Hao Xu, Xiang Zhao
15. [2026-06-18] SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
Ruiyang Ma, Teng Ma, Junru Li, Hantian Zha, Xuchun Shang et al.
16. [2026-06-17] EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim, Donghoon Kim et al.
17. [2026-06-17] From Tokens to Energy Flexibility: Quantization-Enabled Demand Response for Data Centers with LLM Inference Workloads
Bojun Du, Xiaoyi Fan, Ershun Du, Long Chen, Jianpei Han et al.
18. [2026-06-16] Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Yueying Li, Yuanfan Chen, Jiayang Chen, Esha Choukse, Haoran Qiu et al.
19. [2026-06-16] JetFlow: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
Lanxiang Hu, Zhaoxiang Feng, Yulun Wu, Haoran Yuan, Yujie Zhao et al.
20. [2026-06-16] Latency Prediction for LLM Inference on NPU Systems
Juhyun Park, Seungwoo Jeong, Jingyu Lee, Kyungyong Lee
In [ ]:
import pathlib
import pypdf
N_FULL_PAPERS = 5
MAX_PDF_CHARS = 30_000
PDF_CACHE_DIR = pathlib.Path("./arxiv_pdfs")
PDF_CACHE_DIR.mkdir(exist_ok=True)
def download_and_extract(paper: dict) -> str:
cache_path = PDF_CACHE_DIR / f"{paper['id']}.pdf"
if not cache_path.exists():
resp = requests.get(paper["pdf_url"], timeout=60)
resp.raise_for_status()
cache_path.write_bytes(resp.content)
text = ""
for page in pypdf.PdfReader(str(cache_path)).pages:
text += page.extract_text() or ""
if len(text) >= MAX_PDF_CHARS:
break
return text[:MAX_PDF_CHARS]
for i, paper in enumerate(papers[:N_FULL_PAPERS]):
try:
paper["full_text"] = download_and_extract(paper)
print(
f"[{i + 1}/{N_FULL_PAPERS}] {paper['title'][:70]} — {len(paper['full_text']):,} chars"
)
except Exception as e:
paper["full_text"] = None
print(f"[{i + 1}/{N_FULL_PAPERS}] {paper['title'][:70]} — failed: {e}")[1/5] Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Fr — 30,000 chars [2/5] Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tole — 30,000 chars [3/5] EnerInfer: Energy-Aware On-Device LLM Inference — 30,000 chars [4/5] MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelinin — 30,000 chars [5/5] RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LL — 30,000 chars
In [ ]:
def format_paper(paper: dict, index: int) -> str:
parts = [
f"## Paper {index}: {paper['title']}",
f"**Authors:** {paper['authors']}",
f"**Published:** {paper['published']} | **arXiv ID:** {paper['id']}",
f"**Categories:** {paper['categories']}",
"",
"**Abstract:**",
paper["abstract"],
]
if paper.get("full_text"):
parts += ["", "**Full Text (excerpt):**", paper["full_text"]]
return "\n".join(parts)
bundle_sections = [
f"# Research Bundle: Efficient LLM Inference\n**Papers:** {len(papers)}"
]
for i, paper in enumerate(papers, 1):
bundle_sections.append(format_paper(paper, i))
research_bundle = "\n\n---\n\n".join(bundle_sections)
print(
f"{len(research_bundle):,} chars | ~{len(research_bundle) // 4:,} tokens | "
f"{sum(1 for p in papers if p.get('full_text'))} papers with full text"
)185,132 chars | ~46,283 tokens | 5 papers with full text
In [ ]:
RESEARCH_MAP_PROMPT = """\
Here is a bundle of research papers on efficient LLM inference:
{bundle}
---
Produce a Research Map with these sections:
1. **Field Overview** (3–5 sentences): The core problem and why it matters now.
2. **Research Themes**: 4–6 distinct themes across these papers. For each:
- Name and one-sentence description
- Which papers belong to it (by number and title)
- Key insight that defines the theme
3. **Chronological Progression**: How has the field evolved? What ideas were superseded?
4. **Open Problems**: 3–5 important unsolved problems.
5. **Practitioner Takeaways**: Which 3 ideas are most ready to deploy today?
"""
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "user", "content": RESEARCH_MAP_PROMPT.format(bundle=research_bundle)}
],
max_tokens=3000,
temperature=0.3,
)
print(response.choices[0].message.content)Here is the Research Map based on the provided bundle of 20 papers on efficient LLM inference. --- ### 1. Field Overview The core problem is that **Large Language Model (LLM) inference is becoming prohibitively expensive and complex** as models scale, context windows lengthen, and deployment moves toward long-running, multi-turn agents. The field is shifting from optimizing for raw throughput to a **holistic systems-level approach** that must jointly manage memory (KV cache), latency (tail & prefill), energy (power & thermal), and reliability (fault tolerance & security) across heterogeneous hardware (GPUs, NPUs, WSCs) and deployment models (cloud, on-device, third-party APIs). This bundle of 2026 papers reflects a mature field where the "low-hanging fruit" of basic batching and parallelism has been exhausted, and the frontier is now defined by **specialized, co-designed, and often training-free** techniques that exploit the unique structural properties of LLM inference. ### 2. Research Themes **Theme 1: KV Cache Management & Memory Orchestration** *Focuses on the KV cache as the primary bottleneck, moving beyond simple eviction to proactive, structured, and hardware-aware management.* - **Papers:** 1 (Kamera), 4 (MOCAP), 10 (Recency/Frequency Adaptive), 13 (UltraQuant), 15 (SAC) - **Key Insight:** The KV cache is not a monolithic buffer but a structured, position-dependent asset. The most impactful optimizations come from **separating the cache's content (what) from its position (where)** (Kamera), **redistributing it across pipeline stages** to balance memory pressure (MOCAP), or **selectively fetching only the active "sparse" entries** via low-latency interconnects like CXL (SAC). This represents a shift from reactive eviction (LRU) to proactive, system-aware orchestration. **Theme 2: Speculative Decoding & Draft-Verify Pipelines** *Exploits the asymmetry in cost/speed between a "draft" model and a "verify" model to accelerate generation or reduce API costs.* - **Papers:** 5 (RLM-Cascade), 12 (SSD), 16 (EfficientRollout), 19 (JetFlow) - **Key Insight:** The core idea has been generalized from **token-level** to **response-level** (RLM-Cascade) and **spatial-level** (SSD). The critical challenge is no longer just the draft-verify mechanism itself, but **managing the "draft budget"** : how many tokens to draft, when to stop, and how to handle the evolving distribution of the target model (EfficientRollout). JetFlow breaks the scaling ceiling by using a **causal parallel draft head** that avoids the inefficiency of autoregressive drafting. **Theme 3: Energy & Resource-Aware Inference** *Treats energy, power, and thermal constraints as first-class optimization objectives, not just secondary concerns.* - **Papers:** 3 (EnerInfer), 7 (Language-Energy Divide), 17 (From Tokens to Energy Flexibility) - **Key Insight:** The field is moving from "fastest is best" to **"efficient-enough is best"** . There is significant, exploitable "configuration slack" (EnerInfer) where modestly lowering frequencies preserves QoE but drastically improves energy efficiency. This is a **systemic inequity** (Paper 7) that is not just a hardware problem but is also **model-dependent** (e.g., low-resource languages generate more tokens, compounding the cost). **Theme 4: Fault Tolerance & Security for Long-Running Agents** *Addresses the unique reliability and security challenges of persistent, stateful LLM agents.* - **Papers:** 2 (Concordia), 6 (Evidence-Bound Gateway), 8 (Agent-Assisted Side-Channel), 11 (HEAL) - **Key Insight:** The "always-on" nature of agents creates a **new threat model** where state (KV cache, scheduler) is valuable and must be protected. Concordia argues for **GPU-resident, persistent checkpointing** as the substrate for fault tolerance, moving the recovery contract below the framework. HEAL addresses a different, but related, reliability problem: **numerical instability** in mission-critical tasks, where FP16 precision can cause catastrophic output divergence. **Theme 5: Systems Co-Design & Hardware-Aware Scheduling** *Optimizes the entire serving stack by co-designing algorithms with specific hardware (WSCs, NPUs, CXL) and scheduling policies.* - **Papers:** 4 (MOCAP), 15 (SAC), 18 (Beyond Prediction), 20 (LENS) - **Key Insight:** The most effective optimizations are **not portable** across hardware. MOCAP is explicitly designed for the unique communication topology of Wafer-Scale Chips. SAC leverages CXL's cache-line granularity. LENS provides a prediction methodology for **opaque NPU systems** where microarchitecture is undisclosed. This theme highlights the move away from "one-size-fits-all" GPU solutions toward **specialized, hardware-aware** frameworks. **Theme 6: Cost & Latency Optimization at the API/Service Layer** *Focuses on the economic and operational realities of serving LLMs through third-party APIs and managing request-level variability.* - **Papers:** 5 (RLM-Cascade), 18 (Beyond Prediction), 20 (LENS) - **Key Insight:** The "last mile" of inference is the **API call**. RLM-Cascade shows that response-level speculative decoding can be **cheaper, faster, and more accurate** than a direct call to a frontier model, because the "SKIPPED" path (cheap model only) dominates the workload. This is a counter-intuitive result that challenges the assumption that "better" models are always better. The key is **routing precision**, not just model quality. ### 3. Chronological Progression - **Superseded Idea: "Faster is always better."** The field has moved from a single-minded focus on maximizing throughput (tokens/s) to a multi-objective optimization that includes energy efficiency (EnerInfer), tail latency (Beyond Prediction), and cost (RLM-Cascade). The idea that "maximum frequency" is the default is now recognized as a source of significant inefficiency. - **Superseded Idea: "One-size-fits-all GPU optimization."** The success of MOCAP (WSC-specific), SAC (CXL-specific), and LENS (NPU-specific) shows that the most impactful optimizations are now **hardware-specific**. The era of generic "GPU-optimized" serving is giving way to specialized, co-designed systems. - **Superseded Idea: "Token-level speculative decoding is the only way."** The field has generalized the draft-verify concept to **response-level** (RLM-Cascade) and **spatial-level** (SSD). This opens up new deployment models (e.g., API proxies) and new application domains (e.g., image generation) that were previously inaccessible to the technique. - **Superseded Idea: "KV cache is a simple, linear structure."** The work on Kamera (position-invariant), MOCAP (memory-balanced), and SAC (sparse attention) has fundamentally reframed the KV cache as a **complex, structured, and dynamic asset** that can be decomposed, relocated, and selectively accessed, rather than just a contiguous block to be evicted. ### 4. Open Problems 1. **The "Draft Budget" Problem for Speculative Decoding:** How many tokens should a draft model generate? The optimal budget is highly dynamic, depending on the model, task, and hardware. JetFlow's causal parallel head is a step, but a **general, online, and system-aware** solution for managing the draft budget remains an open problem. 2. **Cross-Architecture Portability for Hardware-Aware Systems:** MOCAP, SAC, and LENS are all highly specialized. How can the insights from these systems be **generalized** into a portable framework that works across GPUs, NPUs, WSCs, and CXL-based memory pools without requiring a complete re-implementation for each target? 3. **The "Energy Divide" as a First-Class Metric:** Paper 7 shows a 179x energy cost disparity between languages. The field lacks a **standardized, reproducible methodology** for measuring and reporting this cost. Until energy is a standard axis in model cards and leaderboards, this systemic inequity will remain unaddressed. 4. **Reconciling Memory Deduplication with Security:** SpliceLeak (Paper 8) demonstrates a fundamental conflict: the very mechanisms (chunk alignment, boundary fusion) that enable efficient KV cache sharing also create a **deterministic timing side-channel**. A principled, provably secure solution that preserves the benefits of global cache sharing is an open problem. 5. **Fault Tolerance for "Stateless" Agents:** The Concordia model assumes a persistent, stateful agent. However, many current deployments are "stateless" (each request is a fresh context). How can the principles of GPU-resident checkpointing be applied to this **more common, but equally failure-prone**, deployment model without requiring a fundamental redesign of the serving stack? ### 5. Practitioner Takeaways 1. **Deploy a "Response-Level Speculative Decoding" API Proxy (RLM-Cascade).** This is the most immediately deployable idea. It requires no model internals access, works across any HTTP API, and can be **cheaper, faster, and more accurate** than a direct call to a frontier model. The key is a simple, rule-based router that identifies "simple" turns. This is a direct path to reducing API costs by 45-67% in production. 2. **Adopt a "Configuration Slack" Energy Management Policy (EnerInfer).** For on-device or edge deployments, the default "max frequency" setting is wasteful. Implement a simple, offline prediction pipeline that maps model structure (e.g., number of layers, heads) to the most energy-efficient NPU/DDR frequency that still meets a user-defined QoE threshold (e.g., 10 tokens/s). This can yield 9-65% energy efficiency improvements with no QoE violation. 3. **Use a "Position-Invariant" KV Cache for Long-Horizon Agents (Kamera).** If your agent repeatedly re-examines the same video frames or documents, the standard prefix cache is a bottleneck. Store the KV cache as a **position-free "canonical" chunk** plus a **small, low-rank "conditioning patch"** . This allows for O(1) reordering, sliding-window survival, and recall, turning expensive re-prefills into cheap cache edits. The patch is training-free and can be deployed in a production SGLang kernel.
In [ ]:
DEEP_DIVE_QUESTIONS = [
(
"Methodology comparison",
"Compare the memory management strategies used across the papers. "
"Which papers share the same fundamental approach, and which ones represent genuine departures? "
"Cite specific paper numbers.",
),
(
"Benchmarking gaps",
"Which evaluation benchmarks and metrics appear most often across these papers? "
"What important real-world performance dimensions are missing from the evaluations?",
),
(
"Implementation guidance",
"A small team wants to build a production LLM inference system from scratch. "
"Synthesize the most actionable engineering lessons from this bundle into a prioritized roadmap.",
),
]
for label, question in DEEP_DIVE_QUESTIONS:
print(f"\n[{label}]")
print("─" * 60)
resp = client.chat.completions.create(
model=MODEL,
messages=[
{
"role": "system",
"content": "Reference specific papers by number and title. Be analytical and concrete.",
},
{
"role": "user",
"content": f"Research bundle:\n\n{research_bundle}\n\n---\n\nQuestion: {question}",
},
],
max_tokens=1200,
temperature=0.2,
)
print(resp.choices[0].message.content)
print(
f"\n[prompt: {resp.usage.prompt_tokens:,} completion: {resp.usage.completion_tokens:,} tokens]"
)
[Methodology comparison]
────────────────────────────────────────────────────────────
# Comparative Analysis of Memory Management Strategies in LLM Inference
## Overview
The 20 papers in this bundle address memory management at different levels of the LLM inference stack, from low-level KV cache manipulation to system-level memory orchestration across devices. I identify three distinct clusters of shared approach and two genuine departures.
## Cluster 1: KV Cache as Reusable Content (Papers 1, 8, 9, 10, 13, 15)
**Shared fundamental approach:** These papers treat the KV cache as a *storable, reusable asset* that can be retained, compressed, or selectively retrieved across inference steps. They all operate on the principle that the KV cache contains redundant information that can be exploited to reduce recomputation.
**Paper 1 (Kamera)** is the most sophisticated member of this cluster. It introduces **position-invariant KV caching** where chunks are stored with a canonical content channel and a separate positional RoPE rotation. The key innovation is the **low-rank conditioning patch** (Equation 1) that restores cross-chunk binding lost during naive reuse. This is a genuine extension of the reuse paradigm, not a departure—it shares the same "store and reuse" philosophy as papers 8, 9, 10, and 13, but adds a *correction term* for the specific failure mode of multi-hop reasoning.
**Paper 8 (SpliceLeak)** takes the opposite stance: it treats KV cache reuse as a *security vulnerability*. The "Step-Wave" timing signature from chunk-aware memory scheduling is exploited for side-channel attacks. This paper shares the same *observation* that KV cache is structured and reusable, but frames it as an attack surface rather than an optimization. It is **not a departure** in approach—it uses the same chunked storage model as Kamera—but it is a departure in *objective*.
**Paper 9 (HERALD)** and **Paper 15 (SAC)** form a sub-cluster within this group. Both address **sparse attention** models where only a fraction of KV entries are active. HERALD uses block-level consistency across denoising steps to select top-k entries once and reuse them throughout a block. SAC uses CXL's cache-line granularity to fetch only the required top-k entries on demand. Both papers share the **selective retrieval** approach: identify the relevant subset of KV cache and fetch only that. This is a genuine *departure* from the full-cache-reuse paradigm of Papers 1, 10, and 13, because they *do not* attempt to store or reconstruct the full cache—they accept that most entries are irrelevant and optimize for the sparse subset.
**Paper 10 (Recency/Frequency Adaptive KV Caching)** and **Paper 13 (UltraQuant)** are the most conventional members of this cluster. Paper 10 applies **LRU-like adaptive caching** to KV blocks, distinguishing between recently and frequently occurring blocks. This is a *cache eviction policy* approach, not a content-aware reuse strategy. Paper 13 applies **4-bit quantization** to the KV cache, using TurboQuant-style rotation and codebook quantization. This is a *compression* approach that reduces memory footprint without changing the fundamental storage model. Both are **incremental improvements** to the existing KV cache paradigm, not departures.
## Cluster 2: Pipeline-Stage Memory Orchestration (Papers 2, 4)
**Shared fundamental approach:** These papers manage memory *across pipeline stages* in a distributed system, treating KV cache as a *resource to be balanced* rather than reused.
**Paper 2 (Concordia)** introduces **persistent-kernel checkpointing** for fault tolerance. The key insight is that KV cache and adapter pages change *sparsely* and can be detected on-GPU at HBM bandwidth. The **persistent executor** (Section 3.1) is a device-resident control loop that scans dirty pages and appends to a recovery log. This is a *systems-level* approach to memory management, not a *content-level* one—it cares about *where* the data is and *whether* it changed, not *what* the data means.
**Paper 4 (MOCAP)** addresses **memory imbalance** in chunked pipelining on wafer-scale chips. The **Memory-Balanced KV Reallocation (MBKR)** (Section 4.1) uses a *fixed cross-half pairing* strategy where stage i is paired with stage i+N/2. This is a *topological* approach to memory management—it cares about *which stage* holds which KV cache, not *which tokens* are in the cache. This is a **genuine departure** from the content-reuse paradigm of Cluster 1, because it treats KV cache as a *distributed resource* to be balanced across physical devices, not as a *semantic structure* to be exploited for reuse.
## Cluster 3: Energy-Aware Memory Management (Papers 3, 7, 17)
**Shared fundamental approach:** These papers treat memory *as a component of energy consumption*, optimizing for energy efficiency rather than throughput or latency.
**Paper 3 (EnerInfer)** is the most explicit member. It proposes **NPU/DDR frequency scaling** as a control knob for energy efficiency. The key insight is that *modestly lowering frequencies* preserves QoE while improving energy efficiency (Figure 4 shows 62% improvement). This is a *hardware-configuration* approach to memory management—it controls *how fast* the memory operates, not *what* is stored in it.
**Paper 7 (Language-Energy Divide)** measures **energy per output token** across languages, finding up to 8.3× variation. This is a
[prompt: 45,960 completion: 1,200 tokens]
[Benchmarking gaps]
────────────────────────────────────────────────────────────
Of the 20 papers in this research bundle, I will analyze the evaluation benchmarks and metrics that appear most frequently, then identify important real-world performance dimensions that are missing from these evaluations.
## Most Frequently Appearing Benchmarks and Metrics
### Benchmarks
1. **MM-NIAH (Multi-Modal Needle-in-a-Haystack)** - Appears in Paper 1 (Kamera) as a cross-chunk binding benchmark for evaluating KV cache reuse quality. Tests multi-hop reasoning across long contexts.
2. **Video-MME** - Appears in Paper 1 (Kamera) for evaluating video understanding under KV cache reuse.
3. **EgoSchema** - Appears in Paper 1 (Kamera) for long-form video understanding.
4. **MileBench** - Appears in Paper 1 (Kamera) for temporal reasoning across cached video clips.
5. **DPG-Bench / GenEval** - Appears in Paper 12 (SSD) for evaluating autoregressive image generation quality.
6. **MATH-500** - Appears in Paper 19 (JetFlow) for mathematical reasoning under speculative decoding.
7. **Claude Code / Agentic coding workloads** - Appears in Paper 5 (RLM-Cascade) as a real-world production workload for evaluating cost savings.
### Metrics
1. **Throughput (tokens/s, req/s)** - The most common metric across nearly all papers. Papers 1, 3, 4, 5, 9, 13, 15, 16, 19 all report throughput improvements.
2. **Latency (TTFT, TBT, P50/P95/P99)** - Appears in Papers 1, 3, 4, 5, 8, 10, 13, 15, 18, 20. Time-to-first-token (TTFT) is the most frequently cited latency metric.
3. **Energy efficiency (tokens/J)** - Appears in Papers 3 (EnerInfer), 7 (Language-Energy Divide), and 17 (Quantization-Enabled DR).
4. **KV cache hit rate** - Appears in Papers 1 (Kamera), 10 (Recency/Frequency Adaptive), and 13 (UltraQuant).
5. **Accuracy / Quality (pass rate, KL divergence, task accuracy)** - Appears in Papers 1, 5, 7, 9, 11, 14, 19.
6. **Cost savings (USD, % reduction)** - Appears in Papers 5 (RLM-Cascade) and 17 (Quantization-Enabled DR).
7. **End-to-end latency reduction** - Appears in Papers 4 (MOCAP), 16 (EfficientRollout), 19 (JetFlow).
8. **Prediction error (%)** - Appears in Paper 20 (LENS) for latency prediction.
## Patterns in Evaluation
The papers cluster around several evaluation themes:
- **KV cache efficiency** (Papers 1, 9, 10, 13, 15) - measured by hit rate, memory savings, throughput at reduced cache
- **Speculative decoding speedup** (Papers 5, 12, 16, 19) - measured by end-to-end latency reduction, draft acceptance rate
- **Energy/cost** (Papers 3, 7, 17) - measured by energy per token, cost per request
- **System throughput** (Papers 4, 15, 18) - measured by requests/sec, pipeline efficiency
- **Quality preservation** (Papers 1, 5, 9, 11, 14) - measured by task accuracy, KL divergence from baseline
## Missing Real-World Performance Dimensions
### 1. **Multi-tenant interference and isolation**
No paper evaluates how techniques degrade under concurrent, heterogeneous workloads from different tenants. Production systems serve many users simultaneously with varying context lengths, model versions, and latency requirements. Papers 1, 10, and 18 touch on this but do not systematically measure interference.
### 2. **Cold-start behavior**
Papers focus on steady-state performance after caches are warm. Real deployments face cold-start penalties when models are loaded, caches are empty, or new users arrive. Paper 13 (UltraQuant) mentions "cache-pressured late rounds" but does not quantify cold-start overhead.
### 3. **Long-tail quality under distribution shift**
Paper 18 (Tail-Aware Scheduling) is the only one that explicitly addresses distribution shift. Most papers assume stationary workloads. Real-world LLM traffic has bursty arrivals, changing query types, and evolving user behavior that can invalidate assumptions.
### 4. **Hardware heterogeneity and portability**
Papers evaluate on specific hardware (H100, Blackwell, RTX PRO 6000, specific phones/boards). No paper measures how techniques transfer across different GPU generations, memory configurations, or cloud vs. edge deployments. Paper 20 (LENS) addresses this for NPU prediction but only for latency.
### 5. **End-to-end system reliability under failures**
Paper 2 (Concordia) is the only one addressing fault tolerance. Real production systems experience GPU failures, network partitions, and software crashes. No other paper measures recovery time or availability impact.
### 6. **Memory bandwidth vs. compute trade-off at scale**
Papers report throughput and latency but rarely measure the memory bandwidth utilization or compute-to-memory ratio that determines real-world scaling. Paper 1 (Kamera) notes "decode is memory-bandwidth-bound" but does not quantify this.
### 7. **User-perceived quality beyond token-level metrics**
Papers use token-level metrics (TTFT, TBT, throughput)
[prompt: 45,953 completion: 1,200 tokens]
[Implementation guidance]
────────────────────────────────────────────────────────────
Here is a prioritized, actionable engineering roadmap for building a production LLM inference system from scratch, synthesized from the provided research bundle.
### Executive Summary
Building a production-grade LLM inference system requires moving beyond a single "fast decode" metric. The research bundle reveals a multi-dimensional optimization space where **cost, latency, quality, and reliability** are in constant tension. The most impactful lessons are not about a single breakthrough, but about architecting a system that can **dynamically trade off** these dimensions based on workload, context, and hardware constraints.
The roadmap is structured into three phases: **Foundation** (building the core, efficient serving stack), **Optimization** (exploiting workload-specific slack for cost and speed), and **Resilience** (ensuring correctness and fault tolerance in production).
---
### Phase 1: Foundation – Build a Correct, Efficient, and Predictable Core
**Goal:** Establish a baseline serving system that is fast, reproducible, and can handle the fundamental memory and compute bottlenecks. This is the non-negotiable foundation.
#### Priority 1: Master the KV-Cache Bottleneck with a Memory-First Architecture
**Key Lesson:** The KV cache is the dominant cost and bottleneck. Its management is the central systems problem, not just the model's. (Papers 1, 4, 10, 13)
- **Actionable Engineering:**
1. **Adopt a Paged-Attention-like Architecture (vLLM):** This is the de-facto standard. It solves the memory fragmentation problem of naive KV caching, enabling high throughput and dynamic batching. **Do not build your own from scratch**; integrate vLLM or its core principles.
2. **Implement a Multi-Tiered Cache:** Don't treat all KV cache as equal. Use a **hierarchy**:
- **L1: GPU HBM (Fast, Scarce):** For the current active context window.
- **L2: Host DRAM (Slower, Abundant):** For offloaded, long-lived context (e.g., from RAG). This is critical for long-context workloads. (Paper 9)
- **L3: CXL/Disaggregated Memory (Future):** For scaling beyond a single node. (Paper 15)
3. **Use a Recency/Frequency (RF) Adaptive Eviction Policy:** Replace the default LRU. An RF policy (Paper 10) is crucial for mixed workloads (e.g., a few long RAG queries interleaved with many short chat turns). It prevents the long-context queries from being evicted by the short ones, directly improving cache hit rate and TTFT.
4. **Aggressively Quantize the KV Cache:** This is the single most impactful lever for reducing memory pressure.
- **Start with FP8** (the current safe baseline).
- **Target 4-bit (UltraQuant)** for context-heavy agents (Paper 13). This requires careful engineering: use asymmetric K/V quantization, Walsh-Hadamard rotation to reduce outliers, and block-scale scaling. This can yield **3.47x TTFT reduction** in cache-pressured scenarios.
#### Priority 2: Ensure Numerical Reproducibility for Mission-Critical Tasks
**Key Lesson:** FP16 inference is not deterministic across different GPUs. For finance, law, and medicine, this is a showstopper. (Paper 11)
- **Actionable Engineering:**
1. **Adopt a "Hybrid Precision" Strategy (HEAL):** Do not use a global, slow FP32 pipeline. Instead:
- **Quantize Q, K, V to INT16:** This preserves numerical stability without expanding the KV cache footprint.
- **Use Algebraic Error Compensation:** Execute high-precision matrix multiplications on fast FP16 Tensor Cores by correcting the truncation error.
2. **Implement a Reproducibility Benchmark (MCR-Bench):** Before deploying to production, validate that your system's outputs are deterministic across your target hardware (e.g., A100 vs. H100). This is a **pre-deployment gate**.
#### Priority 3: Build a Latency Prediction System (LENS)
**Key Lesson:** You cannot exhaustively profile every configuration. You need a model to predict latency. (Paper 20)
- **Actionable Engineering:**
1. **Profile each "bucket" (batch size, sequence length) with just 2 end-to-end measurements.** This captures the non-linear latency behavior caused by compiler optimizations and hardware bucketing.
2. **Use this predictor for system-level decisions:** It enables you to:
- **Optimize batching:** Dynamically choose the optimal batch size for a given request.
- **Tune parallelism:** Decide on the right model parallelism (e.g., tensor vs. pipeline) for a given workload.
- **Schedule requests:** Use predicted latency for better load balancing.
---
### Phase 2: Optimization – Exploit Workload-Specific Slack for Cost and Speed
**Goal:** Move beyond "max throughput" to a system that dynamically trades off performance for cost and energy efficiency based on the *user's actual needs*.
#### Priority 4: Implement a "Complexity Router" for Cost-Efficient Serving
**Key Lesson:** The most expensive model (e.g., Opus) is not needed for most requests. A simple, fast model is often sufficient. (Paper 5)
- **Actionable Engineering:**
1. **Build a Rule-Based Router (RLM-Cascade):** This is the most practical starting point. Classify requests in O(1) time:
- **Simple:** "what is", "summarize", "hello
[prompt: 45,960 completion: 1,200 tokens]
In [ ]:
HARD_QUESTION = (
"Speculative decoding and continuous batching are both presented as solutions to LLM inference latency. "
"Based on the papers in this bundle, under what conditions does each approach win? "
"Are they complementary or in tension? "
"If you had to pick one for a production deployment serving mixed workloads (short and long outputs, "
"bursty traffic), which would you recommend and why?"
)
base_message = [
{
"role": "user",
"content": f"Research bundle:\n\n{research_bundle}\n\n---\n\nQuestion: {HARD_QUESTION}",
}
]
resp_nothink = client.chat.completions.create(
model=MODEL,
messages=base_message,
max_tokens=1500,
temperature=0.2,
)
resp_think = client.chat.completions.create(
model=MODEL,
messages=base_message,
max_tokens=4000,
temperature=0.6,
extra_body={"chat_template_kwargs": {"thinking": True}},
)In [ ]:
print("NON-THINK")
print("─" * 60)
print(resp_nothink.choices[0].message.content)
think_msg = resp_think.choices[0].message
reasoning = (
getattr(think_msg, "reasoning_content", None)
or (think_msg.model_extra or {}).get("reasoning_content")
or (think_msg.model_extra or {}).get("reasoning")
)
print("\n\nTHINK-HIGH")
print("─" * 60)
if reasoning:
print(f"[reasoning — {len(reasoning.split()):,} words]\n")
print(reasoning)
print("\n[answer]\n")
print(think_msg.content)
reasoning_words = len(reasoning.split()) if reasoning else 0
answer_words = len((think_msg.content or "").split())
print(f"\nTOKEN SUMMARY")
print(f" Non-think completion tokens : {resp_nothink.usage.completion_tokens:>6,}")
print(f" Think-High completion tokens : {resp_think.usage.completion_tokens:>6,}")
print(f" ├─ reasoning block (~words): {reasoning_words:>6,}")
print(f" └─ final answer (~words) : {answer_words:>6,}")
print(
f" Extra tokens spent thinking : {resp_think.usage.completion_tokens - resp_nothink.usage.completion_tokens:>+6,}"
)NON-THINK
────────────────────────────────────────────────────────────
Based on the papers in this bundle, here's my analysis of speculative decoding vs. continuous batching for LLM inference latency, and my recommendation for production deployment.
## When Each Approach Wins
### Speculative Decoding (Papers 5, 12, 16, 19)
**Wins when:**
- **Draft-verify cost asymmetry is large** — The draft model is significantly cheaper/faster than the verify model (Paper 5: 45.8% cost reduction, 1.83× speedup)
- **Workload has exploitable structure** — Many requests are simple enough for a small draft model (Paper 5: 64-70% SKIPPED path)
- **Output quality is the priority** — Speculative decoding preserves the verify model's distribution (Paper 5: 100% vs 95% pass rate)
- **Latency is secondary to cost** — Accepts TTFT regression (2.1× slower) in exchange for lower end-to-end latency and cost (Paper 5, §6.2)
- **Draft budget can be large** — JetFlow (Paper 19) shows up to 9.64× speedup with parallel tree drafting when acceptance remains high
**Fails when:**
- Draft and verify models are from different providers with no shared vocabulary (Paper 5's key limitation)
- Draft quality degrades under evolving policies (Paper 16: RL rollouts)
- The workload is dominated by complex, multi-hop reasoning (Paper 1's cross-chunk binding)
### Continuous Batching (Papers 5, 18)
**Wins when:**
- **Throughput is the primary metric** — Maximizes GPU utilization by batching multiple requests
- **Workload is homogeneous** — Similar-length requests benefit from predictable scheduling
- **Latency variance is acceptable** — Mean-centric metrics improve, but tail latency may suffer
- **Memory pressure is manageable** — KV cache fits within GPU memory
**Fails when:**
- **Workload has extreme length variability** — Paper 18 shows SRPT with perfect knowledge still has P99 TTLT 35-50% worse than tail-aware scheduling
- **Bursty arrivals** — Continuous batching can't handle sudden spikes without preemption
- **Memory is the bottleneck** — Paper 1 shows KV cache becomes the binding constraint, not compute
## Are They Complementary or in Tension?
**They are complementary, not in tension.**
Speculative decoding operates at the *model level* (draft-verify), while continuous batching operates at the *system level* (request scheduling). They address different bottlenecks:
- **Speculative decoding** reduces per-request latency by exploiting model asymmetry
- **Continuous batching** improves throughput by exploiting request-level parallelism
Paper 5 (RLM-Cascade) explicitly shows they can be combined: "These operate at the model layer and are orthogonal to—and composable with—RLM-Cascade's application-layer response composition" (§5.3).
The tension is only in resource allocation: both consume GPU compute, but speculative decoding's draft-verify pipeline adds sequential latency that continuous batching's parallelism can't mask.
## Recommendation for Production Deployment
**For a production deployment serving mixed workloads (short and long outputs, bursty traffic), I recommend: Continuous batching with tail-aware scheduling, not speculative decoding.**
### Why:
1. **Bursty traffic is the killer** — Paper 18 shows that even with perfect decode-length knowledge, SRPT fails under bursty arrivals. The tail-aware scheduling framework reduces P99 TTLT by 35-50% without requiring prediction.
2. **Length variability is extreme** — LLM serving exhibits "extreme length variability" (Paper 18). Continuous batching with tail-aware scheduling handles this naturally; speculative decoding's draft-verify pipeline breaks when draft quality varies.
3. **Memory is the bottleneck, not compute** — Paper 1 shows that for multimodal agents, "Memory, not the nominal window, is the operative bound." Continuous batching with KV cache management (Paper 10's adaptive caching) addresses this directly.
4. **Speculative decoding's TTFT regression is unacceptable** — Paper 5 reports 2.1× slower TTFT. For interactive workloads, this is a deal-breaker. The paper itself acknowledges this as a "Pareto trade-off" (§6.2).
5. **Continuous batching + tail-aware scheduling is proven** — Paper 18 achieves 34-47% TTFT reduction across reasoning-heavy and chat-heavy workloads. This is robust to distribution shifts.
### The Hybrid Approach
If you must use both, use **speculative decoding only for the simple, short-output requests** (the 64-70% SKIPPED path in Paper 5) and **continuous batching for the complex, long-output requests**. This is exactly what Paper 5's router does: "Tool-selection turns are identified by a non-empty tools array and forwarded directly to claude-opus-4-8 with no draft stage" (§3.5).
### Practical Deployment
For a production system:
1. **Primary**: Continuous batching with tail-aware scheduling (Paper 18)
2. **Secondary**: Speculative decoding for simple requests (Paper 5's router)
3. **Fallback**: Direct Opus for complex, schema-critical turns (Paper 5's tool-call carve-out)
This gives you the throughput of continuous batching with the cost savings of speculative decoding where it works, without the TTFT regression where it doesn't.
THINK-HIGH
────────────────────────────────────────────────────────────
[reasoning — 736 words]
Okay, this is a complex question about comparing speculative decoding and continuous batching for LLM inference. I need to analyze the papers in the bundle to understand their strengths, weaknesses, and how they relate. The user wants to know under what conditions each wins, if they're complementary or in tension, and which to recommend for a production deployment with mixed workloads.
First, I should recall what each technique is. Speculative decoding uses a small draft model to generate tokens quickly, which a large model verifies in parallel. This reduces latency by making fewer serial passes through the large model. Continuous batching is about processing multiple requests together in a batch, dynamically adding and removing requests as they finish, to improve throughput and GPU utilization.
Looking at the papers, Paper 5 (RLM-Cascade) is about response-level speculative decoding for API cost reduction. Paper 18 (Beyond Prediction) is about tail-aware scheduling, which relates to batching. Paper 19 (JetFlow) is about improving speculative decoding with parallel tree drafting. Paper 16 (EfficientRollout) is about self-speculative decoding for RL rollouts. There's also Paper 1 (Kamera) about KV cache reuse, which is related to batching efficiency. Paper 10 (Recency/Frequency) about KV caching strategies.
The key insight is that these two approaches operate at different levels. Speculative decoding reduces the number of serial steps per request by having a drafter predict multiple tokens that get verified. Continuous batching increases throughput by packing multiple requests into a single batch, but each request still goes through its own decoding steps.
Are they complementary? Yes, they can be. Speculative decoding reduces the effective number of decoding steps per token, which means each request finishes faster. Continuous batching increases the number of requests processed per unit time. They address different bottlenecks: speculative decoding targets the memory-bound decode phase where each step is slow, while continuous batching targets the compute-bound prefill phase where parallelism helps.
But there's a tension. Speculative decoding works best when the draft model is accurate and the acceptance rate is high. If the draft is poor, you waste compute on verification. Continuous batching works best when there are many requests to batch together. Under bursty traffic, batching helps by filling the GPU, but speculative decoding might be less effective if the draft model can't keep up with the diversity of requests.
For production with mixed workloads (short and long outputs, bursty traffic), I need to think about which is more robust. Continuous batching is well-established and works across all workloads. Speculative decoding requires a good draft model and can have variable speedups. The papers suggest that speculative decoding can achieve significant speedups (like 9.64x in Paper 19), but that's under ideal conditions. The user's question about "under what conditions does each approach win" suggests I need to identify the regimes.
Looking at Paper 5, RLM-Cascade shows that for simple requests, the draft model can be used directly (SKIPPED path) and that's the majority. For complex requests, the draft-verify pipeline adds latency. So speculative decoding wins when the draft is accurate and the verify overhead is low. Continuous batching wins when there are many requests to batch, which is the case under high load.
I think the answer is that they are complementary. Speculative decoding reduces per-request latency, continuous batching increases throughput. For a deployment with mixed workloads, I'd recommend continuous batching as the primary mechanism because it's more robust, and then layer speculative decoding on top for the long-output requests where it can help. But the user asks "if you had to pick one" - that's a tough choice. I think continuous batching is more fundamental and works across all workloads, while speculative decoding is an optimization that may not always apply. For bursty traffic, continuous batching handles the load better by keeping the GPU busy. For short outputs, speculative decoding might not help much because the verification overhead could outweigh the gain. So for mixed workloads, continuous batching is the safer bet.
But I need to support this with evidence from the papers. Paper 18 shows that continuous batching with proper scheduling (tail-aware) can reduce P99 latency by 35-50%. That's significant. Paper 19 shows speculative decoding can achieve high speedups but under specific conditions. Paper 16 shows that for RL rollouts, which are long, speculative decoding helps but needs system-aware tuning.
I'll structure my answer: first explain each approach, then analyze conditions where each wins, discuss complementarity/tension, and finally give a recommendation with reasoning.
[answer]
# Speculative Decoding vs. Continuous Batching: Analysis for Production LLM Serving
Based on the papers in this bundle, I'll analyze the tradeoffs between these two latency-reduction techniques and provide a recommendation for mixed workloads.
## What Each Approach Does
**Speculative Decoding** (Papers 5, 16, 19, 12) uses a fast draft model to generate token candidates that a larger verify model accepts in parallel. This reduces the number of serial autoregressive steps. Key variants:
- **Response-level** (Paper 5, RLM-Cascade): Draft entire response, verify with a single API call
- **Self-speculative** (Paper 16, EfficientRollout): Draft from the target model itself (quantized)
- **Tree drafting** (Paper 19, JetFlow): Generate multiple candidate paths for higher acceptance
**Continuous Batching** (implicit in Papers 18, 10, 15) processes multiple requests simultaneously by dynamically adding/removing sequences from a batch as they complete decode steps. This improves GPU utilization by keeping compute units busy during memory-bound phases.
## Conditions Where Each Approach Wins
### Speculative Decoding Wins When:
1. **Draft model is accurate** → high acceptance rate (Paper 5: 88.8% draft-use rate on simple agentic turns)
2. **Outputs are long** → more tokens to speculate over (Paper 19: up to 9.64× speedup on MATH-500)
3. **Draft cost is low** relative to verify cost (Paper 5: DeepSeek costs ~2% of Opus)
4. **Workload is structured** → router can classify simple vs. complex (Paper 5: SKIPPED path dominates at 64-70%)
5. **Latency is acceptable** for the verify path (Paper 5: 2.1× TTFT regression on complex turns)
**Key insight from Paper 5**: The SKIPPED path actually *replaces* the expensive model for most requests, making the system *faster* overall despite adding a verify stage for complex turns. This counter-intuitive result holds because simple requests dominate.
### Continuous Batching Wins When:
1. **High concurrency** → many requests to fill the batch (Paper 18: tail-aware scheduling)
2. **Outputs are short** → batching overhead is small relative to per-request gain
3. **Memory pressure is high** → KV cache management matters (Papers 10, 15)
4. **Workload is homogeneous** → predictable decode lengths (Paper 18: SRPT with perfect knowledge)
5. **GPU compute is underutilized** → batching fills idle cycles
**Key insight from Paper 18**: Prediction-driven policies (like speculative decoding's draft/verify) are *fragile under distribution shifts*—bursty arrivals and memory pressure degrade tail latency. Continuous batching with proper preemption handles this more robustly.
## Are They Complementary or in Tension?
**They are complementary**, not in tension. They operate at different levels:
| Dimension | Speculative Decoding | Continuous Batching |
|-----------|---------------------|---------------------|
| **What it reduces** | Number of serial decode steps per request | Idle GPU time between requests |
| **Bottleneck** | Memory bandwidth (single request) | Compute utilization (many requests) |
| **When applied** | Per-token generation | Request scheduling |
| **Dependency** | Needs accurate draft model | Needs batchable workload |
Speculative decoding reduces *per-request latency*; continuous batching increases *system throughput*. They can be composed: use speculative decoding to reduce each request's decode steps, then batch those shorter requests together.
**Tension arises** only if speculative decoding makes outputs *too short* to batch effectively (e.g., Paper 5's SKIPPED path completes in 800-1200ms, which is fast but may leave GPU idle between batches). This is a scheduling optimization, not a fundamental conflict.
## Recommendation for Production Deployment: Continuous Batching (with caveats)
For a **mixed workload** (short/long outputs, bursty traffic), I recommend **continuous batching as the primary mechanism**, with speculative decoding as an optional accelerator for long-output requests.
### Why Continuous Batching Wins for Mixed Workloads
1. **Robustness to distribution shifts** (Paper 18): Continuous batching with proper scheduling (tail-aware, cache-aware preemption) handles bursty arrivals and memory pressure *without* requiring a draft model that may be mismatched. Speculative decoding's draft quality degrades under distribution shift (Paper 16: "evolving policy makes any fixed drafter increasingly mismatched").
2. **No accuracy risk**: Continuous batching is lossless—it changes only scheduling, not model output. Speculative decoding introduces quality risk (Paper 5: 100% vs 95% on some benchmarks, but the ENHANCED path can *increase* cost on mteb-retrieve).
3. **Handles short outputs**: Speculative decoding's benefit on short outputs is minimal (Paper 5: 64-70% of requests are simple, but the SKIPPED path's speedup comes from *avoiding* the verify model, not from speculation itself). For short outputs, batching is more impactful.
4. **Memory efficiency**: Continuous batching with KV cache management (Papers 10, 15) directly addresses the memory bottleneck that limits throughput. Speculative decoding's draft models add memory pressure.
### When to Add Speculative Decoding
Use speculative decoding **only for long-output requests** (Paper 5's "complex" classification, Paper 19's long-context tasks). This requires:
- A lightweight complexity router (Paper 5's keyword-based approach)
- A fallback mechanism for when the draft is wrong (Paper 5's ENHANCED path)
- Monitoring for cost inversion (Paper 5's mteb-retrieve case)
### Practical Deployment Strategy
```
Primary: Continuous batching (vLLM-style)
+ Tail-aware scheduling (Paper 18)
+ KV cache management (Papers 10, 15)
Optional: Speculative decoding for long outputs
+ Complexity router (Paper 5)
+ Self-speculative draft (Paper 16) — no separate model
+ Fallback to direct generation when draft quality drops
```
This hybrid approach gives you the robustness of batching for bursty traffic and the latency reduction of speculation for long outputs, without the fragility of either alone.
**Final answer**: Continuous batching for robustness, speculative decoding as an accelerator for long-output tails.
TOKEN SUMMARY
Non-think completion tokens : 1,156
Think-High completion tokens : 2,337
├─ reasoning block (~words): 736
└─ final answer (~words) : 876
Extra tokens spent thinking : +1,181