Refactor Req.fill_ids into full_untruncated_fill_ids + fill_len with equivalence (#26637)

This commit is contained in:
fzyzcjy
2026-06-08 14:52:18 +08:00
committed by GitHub
parent 4201de11de
commit 259a2da3e0
29 changed files with 136 additions and 103 deletions
@@ -3,14 +3,14 @@
Covers two bugs with the same crash signature
(RuntimeError: shape mismatch in set_kv_buffer) but opposite polarity:
- Chunked prefill truncation (#20376): PrefillAdder truncates fill_ids and
- Chunked prefill truncation (#20376): PrefillAdder shrinks fill_len and
extend_input_len on chunk overflow but not input_embeds, so the full array
flows through while out_cache_loc is sized for the truncated length.
Polarity: cache_k > loc.
- Retraction with output_ids (#14110): after retraction, fill_ids includes
accumulated output_ids but input_embeds only covers origin_input_ids.
Polarity: cache_k < loc.
- Retraction with output_ids (#14110): after retraction, get_fill_ids()
includes accumulated output_ids but input_embeds only covers
origin_input_ids. Polarity: cache_k < loc.
"""
import unittest
@@ -162,8 +162,8 @@ class TestInputEmbedsChunkedAndRetract(CustomTestCase):
SGLANG_TEST_RETRACT forces retraction every few scheduler iterations.
Combined with ignore_eos and a reasonable max_new_tokens, at least one
request is retracted mid-decode with non-empty output_ids, then
re-prefilled. Pre-#14110 this crashes (cache_k < loc) because fill_ids
includes output_ids but input_embeds does not.
re-prefilled. Pre-#14110 this crashes (cache_k < loc) because the
filled token sequence includes output_ids but input_embeds does not.
"""
text = "The quick brown fox jumps over the lazy dog. " * 4
embeds = _embeds_for(text)