feat(scheduler): live-tail allocation for sliding groups - #694
feat(scheduler): live-tail allocation for sliding groups#694nperrin-fr wants to merge 7 commits into
Conversation
Opt-in live_tail_alloc for sliding-window paged-cache groups: SwaManager allocates real pages only for the live tail and the resume pages behind each LCM-aligned restore boundary, appending null holes for slots the retention can never read back. This clips the full-chunk admission transient at allocation time instead of punching pages a chunk later. Safe only for groups whose in-flight reads never come from the pool (conv kernels read the chunk from packed activations); SWA attention groups keep full allocation. MakeCoordinator wires the groups' LCM block size as the alignment; BlocksNeededFor mirrors the clipped math so admission stops charging the transient; reclaim walks the interleaved holes instead of early-breaking on the first null. Adds 10 unit tests. Signed-off-by: nperrin-fr <nperrin.ai@gmail.com>
kvconv/hiddenconv specs opt into the scheduler's allocate-time window clipping, removing the ~17 GB/GPU chunked-prefill admission transient for the tiny-window conv-state groups (second-wave shared-prefix hit rate recovers under concurrency with no workaround). Signed-off-by: nperrin-fr <nperrin.ai@gmail.com>
1d98ee0 to
89ca28c
Compare
nperrin-fr
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 626ea7c8c2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // Live-tail interleaves holes with real checkpoint pages, so don't early-break. | ||
| if (live_alloc_alignment_ == 0) { | ||
| break; // already null -> earlier slots are null too | ||
| } | ||
| continue; |
There was a problem hiding this comment.
With live_tail_alloc enabled, continuing after every null makes each ReclaimExpired call walk from the current slide bound back to slot 0, even though almost all earlier slots were already punched on prior decode steps. For Inkling hiddenconv's 8/16-token pages, long-context decoding scans tens or hundreds of thousands of holes per group per token (and BlocksReclaimableAt repeats the same pattern gate-side), which can dominate scheduler latency; keep a reclaimed cursor or skip directly to possible checkpoint residues instead of rescanning the entire sparse history.
Useful? React with 👍 / 👎.
Signed-off-by: Nicolas <nperrin.ai@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75439f6a11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (num_tokens <= tail_avail) { | ||
| return 0; | ||
| } | ||
| const std::int32_t end = TableExtentTokens(table, block_size_) + num_tokens; |
There was a problem hiding this comment.
For live-tail groups this prediction is not monotonic because it classifies all new slots against the final frontier. For example with block_size=8/window=12, a fresh BlocksNeededFor(19) returns 2 while the actual first Acquire(17) needs 3 pages (the later frontier turns slot 0 into a hole). flatAdmitFirstChunk gates on BlocksNeededFor(chunk + decode_reserve) before the C slice actually acquires only chunk_tokens, so under page pressure admission can succeed with two free blocks and then hit the allocation-failure assert when the chunk needs three. Please charge the peak/sequential acquire demand, not just the final table shape.
Useful? React with 👍 / 👎.
Two review findings on the live-tail allocation path: BlocksNeededFor is non-monotonic for live-tail groups (a later frontier holes early slots: BlocksNeededFor(17)=3 but BlocksNeededFor(19)=2 at block 8 / window 12), so gating chunk + decode-reserve on the combined query could under-charge the chunk's own Acquire and compute a negative reserve that the >0 guard silently dropped. Admission now charges BlocksNeededForSequential (first batch at its own frontier + extra batch at the final frontier), which equals the sequential acquires' peak pin and makes the recorded reserve exactly the second acquire's demand. ReclaimExpired walked from the slide bound back to slot 0 every call (holes forbid the early break), rescanning the entire punched history per decode step -- O(N^2) over a long generation, twice per token with the BlocksReclaimableAt gate. A reclaim floor on BlockTable now bounds both scans to the newly-expired band; EvictToNull is permanent, so slots below the floor can never re-materialize. Signed-off-by: Nicolas <nperrin.ai@gmail.com>
…ent coupling Adversarial review of the floor invariant found the "known-null" claim false on reachable tables: SwaManager::Match emits holes below hits_begin and AppendHostExtension appends null slots, so an alignment==0 early break can advance the floor over stranded real pages (the pre-floor scan stranded the exact same pages; Free releases them). State the comment as the scan-bound guarantee it actually is. Also record why flatAdmitFirstChunk's fresh-table sequential charge is exact: SweepThenConverge pins claim extents to the same lcm MakeCoordinator wires as the live alignment. slotIsLive is not shift-invariant, so decoupling those would silently under-charge. Signed-off-by: Nicolas <nperrin.ai@gmail.com>
1 participant
Summary
Chunked prefill previously allocated the FULL chunk range in every
paged-cache group and punched the slid-out pages one chunk later. For
tiny-window conv-state groups (Inkling hiddenconv: block 16, wide
columns) that transient is ~9 GB/GPU per 8k in-flight chunk — the
documented admission transient that evicts every retained prefix entry
under concurrency.
This adds opt-in live-tail allocation for sliding-window groups
(PagedCacheGroupConfig.live_tail_alloc): SwaManager::Acquire appends
null holes for new slots the retention can never read back and real
blocks only for
(a) slots not fully slid out at the post-acquire frontier, and
(b) the pagesNeededToResume() slots behind every LCM-aligned restore
boundary (the pages the engine's lcm_align persist set writes and
a future prefix-cache restore must find registered).
MakeCoordinator wires the groups' LCM block size as the alignment;
BlocksNeededFor mirrors the clipped math so admission stops charging
the transient; ReclaimExpired/BlocksReclaimableAt walk through the
interleaved holes instead of early-breaking on the first null.
Safe only for groups whose in-flight reads never come from the pool
(conv kernels read the chunk from packed activations); SWA attention
groups keep full allocation. hiddenconv math: 2 of every 32 slots stay
real once slid out -> the transient drops ~16x.
10 new unit tests (manager + coordinator lcm wiring); full scheduler
suite: failure set identical to the parent commit (the 103 known
radix-oriented failures under -DTOKENSPEED_FLAT_KVCACHE=ON).
Test Plan