Repository navigation
fix(inkless:switch): cap tiered reads at the seal [KC-739] - #867
Merged
Merged
Conversation
After an unclean leader election before the switch, a remote segment can cross the classic-to-diskless seal and hold discarded records past it. `RemoteLogManager.read` returns records up to the fetch size, so a consumer that reads below the seal receives those discarded records and then skips the diskless records at the same offsets, without an error. `ReplicaManager` now passes the seal as an upper bound offset in `RemoteStorageFetchInfo` when a switched partition reads its classic prefix from remote storage. `RemoteLogManager.read` keeps only the whole batches that end below the bound, and the aborted transaction lookup stops at the bound. Every other partition passes no bound, and its read is unchanged. Testing `RemoteLogManagerTest` covers a segment that crosses the bound in one epoch, one with a discarded epoch past the bound, one that ends below the bound, a read with no bound, and the aborted transaction lookup. `ReplicaManagerInklessTest` and `ReplicaManagerTest` check that the fetch info carries the seal only for a switched partition. Assisted-by: Claude Fable 5.2 Assisted-by: Claude Sonnet 5.5
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Tiered reads remain unbounded during phase 1 after the seal is committed, allowing discarded records past the seal to be returned and causing consumers to skip diskless records.
1 open issue
What changed in this PR
This PR caps tiered reads at the classic-to-diskless seal to prevent discarded remote records from replacing diskless records at the same offsets.
Changes:
- Add an optional exclusive offset bound to remote fetch requests and populate it for phase 2 classic-prefix reads.
- Filter remote responses at whole-batch boundaries and cap aborted-transaction lookups.
- Add regression tests and update switch documentation and ownership tracking.
| File | Description |
|---|---|
storage/src/test/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerTest.java |
Test bounded records and aborted-transaction results. |
storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteStorageFetchInfo.java |
Add an optional upper bound while preserving the existing constructor. |
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java |
Apply the bound to returned batches and transaction lookups. |
INKLESS_OWNERSHIP |
Track ownership of the remote-read changes. |
docs/inkless/CLASSIC_TO_DISKLESS_SWITCH.md |
Document seal-bounded tiered reads. |
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala |
Verify classic partitions pass no bound. |
core/src/test/scala/unit/kafka/server/ReplicaManagerInklessTest.scala |
Verify phase 2 fetches carry the seal. |
core/src/main/scala/kafka/server/ReplicaManager.scala |
Attach the seal to phase 2 classic-prefix remote fetches. |
Review effort: Balanced
The bound applied only in phase 2, because it reused the check that requires the classic prefix in remote storage. In phase 1 the seal is already committed, and a fetch below the local log start offset also reads from remote storage, so it could return discarded records past the seal. `ReplicaManager` now sets the bound whenever the seal is committed and the fetch starts below it. Testing `ReplicaManagerInklessTest` covers a phase 1 tiered read below the local log start offset. Assisted-by: Claude Fable 5.2
jeqo
reviewed
Nov 12, 2026
jeqo
left a comment
Contributor
There was a problem hiding this comment.
LGTM, just a minor comment to improve logging on an edge case.
…h [KC-739] If the first batch of a tiered read crosses the upper bound offset, the response is empty and the consumer fetches the same offset again. Log the cause so the stall is diagnosable. Assisted-by: Claude Sonnet 5.5
jeqo
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

A consumer that reads the classic prefix of a switched partition can receive discarded records from remote storage in place of the diskless records at the same offsets. The read succeeds, so the consumer skips the diskless records without an error. PR #857 lists this case under Limitations.
Problem
RemoteLogManager.readreturns records from a remote segment until the fetch size runs out. It doesn't stop at the classic-to-diskless seal. If a remote segment crosses the seal, a tiered read that starts below the seal returns the classic records past it. Those offsets belong to the diskless log.A remote segment crosses the seal after an unclean leader election before the switch. The following example builds the case step by step. Each diagram uses the same offset axis.
Step 1: X leads and uploads a segment
Broker X becomes the leader in epoch 3 at offset 100 and appends
[100, 120). It uploads one remote segment that covers offsets 50 to 119 and holds epochs 2 and 3.Step 2: an unclean leader election discards the tail
An unclean leader election makes Y the leader in epoch 4. The log of Y ends at 100, so the partition loses
[100, 120). The remote segment still holds those records.Step 3: the switch seals at 100 before any produce
Y seals at 100 with no produce between the election and the seal. The diskless log starts at 100, and producers write the real records for 100 and up there. The leader epoch cache of Y maps
[50, 100)to epoch 2, and remote storage already has epoch 2 up to 99. Y never uploads its own segment for that range, so the segment of X stays.Step 4: a consumer reads below the seal
A consumer fetches at 60. The offset is below the seal, so the broker takes the tiered path.
RemoteLogManager.readfinds the segment of X and returns records through 119. The consumer then fetches at 120 and never reads the diskless records[100, 120).The case needs all of these:
ElectLeadersrequest. Phase 1 blocks unclean leader election.Fix
The broker caps a tiered read of a switched partition at the seal:
RemoteStorageFetchInfohas a newupperBoundOffsetfield. The existing constructor leaves it empty.ReplicaManagersets the field to the seal. The bound applies in both phases of the switch: in phase 1, a fetch below the local log start offset also reads from remote storage. Every other partition passes no bound, and its read is unchanged.RemoteLogManager.readkeeps only the whole batches that end below it. It never cuts inside a batch.With the fix, the same fetch at 60 ends at 99. The next fetch is at 100, which is the seal, so the broker routes it to the diskless log.
A batch that spans the seal can't exist in this case: the log end of Y is a batch boundary of the same batches that X uploaded. If one did exist, the read would leave it out.
Testing
RemoteLogManagerTest.testReadEndsAtUpperBoundOffsetreads a segment that crosses the bound in one epoch, one with a discarded epoch past the bound, one that ends below the bound, and one with no bound.RemoteLogManagerTest.testReadCollectsAbortedTransactionsBelowUpperBoundOffsetchecks that the read reports only the aborted transaction below the bound, and both transactions with no bound.ReplicaManagerInklessTestchecks that the fetch info of a switched partition carries the seal in phase 2, and in phase 1 for a fetch below the local log start offset.ReplicaManagerTestchecks that a classic tiered partition passes no bound.KC-739
🤖 Generated with Claude Code
Assisted-by: Claude Fable 5.2
Assisted-by: Claude Sonnet 5.5