Skip to content

fix(inkless:switch): cap tiered reads at the seal [KC-739] - #867

Merged
jeqo merged 3 commits into
mainfrom
claude/kc-739-analysis-eee602
Nov 12, 2026
Merged

jeqo merged 3 commits into
mainfrom
claude/kc-739-analysis-eee602

Conversation

@giuseppelillo

@giuseppelillo giuseppelillo commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

A consumer that reads the classic prefix of a switched partition can receive discarded records from remote storage in place of the diskless records at the same offsets. The read succeeds, so the consumer skips the diskless records without an error. PR #857 lists this case under Limitations.

Problem

RemoteLogManager.read returns records from a remote segment until the fetch size runs out. It doesn't stop at the classic-to-diskless seal. If a remote segment crosses the seal, a tiered read that starts below the seal returns the classic records past it. Those offsets belong to the diskless log.

A remote segment crosses the seal after an unclean leader election before the switch. The following example builds the case step by step. Each diagram uses the same offset axis.

Step 1: X leads and uploads a segment

Broker X becomes the leader in epoch 3 at offset 100 and appends [100, 120). It uploads one remote segment that covers offsets 50 to 119 and holds epochs 2 and 3.

offset            50                             100                 120
                  |                               |                   |
log of X          [ epoch 2, 50..99               | epoch 3, 100..119 ]
                                  |
                                  | uploads
                                  v
remote storage    [ one segment, 50..119                              ]

Step 2: an unclean leader election discards the tail

An unclean leader election makes Y the leader in epoch 4. The log of Y ends at 100, so the partition loses [100, 120). The remote segment still holds those records.

offset            50                             100                 120
                  |                               |                   |
log of Y          [ 50..99, ends at 100           ]  lost, never on Y
remote storage    [ epoch 2, 50..99               | discarded 100..119 ]

Step 3: the switch seals at 100 before any produce

Y seals at 100 with no produce between the election and the seal. The diskless log starts at 100, and producers write the real records for 100 and up there. The leader epoch cache of Y maps [50, 100) to epoch 2, and remote storage already has epoch 2 up to 99. Y never uploads its own segment for that range, so the segment of X stays.

offset            50                             100 = seal          120
                  |                               |                   |
classic log of Y  [ 50..99                        ]
diskless log                                      [ real records from 100 ...
remote storage    [ epoch 2, 50..99               | discarded 100..119 ]

Step 4: a consumer reads below the seal

A consumer fetches at 60. The offset is below the seal, so the broker takes the tiered path. RemoteLogManager.read finds the segment of X and returns records through 119. The consumer then fetches at 120 and never reads the diskless records [100, 120).

offset            50                             100 = seal          120
                  |                               |                   |
remote storage    [ epoch 2, 50..99               | discarded 100..119 ]
diskless log                                      [ real records from 100 ...

fetch at 60            o-------------------------->------------------->
                       returns 60..99: correct     returns 100..119: wrong

next fetch at 120: the consumer never reads diskless 100..119

The case needs all of these:

  • An unclean leader election before the switch, through the cluster default or an ElectLeaders request. Phase 1 blocks unclean leader election.
  • A discarded tail that the old leader already uploaded.
  • A seal with no produce between the election and the seal. With produce in between, the segment crosses the old log end and not the seal. That is the gap that upstream tiered storage has after any unclean leader election, and this change doesn't cover it.

Fix

The broker caps a tiered read of a switched partition at the seal:

  • RemoteStorageFetchInfo has a new upperBoundOffset field. The existing constructor leaves it empty.
  • If a partition has a committed seal and a tiered read starts below it, ReplicaManager sets the field to the seal. The bound applies in both phases of the switch: in phase 1, a fetch below the local log start offset also reads from remote storage. Every other partition passes no bound, and its read is unchanged.
  • With a bound, RemoteLogManager.read keeps only the whole batches that end below it. It never cuts inside a batch.
  • The aborted transaction lookup of the same read stops at the bound, so the response doesn't report transactions past the seal.

With the fix, the same fetch at 60 ends at 99. The next fetch is at 100, which is the seal, so the broker routes it to the diskless log.

offset            50                             100 = seal          120
                  |                               |                   |
remote storage    [ epoch 2, 50..99               | discarded 100..119 ]
diskless log                                      [ real records from 100 ...

fetch at 60            o-------------------------->|
                       returns 60..99: correct     the read ends at the seal

next fetch at 100                                 o------------------->
                                                  returns diskless 100..119: correct

A batch that spans the seal can't exist in this case: the log end of Y is a batch boundary of the same batches that X uploaded. If one did exist, the read would leave it out.

Testing

  • RemoteLogManagerTest.testReadEndsAtUpperBoundOffset reads a segment that crosses the bound in one epoch, one with a discarded epoch past the bound, one that ends below the bound, and one with no bound.
  • RemoteLogManagerTest.testReadCollectsAbortedTransactionsBelowUpperBoundOffset checks that the read reports only the aborted transaction below the bound, and both transactions with no bound.
  • ReplicaManagerInklessTest checks that the fetch info of a switched partition carries the seal in phase 2, and in phase 1 for a fetch below the local log start offset. ReplicaManagerTest checks that a classic tiered partition passes no bound.

KC-739

🤖 Generated with Claude Code

Assisted-by: Claude Fable 5.2
Assisted-by: Claude Sonnet 5.5

After an unclean leader election before the switch, a remote segment
can cross the classic-to-diskless seal and hold discarded records past
it. `RemoteLogManager.read` returns records up to the fetch size, so a
consumer that reads below the seal receives those discarded records and
then skips the diskless records at the same offsets, without an error.

`ReplicaManager` now passes the seal as an upper bound offset in
`RemoteStorageFetchInfo` when a switched partition reads its classic
prefix from remote storage. `RemoteLogManager.read` keeps only the
whole batches that end below the bound, and the aborted transaction
lookup stops at the bound. Every other partition passes no bound, and
its read is unchanged.

Testing

`RemoteLogManagerTest` covers a segment that crosses the bound in one
epoch, one with a discarded epoch past the bound, one that ends below
the bound, a read with no bound, and the aborted transaction lookup.
`ReplicaManagerInklessTest` and `ReplicaManagerTest` check that the
fetch info carries the seal only for a switched partition.

Assisted-by: Claude Fable 5.2
Assisted-by: Claude Sonnet 5.5

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Tiered reads remain unbounded during phase 1 after the seal is committed, allowing discarded records past the seal to be returned and causing consumers to skip diskless records.

1 open issue
What changed in this PR

This PR caps tiered reads at the classic-to-diskless seal to prevent discarded remote records from replacing diskless records at the same offsets.

Changes:

  • Add an optional exclusive offset bound to remote fetch requests and populate it for phase 2 classic-prefix reads.
  • Filter remote responses at whole-batch boundaries and cap aborted-transaction lookups.
  • Add regression tests and update switch documentation and ownership tracking.
File Description
storage/​src/​test/​java/​org/​apache/​kafka/​server/​log/​remote/​storage/​RemoteLogManagerTest.java Test bounded records and aborted-transaction results.
storage/​src/​main/​java/​org/​apache/​kafka/​storage/​internals/​log/​RemoteStorageFetchInfo.java Add an optional upper bound while preserving the existing constructor.
storage/​src/​main/​java/​org/​apache/​kafka/​server/​log/​remote/​storage/​RemoteLogManager.java Apply the bound to returned batches and transaction lookups.
INKLESS_OWNERSHIP Track ownership of the remote-read changes.
docs/​inkless/​CLASSIC_TO_DISKLESS_SWITCH.md Document seal-bounded tiered reads.
core/​src/​test/​scala/​unit/​kafka/​server/​ReplicaManagerTest.scala Verify classic partitions pass no bound.
core/​src/​test/​scala/​unit/​kafka/​server/​ReplicaManagerInklessTest.scala Verify phase 2 fetches carry the seal.
core/​src/​main/​scala/​kafka/​server/​ReplicaManager.scala Attach the seal to phase 2 classic-prefix remote fetches.

Review effort: Balanced

The bound applied only in phase 2, because it reused the check that
requires the classic prefix in remote storage. In phase 1 the seal is
already committed, and a fetch below the local log start offset also
reads from remote storage, so it could return discarded records past
the seal.

`ReplicaManager` now sets the bound whenever the seal is committed and
the fetch starts below it.

Testing

`ReplicaManagerInklessTest` covers a phase 1 tiered read below the
local log start offset.

Assisted-by: Claude Fable 5.2

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the changes that bound tiered reads at the classic-to-diskless seal.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@giuseppelillo
giuseppelillo requested a review from jeqo November 11, 2026 14:46

@jeqo jeqo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a minor comment to improve logging on an edge case.

…h [KC-739]

If the first batch of a tiered read crosses the upper bound offset, the
response is empty and the consumer fetches the same offset again. Log the
cause so the stall is diagnosable.

Assisted-by: Claude Sonnet 5.5
@jeqo
jeqo merged commit 0e3aed8 into main Nov 12, 2026
7 checks passed
@jeqo
jeqo deleted the claude/kc-739-analysis-eee602 branch November 12, 2026 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants