Repository navigation
fix(inkless:switch): skip ISR expansion while a phase 1 leader's log ends below the seal [KC-773] - #880
Merged
Conversation
…ends below the seal [KC-773] After a last-known-leader election in phase 1 of the classic-to-diskless switch, the restarted leader runs `makeLeader` and then fences itself when its log ends below the committed seal. A follower fetch between those two steps expands the ISR. The controller then records a full ISR and a recovered leader for a partition that the leader marks offline. When the operator lowers the seal and restarts the leader, the controller elects another ISR member, and `ClassicToDisklessSwitchLastKnownLeaderClusterTest` times out waiting for the restarted broker to lead. `Partition.isFollowerInSync` now returns false while the leader's log end offset is below the committed seal. The check reads the stop offset directly, because `seal()` runs after `makeLeader` on a restarted leader. Assisted-by: Claude Sonnet 5.5
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the ISR expansion guard and its regression test.
No open issues 🎉
What changed in this PR
This PR closes a race during phase 1 of the classic-to-diskless switch: a restarted leader whose log ends below the committed seal must not expand the in-sync replica set (ISR) before it fences itself.
Changes:
- Block ISR expansion when the leader’s log end offset is below the committed seal, even before
seal()runs. - Add a regression test that verifies expansion stays blocked below the seal and resumes when the seal matches the log end offset.
| File | Description |
|---|---|
core/src/test/scala/unit/kafka/cluster/PartitionTest.scala |
Tests ISR expansion before sealing and at the seal boundary. |
core/src/main/scala/kafka/cluster/Partition.scala |
Checks the committed replication stop offset before allowing ISR expansion. |
Review effort: Balanced
viktorsomogyi
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ClassicToDisklessSwitchLastKnownLeaderClusterTest(loseTail=true) fails intermittently in CI. The cause is a race in the broker, not in the test.After a last-known-leader election in phase 1 of the classic-to-diskless switch, the restarted leader runs
makeLeaderand then fences itself when its log ends below the committed seal. A follower fetch between those two steps expands the ISR. The controller then records a full ISR and a recovered leader for a partition that the leader marks offline. When the operator lowers the seal and restarts the leader, the controller elects another ISR member, and the test times out waiting for the restarted broker to lead.This contradicts CLASSIC_TO_DISKLESS_SWITCH.md, which says the controller keeps an ISR of one after this election.
Partition.isFollowerInSyncnow returns false while the leader's log end offset is below the committed seal. The check readsreplicationStopOffset()directly instead of the sealed-gated value, becauseseal()runs aftermakeLeaderon a restarted leader.Evidence
In the CI artifact for the failure on
e3913e8c10(run 44107792176), the leader is broker 2. When it shuts down inreSeal, broker 0 takes over, which only an ISR member can do.To reproduce it locally, I added a 1.5-second sleep between
makeLeaderandmaybeReconcileSwitchedLeaderinReplicaManager.applyLocalLeadersDelta. The test then failed on the first run with the ticket's signature:leader=2, isr=[0,1,2], leaderRecoveryState=RECOVERED. The logs of the failing run show:[1]and the state toRECOVERING.ISR updated to 1,2and thenISR updated to 2,1,0before it marks the partition offline.Testing
PartitionTest.testIsrNotExpandedWhileLeaderLogEndsBelowSealfails without the change (expected: <0> but was: <1>) and passes with it. All 106 tests inPartitionTestpass.ClassicToDisklessSwitchLastKnownLeaderClusterTestpasses both cases. The sleep isn't part of this PR.Fixes KC-773.
🤖 Generated with Claude Code