Skip to content

fix(inkless:switch): skip ISR expansion while a phase 1 leader's log ends below the seal [KC-773] - #880

Merged
viktorsomogyi merged 1 commit into
mainfrom
claude/kc-773-investigation-ea3697
Nov 18, 2026
Merged

viktorsomogyi merged 1 commit into
mainfrom
claude/kc-773-investigation-ea3697

Conversation

@giuseppelillo

Copy link
Copy Markdown
Contributor

ClassicToDisklessSwitchLastKnownLeaderClusterTest (loseTail=true) fails intermittently in CI. The cause is a race in the broker, not in the test.

After a last-known-leader election in phase 1 of the classic-to-diskless switch, the restarted leader runs makeLeader and then fences itself when its log ends below the committed seal. A follower fetch between those two steps expands the ISR. The controller then records a full ISR and a recovered leader for a partition that the leader marks offline. When the operator lowers the seal and restarts the leader, the controller elects another ISR member, and the test times out waiting for the restarted broker to lead.

This contradicts CLASSIC_TO_DISKLESS_SWITCH.md, which says the controller keeps an ISR of one after this election.

Partition.isFollowerInSync now returns false while the leader's log end offset is below the committed seal. The check reads replicationStopOffset() directly instead of the sealed-gated value, because seal() runs after makeLeader on a restarted leader.

Evidence

In the CI artifact for the failure on e3913e8c10 (run 44107792176), the leader is broker 2. When it shuts down in reSeal, broker 0 takes over, which only an ISR member can do.

To reproduce it locally, I added a 1.5-second sleep between makeLeader and maybeReconcileSwitchedLeader in ReplicaManager.applyLocalLeadersDelta. The test then failed on the first run with the ticket's signature: leader=2, isr=[0,1,2], leaderRecoveryState=RECOVERED. The logs of the failing run show:

  • The election sets the ISR to [1] and the state to RECOVERING.
  • Broker 1 logs ISR updated to 1,2 and then ISR updated to 2,1,0 before it marks the partition offline.

Testing

  • PartitionTest.testIsrNotExpandedWhileLeaderLogEndsBelowSeal fails without the change (expected: <0> but was: <1>) and passes with it. All 106 tests in PartitionTest pass.
  • With the same 1.5-second sleep and the change, ClassicToDisklessSwitchLastKnownLeaderClusterTest passes both cases. The sleep isn't part of this PR.
  • Without the sleep, 10 consecutive local runs of the cluster test passed before the change, so the local runs don't show the race on their own.

Fixes KC-773.

🤖 Generated with Claude Code

…ends below the seal [KC-773]

After a last-known-leader election in phase 1 of the classic-to-diskless
switch, the restarted leader runs `makeLeader` and then fences itself when its
log ends below the committed seal. A follower fetch between those two steps
expands the ISR. The controller then records a full ISR and a recovered
leader for a partition that the leader marks offline. When the operator lowers
the seal and restarts the leader, the controller elects another ISR member,
and `ClassicToDisklessSwitchLastKnownLeaderClusterTest` times out waiting for
the restarted broker to lead.

`Partition.isFollowerInSync` now returns false while the leader's log end
offset is below the committed seal. The check reads the stop offset directly,
because `seal()` runs after `makeLeader` on a restarted leader.

Assisted-by: Claude Sonnet 5.5

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the ISR expansion guard and its regression test.

No open issues 🎉

What changed in this PR

This PR closes a race during phase 1 of the classic-to-diskless switch: a restarted leader whose log ends below the committed seal must not expand the in-sync replica set (ISR) before it fences itself.

Changes:

  • Block ISR expansion when the leader’s log end offset is below the committed seal, even before seal() runs.
  • Add a regression test that verifies expansion stays blocked below the seal and resumes when the seal matches the log end offset.
File Description
core/​src/​test/​scala/​unit/​kafka/​cluster/​PartitionTest.scala Tests ISR expansion before sealing and at the seal boundary.
core/​src/​main/​scala/​kafka/​cluster/​Partition.scala Checks the committed replication stop offset before allowing ISR expansion.

Review effort: Balanced

@viktorsomogyi
viktorsomogyi merged commit 82f45b0 into main Nov 18, 2026
9 checks passed
@viktorsomogyi
viktorsomogyi deleted the claude/kc-773-investigation-ea3697 branch November 18, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants