Skip to content

fix(inkless:controller): block unclean leader election in switch phase 1 [KC-636] - #849

Merged
viktorsomogyi merged 1 commit into
mainfrom
giuseppelillo/kc-636-block-unclean-election-phase-1
Nov 4, 2026
Merged

viktorsomogyi merged 1 commit into
mainfrom
giuseppelillo/kc-636-block-unclean-election-phase-1

Conversation

@giuseppelillo

Copy link
Copy Markdown
Contributor

In phase 1, the switch relies on one invariant: a replica is in the ISR only if it holds the complete classic prefix. Every unclean-election guard keyed on the pending state (classicToDisklessStartOffset == -2). After the seal committed, an unclean election could make a replica outside the ISR the leader and reset the ISR to that replica. Phase 1 lasts until the remote log manager uploads the prefix, which can take hours for a topic that wasn't tiered.

Block unclean leader election until phase 2

The guards now key on isSwitchedWithLocalClassicPrefix, which covers both the pending state and phase 1:

  • validateDisklessSwitchInvariants rejects unclean.leader.election.enable=true for the topic.
  • electLeader rejects an UNCLEAN election with INVALID_REQUEST.
  • Broker fencing, AlterPartition, and the periodic check for leaderless partitions skip the unclean election.
  • A reassignment cancellation whose revert needs an unclean election fails with INVALID_REPLICA_ASSIGNMENT, and the cancellation doesn't elect an unclean leader either.

The election guards also cover a cluster-level unclean.leader.election.enable=true, which the topic config check doesn't see. Phase 2 keeps the born-diskless behavior.

The periodic check now reads the topic config before the switch state, so it warns only when it would otherwise run an unclean election. The check runs every five minutes, and phase 1 can last hours.

The error and log messages now say that the partition is switching to diskless and its classic prefix is not in remote storage yet, because they cover phase 1 too.

CLASSIC_TO_DISKLESS_SWITCH.md gets a row in the controller phase table and an "Unclean leader election" section.

Testing

  • New parameterized ReplicationControlManagerInklessTest cases cover broker fencing, the leaderless check, ElectLeaders, the config check on both AlterConfigs paths, and reassignment cancellation. Each case checks that phase 1 blocks the unclean election and phase 2 allows it, with the cluster default enabling unclean leader election. Without the fix, every phase 1 case fails.
  • AlterPartition has no new case. Its validation requires the new ISR to include the current leader, so the election there never needs to be unclean.
  • The pending-switch tests in ReplicationControlManagerInklessTest and InklessTopicTypeSwitcherClusterTest pass with the new message.
  • ReplicationControlManagerTest and PartitionChangeBuilderTest pass.
  • make vale doesn't report any alerts.

@giuseppelillo

giuseppelillo commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Stacked on #840. Until #840 merges, this PR also shows its three KC-589 commits. Only the last commit, fix(inkless:controller): block unclean leader election in switch phase 1 [KC-636], belongs to this change. It implements KC-636. The reassignment-cancellation guard is an extra path that the ticket did not list.

@giuseppelillo
giuseppelillo changed the base branch from main to giuseppelillo/kc-589-gate-on-switch-phase November 2, 2026 14:24
@giuseppelillo
giuseppelillo requested a balanced review from Copilot November 2, 2026 14:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the changes extending election safeguards through phase 1 of the classic-to-diskless switch.

No open issues 🎉

What changed in this PR

This PR protects the classic data prefix during a classic-to-diskless switch by extending unclean leader election guards through phase 1, until the prefix reaches remote storage. Phase 2 retains the existing born-diskless behavior.

Changes:

  • Block unclean elections and unsafe reassignment cancellations while the classic prefix remains local.
  • Add parameterized tests for phase 1 restrictions and phase 2 behavior, and update error-message assertions.
  • Document the election restrictions and their behavior when the cluster default enables unclean elections.
File Description
metadata/​src/​test/​java/​org/​apache/​kafka/​controller/​ReplicationControlManagerInklessTest.java Add phase-based regression tests and update expected messages.
metadata/​src/​main/​java/​org/​apache/​kafka/​controller/​ReplicationControlManager.java Extend election and configuration guards through phase 1.
docs/​inkless/​CLASSIC_TO_DISKLESS_SWITCH.md Explain unclean election restrictions during the switch.
core/​src/​test/​java/​kafka/​server/​InklessTopicTypeSwitcherClusterTest.java Update the pending-switch error-message assertion.

Review effort: Balanced

@giuseppelillo
giuseppelillo force-pushed the giuseppelillo/kc-636-block-unclean-election-phase-1 branch from 48d4099 to fbfce3c Compare November 3, 2026 13:59
@giuseppelillo
giuseppelillo force-pushed the giuseppelillo/kc-589-gate-on-switch-phase branch from 68b4526 to 0a138bf Compare November 3, 2026 14:02
Base automatically changed from giuseppelillo/kc-589-gate-on-switch-phase to main November 3, 2026 14:44
…e 1 [KC-636]

In phase 1, the switch relies on one invariant: a replica is in the ISR only
if it holds the complete classic prefix. Every unclean-election guard keyed
on the pending state (`classicToDisklessStartOffset == -2`). After the seal
committed, an unclean election could make a replica outside the ISR the
leader and reset the ISR to that replica. Phase 1 lasts until the remote log
manager uploads the prefix, which can take hours for a topic that wasn't
tiered.

The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both
the pending state and phase 1:

- `validateDisklessSwitchInvariants` rejects
  `unclean.leader.election.enable=true` for the topic.
- `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`.
- Broker fencing, `AlterPartition`, and the periodic check for leaderless
  partitions skip the unclean election.
- A reassignment cancellation whose revert needs an unclean election fails
  with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an
  unclean leader either.

The election guards also cover a cluster-level
`unclean.leader.election.enable=true`, which the topic config check doesn't
see. Phase 2 keeps the born-diskless behavior.

The periodic check now reads the topic config before the switch state, so it
warns only when it would otherwise run an unclean election. The check runs
every five minutes, and phase 1 can last hours.

The error and log messages now say that the partition is switching to
diskless and its classic prefix is not in remote storage yet, because they
cover phase 1 too.

- New parameterized `ReplicationControlManagerInklessTest` cases cover
  broker fencing, the leaderless check, `ElectLeaders`, the config check on
  both AlterConfigs paths, and reassignment cancellation. Each case checks
  that phase 1 blocks the unclean election and phase 2 allows it, with the
  cluster default enabling unclean leader election. Without the fix, every
  phase 1 case fails.
- `AlterPartition` has no new case. Its validation requires the new ISR to
  include the current leader, so the election there never needs to be
  unclean.
- The pending-switch tests in `ReplicationControlManagerInklessTest` and
  `InklessTopicTypeSwitcherClusterTest` pass with the new message.
- `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass.
- `make vale` doesn't report any alerts.

Assisted-by: Claude Opus 5.5
@giuseppelillo
giuseppelillo force-pushed the giuseppelillo/kc-636-block-unclean-election-phase-1 branch from fbfce3c to 72b3c42 Compare November 3, 2026 15:39
@giuseppelillo
giuseppelillo marked this pull request as ready for review November 3, 2026 15:49
@viktorsomogyi
viktorsomogyi merged commit 5851744 into main Nov 4, 2026
11 checks passed
@viktorsomogyi
viktorsomogyi deleted the giuseppelillo/kc-636-block-unclean-election-phase-1 branch November 4, 2026 13:43
viktorsomogyi pushed a commit that referenced this pull request Nov 5, 2026
…phase 2 [KC-721] (#854)

`test_switched_leader_promotion_restores_stale_hw` fails on main since
#849 (KC-636). The test sets `unclean.leader.election.enable=true` right
after the seal commit, and the controller now rejects that config until
phase 2. The test cluster had no remote storage, so its switched partition
never left phase 1.

The old scenario can't happen in phase 1 anymore. There, the controller
elects only ISR members, and the old test needed unclean election to
promote a replica outside the ISR. In phase 2, remote storage holds the
prefix and the controller can elect any live replica. A replica with a
stale high watermark can still become leader there. This change reworks
the test to cover that case instead of removing it.

The test runs on a consolidation cluster with tiered storage on MinIO. It
switches with `remote.storage.enable` in the same alter as
`diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before
it injects the stale high watermark checkpoint. The test stops the
promoted replica last instead of isolating it with a network partition.
It's then the last ISR member of the test topic and of the internal
topics, so the controller elects it without unclean election when it
restarts alone. It also leads
`__remote_log_metadata`, which it needs to serve the prefix from remote
storage. The consumer must read every record across the seal.

On main, the follower consolidation fetcher repairs the stale high
watermark before the promotion. The leader-side repair in
`maybeReconcileSwitchedLeader` doesn't run in this test, and
`testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers
it.

The change also removes `_run_unclean_leader_election`, which has no other
callers.

## Testing

- `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker
  on main. The logs show phase 2 at seal 5000, a broker reload with high
  watermark 2500, a local log start offset of 5000, and 6000 records
  consumed.
- `make vale` passes.

Assisted-by: Claude Opus 5.5
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…e 1 [KC-636] (#849)

In phase 1, the switch relies on one invariant: a replica is in the ISR only
if it holds the complete classic prefix. Every unclean-election guard keyed
on the pending state (`classicToDisklessStartOffset == -2`). After the seal
committed, an unclean election could make a replica outside the ISR the
leader and reset the ISR to that replica. Phase 1 lasts until the remote log
manager uploads the prefix, which can take hours for a topic that wasn't
tiered.

The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both
the pending state and phase 1:

- `validateDisklessSwitchInvariants` rejects
  `unclean.leader.election.enable=true` for the topic.
- `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`.
- Broker fencing, `AlterPartition`, and the periodic check for leaderless
  partitions skip the unclean election.
- A reassignment cancellation whose revert needs an unclean election fails
  with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an
  unclean leader either.

The election guards also cover a cluster-level
`unclean.leader.election.enable=true`, which the topic config check doesn't
see. Phase 2 keeps the born-diskless behavior.

The periodic check now reads the topic config before the switch state, so it
warns only when it would otherwise run an unclean election. The check runs
every five minutes, and phase 1 can last hours.

The error and log messages now say that the partition is switching to
diskless and its classic prefix is not in remote storage yet, because they
cover phase 1 too.

- New parameterized `ReplicationControlManagerInklessTest` cases cover
  broker fencing, the leaderless check, `ElectLeaders`, the config check on
  both AlterConfigs paths, and reassignment cancellation. Each case checks
  that phase 1 blocks the unclean election and phase 2 allows it, with the
  cluster default enabling unclean leader election. Without the fix, every
  phase 1 case fails.
- `AlterPartition` has no new case. Its validation requires the new ISR to
  include the current leader, so the election there never needs to be
  unclean.
- The pending-switch tests in `ReplicationControlManagerInklessTest` and
  `InklessTopicTypeSwitcherClusterTest` pass with the new message.
- `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass.
- `make vale` doesn't report any alerts.

Assisted-by: Claude Opus 5.5
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…phase 2 [KC-721] (#854)

`test_switched_leader_promotion_restores_stale_hw` fails on main since
#849 (KC-636). The test sets `unclean.leader.election.enable=true` right
after the seal commit, and the controller now rejects that config until
phase 2. The test cluster had no remote storage, so its switched partition
never left phase 1.

The old scenario can't happen in phase 1 anymore. There, the controller
elects only ISR members, and the old test needed unclean election to
promote a replica outside the ISR. In phase 2, remote storage holds the
prefix and the controller can elect any live replica. A replica with a
stale high watermark can still become leader there. This change reworks
the test to cover that case instead of removing it.

The test runs on a consolidation cluster with tiered storage on MinIO. It
switches with `remote.storage.enable` in the same alter as
`diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before
it injects the stale high watermark checkpoint. The test stops the
promoted replica last instead of isolating it with a network partition.
It's then the last ISR member of the test topic and of the internal
topics, so the controller elects it without unclean election when it
restarts alone. It also leads
`__remote_log_metadata`, which it needs to serve the prefix from remote
storage. The consumer must read every record across the seal.

On main, the follower consolidation fetcher repairs the stale high
watermark before the promotion. The leader-side repair in
`maybeReconcileSwitchedLeader` doesn't run in this test, and
`testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers
it.

The change also removes `_run_unclean_leader_election`, which has no other
callers.

## Testing

- `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker
  on main. The logs show phase 2 at seal 5000, a broker reload with high
  watermark 2500, a local log start offset of 5000, and 6000 records
  consumed.
- `make vale` passes.

Assisted-by: Claude Opus 5.5
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…e 1 [KC-636] (#849)

In phase 1, the switch relies on one invariant: a replica is in the ISR only
if it holds the complete classic prefix. Every unclean-election guard keyed
on the pending state (`classicToDisklessStartOffset == -2`). After the seal
committed, an unclean election could make a replica outside the ISR the
leader and reset the ISR to that replica. Phase 1 lasts until the remote log
manager uploads the prefix, which can take hours for a topic that wasn't
tiered.

The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both
the pending state and phase 1:

- `validateDisklessSwitchInvariants` rejects
  `unclean.leader.election.enable=true` for the topic.
- `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`.
- Broker fencing, `AlterPartition`, and the periodic check for leaderless
  partitions skip the unclean election.
- A reassignment cancellation whose revert needs an unclean election fails
  with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an
  unclean leader either.

The election guards also cover a cluster-level
`unclean.leader.election.enable=true`, which the topic config check doesn't
see. Phase 2 keeps the born-diskless behavior.

The periodic check now reads the topic config before the switch state, so it
warns only when it would otherwise run an unclean election. The check runs
every five minutes, and phase 1 can last hours.

The error and log messages now say that the partition is switching to
diskless and its classic prefix is not in remote storage yet, because they
cover phase 1 too.

- New parameterized `ReplicationControlManagerInklessTest` cases cover
  broker fencing, the leaderless check, `ElectLeaders`, the config check on
  both AlterConfigs paths, and reassignment cancellation. Each case checks
  that phase 1 blocks the unclean election and phase 2 allows it, with the
  cluster default enabling unclean leader election. Without the fix, every
  phase 1 case fails.
- `AlterPartition` has no new case. Its validation requires the new ISR to
  include the current leader, so the election there never needs to be
  unclean.
- The pending-switch tests in `ReplicationControlManagerInklessTest` and
  `InklessTopicTypeSwitcherClusterTest` pass with the new message.
- `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass.
- `make vale` doesn't report any alerts.

Assisted-by: Claude Opus 5.5
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…phase 2 [KC-721] (#854)

`test_switched_leader_promotion_restores_stale_hw` fails on main since
#849 (KC-636). The test sets `unclean.leader.election.enable=true` right
after the seal commit, and the controller now rejects that config until
phase 2. The test cluster had no remote storage, so its switched partition
never left phase 1.

The old scenario can't happen in phase 1 anymore. There, the controller
elects only ISR members, and the old test needed unclean election to
promote a replica outside the ISR. In phase 2, remote storage holds the
prefix and the controller can elect any live replica. A replica with a
stale high watermark can still become leader there. This change reworks
the test to cover that case instead of removing it.

The test runs on a consolidation cluster with tiered storage on MinIO. It
switches with `remote.storage.enable` in the same alter as
`diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before
it injects the stale high watermark checkpoint. The test stops the
promoted replica last instead of isolating it with a network partition.
It's then the last ISR member of the test topic and of the internal
topics, so the controller elects it without unclean election when it
restarts alone. It also leads
`__remote_log_metadata`, which it needs to serve the prefix from remote
storage. The consumer must read every record across the seal.

On main, the follower consolidation fetcher repairs the stale high
watermark before the promotion. The leader-side repair in
`maybeReconcileSwitchedLeader` doesn't run in this test, and
`testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers
it.

The change also removes `_run_unclean_leader_election`, which has no other
callers.

## Testing

- `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker
  on main. The logs show phase 2 at seal 5000, a broker reload with high
  watermark 2500, a local log start offset of 5000, and 6000 records
  consumed.
- `make vale` passes.

Assisted-by: Claude Opus 5.5
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…e 1 [KC-636] (#849)

In phase 1, the switch relies on one invariant: a replica is in the ISR only
if it holds the complete classic prefix. Every unclean-election guard keyed
on the pending state (`classicToDisklessStartOffset == -2`). After the seal
committed, an unclean election could make a replica outside the ISR the
leader and reset the ISR to that replica. Phase 1 lasts until the remote log
manager uploads the prefix, which can take hours for a topic that wasn't
tiered.

The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both
the pending state and phase 1:

- `validateDisklessSwitchInvariants` rejects
  `unclean.leader.election.enable=true` for the topic.
- `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`.
- Broker fencing, `AlterPartition`, and the periodic check for leaderless
  partitions skip the unclean election.
- A reassignment cancellation whose revert needs an unclean election fails
  with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an
  unclean leader either.

The election guards also cover a cluster-level
`unclean.leader.election.enable=true`, which the topic config check doesn't
see. Phase 2 keeps the born-diskless behavior.

The periodic check now reads the topic config before the switch state, so it
warns only when it would otherwise run an unclean election. The check runs
every five minutes, and phase 1 can last hours.

The error and log messages now say that the partition is switching to
diskless and its classic prefix is not in remote storage yet, because they
cover phase 1 too.

- New parameterized `ReplicationControlManagerInklessTest` cases cover
  broker fencing, the leaderless check, `ElectLeaders`, the config check on
  both AlterConfigs paths, and reassignment cancellation. Each case checks
  that phase 1 blocks the unclean election and phase 2 allows it, with the
  cluster default enabling unclean leader election. Without the fix, every
  phase 1 case fails.
- `AlterPartition` has no new case. Its validation requires the new ISR to
  include the current leader, so the election there never needs to be
  unclean.
- The pending-switch tests in `ReplicationControlManagerInklessTest` and
  `InklessTopicTypeSwitcherClusterTest` pass with the new message.
- `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass.
- `make vale` doesn't report any alerts.

Assisted-by: Claude Opus 5.5
gqmelo pushed a commit that referenced this pull request Nov 5, 2026
…phase 2 [KC-721] (#854)

`test_switched_leader_promotion_restores_stale_hw` fails on main since
#849 (KC-636). The test sets `unclean.leader.election.enable=true` right
after the seal commit, and the controller now rejects that config until
phase 2. The test cluster had no remote storage, so its switched partition
never left phase 1.

The old scenario can't happen in phase 1 anymore. There, the controller
elects only ISR members, and the old test needed unclean election to
promote a replica outside the ISR. In phase 2, remote storage holds the
prefix and the controller can elect any live replica. A replica with a
stale high watermark can still become leader there. This change reworks
the test to cover that case instead of removing it.

The test runs on a consolidation cluster with tiered storage on MinIO. It
switches with `remote.storage.enable` in the same alter as
`diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before
it injects the stale high watermark checkpoint. The test stops the
promoted replica last instead of isolating it with a network partition.
It's then the last ISR member of the test topic and of the internal
topics, so the controller elects it without unclean election when it
restarts alone. It also leads
`__remote_log_metadata`, which it needs to serve the prefix from remote
storage. The consumer must read every record across the seal.

On main, the follower consolidation fetcher repairs the stale high
watermark before the promotion. The leader-side repair in
`maybeReconcileSwitchedLeader` doesn't run in this test, and
`testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers
it.

The change also removes `_run_unclean_leader_election`, which has no other
callers.

## Testing

- `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker
  on main. The logs show phase 2 at seal 5000, a broker reload with high
  watermark 2500, a local log start offset of 5000, and 6000 records
  consumed.
- `make vale` passes.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants