Repository navigation
fix(inkless:controller): block unclean leader election in switch phase 1 [KC-636] - #849
Conversation
|
Stacked on #840. Until #840 merges, this PR also shows its three KC-589 commits. Only the last commit, |
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the changes extending election safeguards through phase 1 of the classic-to-diskless switch.
No open issues 🎉
What changed in this PR
This PR protects the classic data prefix during a classic-to-diskless switch by extending unclean leader election guards through phase 1, until the prefix reaches remote storage. Phase 2 retains the existing born-diskless behavior.
Changes:
- Block unclean elections and unsafe reassignment cancellations while the classic prefix remains local.
- Add parameterized tests for phase 1 restrictions and phase 2 behavior, and update error-message assertions.
- Document the election restrictions and their behavior when the cluster default enables unclean elections.
| File | Description |
|---|---|
metadata/src/test/java/org/apache/kafka/controller/ReplicationControlManagerInklessTest.java |
Add phase-based regression tests and update expected messages. |
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java |
Extend election and configuration guards through phase 1. |
docs/inkless/CLASSIC_TO_DISKLESS_SWITCH.md |
Explain unclean election restrictions during the switch. |
core/src/test/java/kafka/server/InklessTopicTypeSwitcherClusterTest.java |
Update the pending-switch error-message assertion. |
Review effort: Balanced
48d4099 to
fbfce3c
Compare
68b4526 to
0a138bf
Compare
…e 1 [KC-636] In phase 1, the switch relies on one invariant: a replica is in the ISR only if it holds the complete classic prefix. Every unclean-election guard keyed on the pending state (`classicToDisklessStartOffset == -2`). After the seal committed, an unclean election could make a replica outside the ISR the leader and reset the ISR to that replica. Phase 1 lasts until the remote log manager uploads the prefix, which can take hours for a topic that wasn't tiered. The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both the pending state and phase 1: - `validateDisklessSwitchInvariants` rejects `unclean.leader.election.enable=true` for the topic. - `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`. - Broker fencing, `AlterPartition`, and the periodic check for leaderless partitions skip the unclean election. - A reassignment cancellation whose revert needs an unclean election fails with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an unclean leader either. The election guards also cover a cluster-level `unclean.leader.election.enable=true`, which the topic config check doesn't see. Phase 2 keeps the born-diskless behavior. The periodic check now reads the topic config before the switch state, so it warns only when it would otherwise run an unclean election. The check runs every five minutes, and phase 1 can last hours. The error and log messages now say that the partition is switching to diskless and its classic prefix is not in remote storage yet, because they cover phase 1 too. - New parameterized `ReplicationControlManagerInklessTest` cases cover broker fencing, the leaderless check, `ElectLeaders`, the config check on both AlterConfigs paths, and reassignment cancellation. Each case checks that phase 1 blocks the unclean election and phase 2 allows it, with the cluster default enabling unclean leader election. Without the fix, every phase 1 case fails. - `AlterPartition` has no new case. Its validation requires the new ISR to include the current leader, so the election there never needs to be unclean. - The pending-switch tests in `ReplicationControlManagerInklessTest` and `InklessTopicTypeSwitcherClusterTest` pass with the new message. - `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass. - `make vale` doesn't report any alerts. Assisted-by: Claude Opus 5.5
fbfce3c to
72b3c42
Compare
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
…e 1 [KC-636] (#849) In phase 1, the switch relies on one invariant: a replica is in the ISR only if it holds the complete classic prefix. Every unclean-election guard keyed on the pending state (`classicToDisklessStartOffset == -2`). After the seal committed, an unclean election could make a replica outside the ISR the leader and reset the ISR to that replica. Phase 1 lasts until the remote log manager uploads the prefix, which can take hours for a topic that wasn't tiered. The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both the pending state and phase 1: - `validateDisklessSwitchInvariants` rejects `unclean.leader.election.enable=true` for the topic. - `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`. - Broker fencing, `AlterPartition`, and the periodic check for leaderless partitions skip the unclean election. - A reassignment cancellation whose revert needs an unclean election fails with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an unclean leader either. The election guards also cover a cluster-level `unclean.leader.election.enable=true`, which the topic config check doesn't see. Phase 2 keeps the born-diskless behavior. The periodic check now reads the topic config before the switch state, so it warns only when it would otherwise run an unclean election. The check runs every five minutes, and phase 1 can last hours. The error and log messages now say that the partition is switching to diskless and its classic prefix is not in remote storage yet, because they cover phase 1 too. - New parameterized `ReplicationControlManagerInklessTest` cases cover broker fencing, the leaderless check, `ElectLeaders`, the config check on both AlterConfigs paths, and reassignment cancellation. Each case checks that phase 1 blocks the unclean election and phase 2 allows it, with the cluster default enabling unclean leader election. Without the fix, every phase 1 case fails. - `AlterPartition` has no new case. Its validation requires the new ISR to include the current leader, so the election there never needs to be unclean. - The pending-switch tests in `ReplicationControlManagerInklessTest` and `InklessTopicTypeSwitcherClusterTest` pass with the new message. - `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass. - `make vale` doesn't report any alerts. Assisted-by: Claude Opus 5.5
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
…e 1 [KC-636] (#849) In phase 1, the switch relies on one invariant: a replica is in the ISR only if it holds the complete classic prefix. Every unclean-election guard keyed on the pending state (`classicToDisklessStartOffset == -2`). After the seal committed, an unclean election could make a replica outside the ISR the leader and reset the ISR to that replica. Phase 1 lasts until the remote log manager uploads the prefix, which can take hours for a topic that wasn't tiered. The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both the pending state and phase 1: - `validateDisklessSwitchInvariants` rejects `unclean.leader.election.enable=true` for the topic. - `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`. - Broker fencing, `AlterPartition`, and the periodic check for leaderless partitions skip the unclean election. - A reassignment cancellation whose revert needs an unclean election fails with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an unclean leader either. The election guards also cover a cluster-level `unclean.leader.election.enable=true`, which the topic config check doesn't see. Phase 2 keeps the born-diskless behavior. The periodic check now reads the topic config before the switch state, so it warns only when it would otherwise run an unclean election. The check runs every five minutes, and phase 1 can last hours. The error and log messages now say that the partition is switching to diskless and its classic prefix is not in remote storage yet, because they cover phase 1 too. - New parameterized `ReplicationControlManagerInklessTest` cases cover broker fencing, the leaderless check, `ElectLeaders`, the config check on both AlterConfigs paths, and reassignment cancellation. Each case checks that phase 1 blocks the unclean election and phase 2 allows it, with the cluster default enabling unclean leader election. Without the fix, every phase 1 case fails. - `AlterPartition` has no new case. Its validation requires the new ISR to include the current leader, so the election there never needs to be unclean. - The pending-switch tests in `ReplicationControlManagerInklessTest` and `InklessTopicTypeSwitcherClusterTest` pass with the new message. - `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass. - `make vale` doesn't report any alerts. Assisted-by: Claude Opus 5.5
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
…e 1 [KC-636] (#849) In phase 1, the switch relies on one invariant: a replica is in the ISR only if it holds the complete classic prefix. Every unclean-election guard keyed on the pending state (`classicToDisklessStartOffset == -2`). After the seal committed, an unclean election could make a replica outside the ISR the leader and reset the ISR to that replica. Phase 1 lasts until the remote log manager uploads the prefix, which can take hours for a topic that wasn't tiered. The guards now key on `isClassicPrefixOnlyOnLocalDisks`, which covers both the pending state and phase 1: - `validateDisklessSwitchInvariants` rejects `unclean.leader.election.enable=true` for the topic. - `electLeader` rejects an `UNCLEAN` election with `INVALID_REQUEST`. - Broker fencing, `AlterPartition`, and the periodic check for leaderless partitions skip the unclean election. - A reassignment cancellation whose revert needs an unclean election fails with `INVALID_REPLICA_ASSIGNMENT`, and the cancellation doesn't elect an unclean leader either. The election guards also cover a cluster-level `unclean.leader.election.enable=true`, which the topic config check doesn't see. Phase 2 keeps the born-diskless behavior. The periodic check now reads the topic config before the switch state, so it warns only when it would otherwise run an unclean election. The check runs every five minutes, and phase 1 can last hours. The error and log messages now say that the partition is switching to diskless and its classic prefix is not in remote storage yet, because they cover phase 1 too. - New parameterized `ReplicationControlManagerInklessTest` cases cover broker fencing, the leaderless check, `ElectLeaders`, the config check on both AlterConfigs paths, and reassignment cancellation. Each case checks that phase 1 blocks the unclean election and phase 2 allows it, with the cluster default enabling unclean leader election. Without the fix, every phase 1 case fails. - `AlterPartition` has no new case. Its validation requires the new ISR to include the current leader, so the election there never needs to be unclean. - The pending-switch tests in `ReplicationControlManagerInklessTest` and `InklessTopicTypeSwitcherClusterTest` pass with the new message. - `ReplicationControlManagerTest` and `PartitionChangeBuilderTest` pass. - `make vale` doesn't report any alerts. Assisted-by: Claude Opus 5.5
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
In phase 1, the switch relies on one invariant: a replica is in the ISR only if it holds the complete classic prefix. Every unclean-election guard keyed on the pending state (
classicToDisklessStartOffset == -2). After the seal committed, an unclean election could make a replica outside the ISR the leader and reset the ISR to that replica. Phase 1 lasts until the remote log manager uploads the prefix, which can take hours for a topic that wasn't tiered.Block unclean leader election until phase 2
The guards now key on
isSwitchedWithLocalClassicPrefix, which covers both the pending state and phase 1:validateDisklessSwitchInvariantsrejectsunclean.leader.election.enable=truefor the topic.electLeaderrejects anUNCLEANelection withINVALID_REQUEST.AlterPartition, and the periodic check for leaderless partitions skip the unclean election.INVALID_REPLICA_ASSIGNMENT, and the cancellation doesn't elect an unclean leader either.The election guards also cover a cluster-level
unclean.leader.election.enable=true, which the topic config check doesn't see. Phase 2 keeps the born-diskless behavior.The periodic check now reads the topic config before the switch state, so it warns only when it would otherwise run an unclean election. The check runs every five minutes, and phase 1 can last hours.
The error and log messages now say that the partition is switching to diskless and its classic prefix is not in remote storage yet, because they cover phase 1 too.
CLASSIC_TO_DISKLESS_SWITCH.mdgets a row in the controller phase table and an "Unclean leader election" section.Testing
ReplicationControlManagerInklessTestcases cover broker fencing, the leaderless check,ElectLeaders, the config check on both AlterConfigs paths, and reassignment cancellation. Each case checks that phase 1 blocks the unclean election and phase 2 allows it, with the cluster default enabling unclean leader election. Without the fix, every phase 1 case fails.AlterPartitionhas no new case. Its validation requires the new ISR to include the current leader, so the election there never needs to be unclean.ReplicationControlManagerInklessTestandInklessTopicTypeSwitcherClusterTestpass with the new message.ReplicationControlManagerTestandPartitionChangeBuilderTestpass.make valedoesn't report any alerts.