Skip to content

Ignore stale primary changes from replicas within a shard - #23

Open
JimB123 wants to merge 1 commit into
unstablefrom
fix-stale-replica-primary-change
Open

JimB123 wants to merge 1 commit into
unstablefrom
fix-stale-replica-primary-change

Conversation

@JimB123

@JimB123 JimB123 commented Nov 12, 2026

Copy link
Copy Markdown
Owner

clusterProcessPacket drops a stale packet from a node it knows as a primary when the packet announces it as a replica with an older config epoch. A sender already known as a replica gets no such check when it announces a different primary, so a delayed packet can roll its primary back to one that predates a failover we have already processed.

This crashed a node in unit/cluster/slave-selection ("New Master down consecutively") under valgrind. After failovers R3 -> R9 (epoch 22) and R9 -> R6 (epoch 23), the node had R3 as a replica of R9 and R9 as a replica of R6. A late packet from R9 still claiming to replicate R3 at epoch 21 was applied, setting R9 -> R3 and forming a replicaof cycle. The following updateShardId() call then hit the cycle assertion in clusterNodeGetPrimary():

cluster_legacy.c:8444 'primary->replicaof == ((void *)0)' is not true

A replica only changes primary within its shard after a failover, so the claimed primary's config epoch cannot be older than what we know. Apply the existing staleness check to that case too and drop such packets. Moves to a primary in a different shard are not affected.

clusterProcessPacket drops a stale packet from a node it knows as a
primary when the packet announces it as a replica with an older config
epoch. A sender already known as a replica gets no such check when it
announces a different primary, so a delayed packet can roll its primary
back to one that predates a failover we have already processed.

This crashed a node in unit/cluster/slave-selection ("New Master down
consecutively") under valgrind. After failovers R3 -> R9 (epoch 22) and
R9 -> R6 (epoch 23), the node had R3 as a replica of R9 and R9 as a
replica of R6. A late packet from R9 still claiming to replicate R3 at
epoch 21 was applied, setting R9 -> R3 and forming a replicaof cycle. The
following updateShardId() call then hit the cycle assertion in
clusterNodeGetPrimary():

    cluster_legacy.c:8444 'primary->replicaof == ((void *)0)' is not true

A replica only changes primary within its shard after a failover, so the
claimed primary's config epoch cannot be older than what we know. Apply
the existing staleness check to that case too and drop such packets.
Moves to a primary in a different shard are not affected.

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
Comment thread src/cluster_legacy.c
* packet predates a failover we have already processed (e.g. it was
* delayed on another link). Applying it could point the replica back
* at a node we know replicates from it, forming a replicaof cycle. */
if (sender_last_reported_as_replica && sender->replicaof &&

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this fix makes sense but maybe it's incomplete because the epoch check doesn't cover a fresh packet that closes the loop.

The epoch check answers "is this packet old?". It doesn't answer "would this change create a loop?". A packet can be new and still create one, because our own entry for another node can be out of date.

Starting point (same state as in the PR description). R6 is primary at epoch 23. We have heard from R9 but not yet from R3, so our entry for R3 is out of date.

        (out of date)            (current)
   R3 ────────────────▶ R9 ────────────────▶ R6   [primary, epoch 23]

Event. R6 dies and R3 wins the election at epoch 24. Two packets reach us in this order:

   1st:  R9  "I replicate from R3, epoch 24"
   2nd:  R3  "I am primary, epoch 24"

With the epoch check only

   Epoch check:   is 24 older than what I know (23)?   No  →  accept

nodeEpoch(R3) doesn't help either: R3 is still a replica of R9 in our view, so it returns R9's stored epoch (22).

We replace R9's entry, but the out-of-date entry for R3 is still there:

   R3 ────────────────▶ R9      out of date
   R3 ◀──────────────── R9      new, and true

   Following the arrows from R9:   R9 → R3 → R9 → R3 → ...   never ends
  • With debug asserts on, this hits the same assertion in clusterNodeGetPrimary().
  • With debug asserts off, clusterNodeGetPrimary() only stops if the walk returns to the node it started from. Any other replica pointing into the loop makes it spin forever, reached via updateShardId() on that replica's next ping.

We should add a loop check after the epoch check

Before changing replicaof, walk from the claimed primary and see if we reach the sender.

   Proposed change:   R9 ──▶ R3

   Walk from R3:      R3 ──▶ R9      reached the sender, so this would close a loop
   Decision:          ignore the change for now

Our view is unchanged:

   R3 ────────────────▶ R9 ────────────────▶ R6

R3's packet arrives ("I am primary, epoch 24"). clusterSetNodeAsPrimary(R3) clears R3's out-of-date entry:

   R3   [primary, epoch 24]

   R9 ────────────────▶ R6

R9's next ping repeats "I replicate from R3, epoch 24":

   Proposed change:   R9 ──▶ R3

   Walk from R3:      R3 ──▶ (end)   no loop
   Decision:          apply

Final view, matching reality:

   R9 ────────────────▶ R3   [primary, epoch 24]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants