Repository navigation
Conversation
clusterProcessPacket drops a stale packet from a node it knows as a
primary when the packet announces it as a replica with an older config
epoch. A sender already known as a replica gets no such check when it
announces a different primary, so a delayed packet can roll its primary
back to one that predates a failover we have already processed.
This crashed a node in unit/cluster/slave-selection ("New Master down
consecutively") under valgrind. After failovers R3 -> R9 (epoch 22) and
R9 -> R6 (epoch 23), the node had R3 as a replica of R9 and R9 as a
replica of R6. A late packet from R9 still claiming to replicate R3 at
epoch 21 was applied, setting R9 -> R3 and forming a replicaof cycle. The
following updateShardId() call then hit the cycle assertion in
clusterNodeGetPrimary():
cluster_legacy.c:8444 'primary->replicaof == ((void *)0)' is not true
A replica only changes primary within its shard after a failover, so the
claimed primary's config epoch cannot be older than what we know. Apply
the existing staleness check to that case too and drop such packets.
Moves to a primary in a different shard are not affected.
Signed-off-by: Jim Brunner <brunnerj@amazon.com>
| * packet predates a failover we have already processed (e.g. it was | ||
| * delayed on another link). Applying it could point the replica back | ||
| * at a node we know replicates from it, forming a replicaof cycle. */ | ||
| if (sender_last_reported_as_replica && sender->replicaof && |
There was a problem hiding this comment.
I think this fix makes sense but maybe it's incomplete because the epoch check doesn't cover a fresh packet that closes the loop.
The epoch check answers "is this packet old?". It doesn't answer "would this change create a loop?". A packet can be new and still create one, because our own entry for another node can be out of date.
Starting point (same state as in the PR description). R6 is primary at epoch 23. We have heard from R9 but not yet from R3, so our entry for R3 is out of date.
(out of date) (current)
R3 ────────────────▶ R9 ────────────────▶ R6 [primary, epoch 23]
Event. R6 dies and R3 wins the election at epoch 24. Two packets reach us in this order:
1st: R9 "I replicate from R3, epoch 24"
2nd: R3 "I am primary, epoch 24"
With the epoch check only
Epoch check: is 24 older than what I know (23)? No → accept
nodeEpoch(R3) doesn't help either: R3 is still a replica of R9 in our view, so it returns R9's stored epoch (22).
We replace R9's entry, but the out-of-date entry for R3 is still there:
R3 ────────────────▶ R9 out of date
R3 ◀──────────────── R9 new, and true
Following the arrows from R9: R9 → R3 → R9 → R3 → ... never ends
- With debug asserts on, this hits the same assertion in
clusterNodeGetPrimary(). - With debug asserts off,
clusterNodeGetPrimary()only stops if the walk returns to the node it started from. Any other replica pointing into the loop makes it spin forever, reached viaupdateShardId()on that replica's next ping.
We should add a loop check after the epoch check
Before changing replicaof, walk from the claimed primary and see if we reach the sender.
Proposed change: R9 ──▶ R3
Walk from R3: R3 ──▶ R9 reached the sender, so this would close a loop
Decision: ignore the change for now
Our view is unchanged:
R3 ────────────────▶ R9 ────────────────▶ R6
R3's packet arrives ("I am primary, epoch 24"). clusterSetNodeAsPrimary(R3) clears R3's out-of-date entry:
R3 [primary, epoch 24]
R9 ────────────────▶ R6
R9's next ping repeats "I replicate from R3, epoch 24":
Proposed change: R9 ──▶ R3
Walk from R3: R3 ──▶ (end) no loop
Decision: apply
Final view, matching reality:
R9 ────────────────▶ R3 [primary, epoch 24]
clusterProcessPacket drops a stale packet from a node it knows as a primary when the packet announces it as a replica with an older config epoch. A sender already known as a replica gets no such check when it announces a different primary, so a delayed packet can roll its primary back to one that predates a failover we have already processed.
This crashed a node in unit/cluster/slave-selection ("New Master down consecutively") under valgrind. After failovers R3 -> R9 (epoch 22) and R9 -> R6 (epoch 23), the node had R3 as a replica of R9 and R9 as a replica of R6. A late packet from R9 still claiming to replicate R3 at epoch 21 was applied, setting R9 -> R3 and forming a replicaof cycle. The following updateShardId() call then hit the cycle assertion in clusterNodeGetPrimary():
A replica only changes primary within its shard after a failover, so the claimed primary's config epoch cannot be older than what we know. Apply the existing staleness check to that case too and drop such packets. Moves to a primary in a different shard are not affected.