Skip to content

fix(inkless:switch): preserve classic prefix on partition increase during switch [KC-387] - #737

Merged
viktorsomogyi merged 2 commits into
mainfrom
svv/KC-387-switch-dataloss
Aug 12, 2026
Merged

viktorsomogyi merged 2 commits into
mainfrom
svv/KC-387-switch-dataloss

Conversation

@viktorsomogyi

@viktorsomogyi viktorsomogyi commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

A CreatePartitions committed during a classic-to-diskless switch created the switching partition's control-plane logs row at high_watermark=0. That made diskless LATEST report 0, so the consolidation fetcher truncated the classic
prefix and lost the pre-switch records.

The create path now snapshots each topic's ID and partition count from one metadata image before the mutation. A matching post-mutation ID creates only [priorCount, count). A missing or changed ID creates [0, count), with overlap
resolved by the control-plane conflict handling. Validation-only requests skip this work.

init_diskless_log_v1 now locks existing target rows in topic-partition order, replaces an empty placeholder with a positive seal, and replaces producer state when that update applies. The in-memory control plane mirrors this behavior.

SQL evolution from V11 to V23:

--- V11__Init_diskless_log.sql
+++ V23__Init_diskless_log_authoritative_seal.sql
@@
 DECLARE
     l_request RECORD;
     l_producer_state RECORD;
-    l_inserted BOOLEAN;
 BEGIN
+    -- Every caller takes existing row locks in the same order; responses still follow arg_requests order.
+    PERFORM 1
+    FROM logs
+    JOIN unnest(arg_requests) AS request
+        ON logs.topic_id = request.topic_id
+        AND logs.partition = request.partition
+    ORDER BY logs.topic_id, logs.partition
+    FOR UPDATE OF logs;
+
     FOR l_request IN
         SELECT *
         FROM unnest(arg_requests)
@@
         INSERT INTO logs (topic_id, partition, topic_name, log_start_offset, high_watermark, byte_size, diskless_start_offset)
         VALUES (l_request.topic_id, l_request.partition, l_request.topic_name,
                 l_request.log_start_offset, l_request.diskless_start_offset, 0, l_request.diskless_start_offset)
-        ON CONFLICT (topic_id, partition) DO NOTHING;
-
-        l_inserted := FOUND;
+        ON CONFLICT (topic_id, partition) DO UPDATE
+            SET log_start_offset      = EXCLUDED.log_start_offset,
+                high_watermark        = EXCLUDED.high_watermark,
+                diskless_start_offset = EXCLUDED.diskless_start_offset
+            WHERE logs.high_watermark = 0
+                AND logs.diskless_start_offset = 0
+                AND logs.byte_size = 0
+                -- A zero-offset re-init would not advance the placeholder, so leaving it alone keeps
+                -- re-initing an existing empty partition a no-op.
+                AND EXCLUDED.high_watermark > 0;

-        IF NOT l_inserted THEN
+        IF NOT FOUND THEN
             RETURN NEXT (l_request.topic_id, l_request.partition, 'already_initialized')::init_diskless_log_response_v1;
             CONTINUE;
         END IF;

+        -- The update path can reach a row an earlier init already wrote producer state for, and
+        -- producer_state is keyed on a BIGSERIAL row_id, so those rows would accumulate rather than be
+        -- replaced. The incoming snapshot is the authority for the partition.
+        DELETE FROM producer_state
+        WHERE topic_id = l_request.topic_id
+            AND partition = l_request.partition;
+
         FOR l_producer_state IN
             SELECT *
             FROM unnest(arg_producer_states)

Tests cover both control-plane backends, exact controller ranges, topic recreation, topics appearing during a request, validation-only behavior, and the end-to-end partition-increase switch path.

Co-authored-by: Cursor cursoragent@cursor.com

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents classic-prefix loss when partitions are added during a classic-to-diskless switch.

Changes:

  • Creates control-plane rows only for newly added partitions.
  • Makes the committed seal authoritative over empty placeholders.
  • Adds backend and end-to-end regression coverage; regenerates jOOQ schema metadata.

Reviewed changes

Copilot reviewed 7 out of 125 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
core/src/main/scala/kafka/server/ControllerApis.scala Passes the prior partition count to control-plane creation.
core/src/test/java/kafka/server/InklessConsolidatedDisklessTopicsTest.java Adds the end-to-end regression test.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/CreateTopicAndPartitionsRequest.java Adds the first-partition range bound.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/InMemoryControlPlane.java Adopts seals over empty placeholders.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/TopicsAndPartitionsCreateJob.java Inserts only the requested partition range.
storage/inkless/src/main/resources/db/migration/V23__Init_diskless_log_authoritative_seal.sql Adds guarded authoritative-seal upsert behavior.
storage/inkless/src/test/java/io/aiven/inkless/control_plane/AbstractControlPlaneTest.java Tests seal adoption on both backends.
storage/inkless/src/main/jooq/org/jooq/generated/UDTs.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/DefaultSchema.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/Domains.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/Indexes.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/Keys.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/Routines.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/Tables.java Regenerates schema-23 metadata.
storage/inkless/src/main/jooq/org/jooq/generated/enums/* Regenerates all changed enum bindings for schema 23.
storage/inkless/src/main/jooq/org/jooq/generated/routines/* Regenerates all changed routine bindings for schema 23.
storage/inkless/src/main/jooq/org/jooq/generated/tables/* Regenerates all changed table bindings for schema 23.
storage/inkless/src/main/jooq/org/jooq/generated/tables/records/* Regenerates all changed table-record bindings for schema 23.
storage/inkless/src/main/jooq/org/jooq/generated/udt/* Regenerates all changed UDT bindings for schema 23.
storage/inkless/src/main/jooq/org/jooq/generated/udt/paths/* Regenerates all changed UDT path bindings for schema 23.
storage/inkless/src/main/jooq/org/jooq/generated/udt/records/* Regenerates all changed UDT record bindings for schema 23.
Suppressed comments (1)

core/src/main/scala/kafka/server/ControllerApis.scala:1006

  • Add a focused controller test that captures the CreateTopicAndPartitionsRequest and verifies firstPartition is the pre-increase count. The new end-to-end test can still pass if this code sends 0, because the authoritative-seal upsert independently repairs that placeholder, and the existing control-plane test uses the three-argument constructor explicitly starting at 0. Thus the stated “only newly added partitions” behavior is currently unprotected.
              val firstPartition = priorPartitionCounts.getOrElse(req.name(), 0)
              Some(new CreateTopicAndPartitionsRequest(topicIdOrError.result(), req.name(), firstPartition, req.count()))

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@viktorsomogyi
viktorsomogyi force-pushed the svv/KC-387-switch-dataloss branch 3 times, most recently from bed7928 to f4d43f3 Compare August 11, 2026 08:56
…ring switch [KC-387]

A CreatePartitions committed during a classic-to-diskless switch created the
switching partition's control-plane logs row at high_watermark=0. That made
diskless LATEST report 0, so the consolidation fetcher truncated the classic
prefix and lost the pre-switch records.

The create path now snapshots each topic's ID and partition count from one
metadata image before the mutation. A matching post-mutation ID creates only
[priorCount, count). A missing or changed ID creates [0, count), with overlap
resolved by the control-plane conflict handling. Validation-only requests skip
this work.

init_diskless_log_v1 now locks existing target rows in topic-partition order,
replaces an empty placeholder with a positive seal, and replaces producer state
when that update applies. The in-memory control plane mirrors this behavior.

SQL evolution from V11 to V23:
```diff
--- V11__Init_diskless_log.sql
+++ V23__Init_diskless_log_authoritative_seal.sql
@@
 DECLARE
     l_request RECORD;
     l_producer_state RECORD;
-    l_inserted BOOLEAN;
 BEGIN
+    -- Every caller takes existing row locks in the same order; responses still follow arg_requests order.
+    PERFORM 1
+    FROM logs
+    JOIN unnest(arg_requests) AS request
+        ON logs.topic_id = request.topic_id
+        AND logs.partition = request.partition
+    ORDER BY logs.topic_id, logs.partition
+    FOR UPDATE OF logs;
+
     FOR l_request IN
         SELECT *
         FROM unnest(arg_requests)
@@
         INSERT INTO logs (topic_id, partition, topic_name, log_start_offset, high_watermark, byte_size, diskless_start_offset)
         VALUES (l_request.topic_id, l_request.partition, l_request.topic_name,
                 l_request.log_start_offset, l_request.diskless_start_offset, 0, l_request.diskless_start_offset)
-        ON CONFLICT (topic_id, partition) DO NOTHING;
-
-        l_inserted := FOUND;
+        ON CONFLICT (topic_id, partition) DO UPDATE
+            SET log_start_offset      = EXCLUDED.log_start_offset,
+                high_watermark        = EXCLUDED.high_watermark,
+                diskless_start_offset = EXCLUDED.diskless_start_offset
+            WHERE logs.high_watermark = 0
+                AND logs.diskless_start_offset = 0
+                AND logs.byte_size = 0
+                -- A zero-offset re-init would not advance the placeholder, so leaving it alone keeps
+                -- re-initing an existing empty partition a no-op.
+                AND EXCLUDED.high_watermark > 0;

-        IF NOT l_inserted THEN
+        IF NOT FOUND THEN
             RETURN NEXT (l_request.topic_id, l_request.partition, 'already_initialized')::init_diskless_log_response_v1;
             CONTINUE;
         END IF;

+        -- The update path can reach a row an earlier init already wrote producer state for, and
+        -- producer_state is keyed on a BIGSERIAL row_id, so those rows would accumulate rather than be
+        -- replaced. The incoming snapshot is the authority for the partition.
+        DELETE FROM producer_state
+        WHERE topic_id = l_request.topic_id
+            AND partition = l_request.partition;
+
         FOR l_producer_state IN
             SELECT *
             FROM unnest(arg_producer_states)
```
Tests cover both control-plane backends, exact controller ranges, topic
recreation, topics appearing during a request, validation-only behavior, and
the end-to-end partition-increase switch path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@viktorsomogyi
viktorsomogyi marked this pull request as ready for review August 11, 2026 09:35
@viktorsomogyi
viktorsomogyi force-pushed the svv/KC-387-switch-dataloss branch from f4d43f3 to a8fd625 Compare August 11, 2026 09:56

@jeqo jeqo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall. Few minor findings

@@ -0,0 +1,52 @@
/*
* Inkless
* Copyright (C) 2024 - 2025 Aiven OY

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
* Copyright (C) 2024 - 2025 Aiven OY
* Copyright (C) 2024 - 2026 Aiven OY

Comment on lines -994 to +1016
Some(new CreateTopicAndPartitionsRequest(topicIdOrError.result(), req.name(), req.count()))
val topicId = topicIdOrError.result()
// The cached range is only usable when it belongs to the topic the controller just mutated.
// Otherwise create the full range and rely on init_diskless_log_v1 to resolve overlap (KC-387).
val firstPartition = priorTopicStates.get(topicName) match {
case Some(state) if state.topicId == topicId => math.min(state.numPartitions, req.count())
case _ => 0
}
Some(new CreateTopicAndPartitionsRequest(topicId, topicName, firstPartition, req.count()))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Claude found an interesting side effect of this: As Controller commits the metadata mutation before Control Plane call: If Control Plane fails to write, nothing reconciles this gap, and these diskless partition will not have rows and fail with UNKNOWN_TOPIC_OR_PARTITION. Also, repair_diskless_log_v1 will not be able to help here.

Before this PR, if a larger increase on the same topic would be attempted, it would be healed by inserting [0, count). With this PR, it would insert [N, count) missing the failed partitions in between.

A same-count retry never reached the control plane either way, then nothing was really depending on this so I don't think we need to fix this here.
Let's have a follow up to reconcile this. Maybe let's also mention this on the commit?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good point, will create a follow-up fix. I was thinking in some kind of reconciliation loop: take the controller metadata as the ground truth, then query the control plane for log row and reconcile where needed (with retry and backoff).


@AfterEach
def tearDown(): Unit = {
metadataCache.setImage(MetadataImage.EMPTY)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Another non-obvious Claude catch: this per-instance val with no @TestInstance(PER_CLASS), so JUnit constructs a fresh KRaftMetadataCache per test, and a fresh one already holds MetadataImage.EMPTY -- so this setImage is inert (same on :1096).

Tested locally and seems to be right.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the redundant setImage instead of adding PER_CLASS as removing seems to be a more narrow change with less possible side effects.

@viktorsomogyi
viktorsomogyi merged commit e006f00 into main Aug 12, 2026
6 checks passed
@viktorsomogyi
viktorsomogyi deleted the svv/KC-387-switch-dataloss branch August 12, 2026 09:23
@viktorsomogyi
viktorsomogyi restored the svv/KC-387-switch-dataloss branch August 12, 2026 09:37
jeqo pushed a commit that referenced this pull request Aug 19, 2026
…ring switch [KC-387] (#737)

A CreatePartitions committed during a classic-to-diskless switch created the switching partition's control-plane logs row at high_watermark=0. That made diskless LATEST report 0, so the consolidation fetcher truncated the classic
prefix and lost the pre-switch records.

The create path now snapshots each topic's ID and partition count from one metadata image before the mutation. A matching post-mutation ID creates only [priorCount, count). A missing or changed ID creates [0, count), with overlap
resolved by the control-plane conflict handling. Validation-only requests skip this work.

init_diskless_log_v1 now locks existing target rows in topic-partition order, replaces an empty placeholder with a positive seal, and replaces producer state when that update applies. The in-memory control plane mirrors this behavior.
---------

Co-authored-by: Cursor <cursoragent@cursor.com>
jeqo pushed a commit that referenced this pull request Aug 19, 2026
…ring switch [KC-387] (#737)

A CreatePartitions committed during a classic-to-diskless switch created the switching partition's control-plane logs row at high_watermark=0. That made diskless LATEST report 0, so the consolidation fetcher truncated the classic
prefix and lost the pre-switch records.

The create path now snapshots each topic's ID and partition count from one metadata image before the mutation. A matching post-mutation ID creates only [priorCount, count). A missing or changed ID creates [0, count), with overlap
resolved by the control-plane conflict handling. Validation-only requests skip this work.

init_diskless_log_v1 now locks existing target rows in topic-partition order, replaces an empty placeholder with a positive seal, and replaces producer state when that update applies. The in-memory control plane mirrors this behavior.
---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants