Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ class ConsolidationReconciler(replicaManager: ReplicaManager,
ConsolidationStartState.Failed(new ReconciliationException(
s"Diskless topic $tp has remote storage disabled but was switched from classic " +
s"(violates diskless.enable implies remote.storage.enable); consolidation cannot start " +
s"(see DisklessWithoutRemoteStorageCount)"))
s"(see FailedPartitionsCount)"))
case seal if seal >= 0 =>
val log = partition.localLogOrException
if (log.logEndOffset < seal) {
Expand Down
8 changes: 5 additions & 3 deletions docs/inkless/DISKLESS_CONSOLIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,15 +240,17 @@ flowchart TD
seal -- "NO_CLASSIC_TO_DISKLESS_START_OFFSET\n(born-diskless)" --> ready[Ready: arm at initialFetchOffset]
seal -- "SWITCH_PENDING" --> retry[Retry: classic switch in progress]
seal -- "seal >= 0" --> remote{remote.storage.enable?}
remote -- "off" --> failed[Failed: invariant violation\nsee DisklessWithoutRemoteStorageCount]
remote -- "off" --> failed[Failed: invariant violation\nsee FailedPartitionsCount]
remote -- "on" --> leo{local LEO vs seal?}
leo -- "LEO < seal, leader" --> rebuild[Ready: arm at LEO\nrebuild classic prefix from remote\nvia OFFSET_MOVED_TO_TIERED_STORAGE]
leo -- "LEO < seal, follower" --> retry2[Retry: wait for classic catch-up fetcher]
leo -- "LEO >= seal" --> resume[Ready: arm at LEO\nprune floor at least max of seal and logStartOffset]
failed --> fenced[Partition stays online for reads/writes\nFailed flag clears on next leader-epoch change]
```

A `Failed` partition stays online and remains readable and writable. Consolidation doesn't run, so the local log doesn't grow unbounded into an untiered diskless log. `FailedPartitionsCount` and the controller-side `DisklessWithoutRemoteStorageCount` metric surface the state to operators. If the failure is an invariant violation, set `remote.storage.enable=true`. The controller co-commits a leader-epoch bump so reconciliation runs again.
A `Failed` partition stays online and remains readable and writable. Consolidation doesn't run, so the local log doesn't grow unbounded into an untiered diskless log. `FailedPartitionsCount` surfaces this per-partition state. If the failure is an invariant violation, set `remote.storage.enable=true`. The controller co-commits a leader-epoch bump so reconciliation runs again.

The controller-side `DisklessWithoutRemoteStorageCount` separately inventories every diskless topic where `remote.storage.enable` is `false` or unset, including born-diskless topics that never enter the `Failed` state. Those topics don't consolidate until an operator sets `remote.storage.enable=true` while consolidation is on.

### Diskless leader epoch for truncation

Expand Down Expand Up @@ -362,7 +364,7 @@ The broker registers these under the `io.aiven.inkless.consolidation` group. The
| `io.aiven.inkless.consolidation:type=ConsolidationFetchMetrics` | `RecentDataRequestRate` / `LaggingConsumerRequestRate` | Hot-path (cache-hit) vs cold-path (object-storage) consolidation fetch rates. |
| `io.aiven.inkless.delete:type=CrossTierLogStartReporter` | `PartitionsReported` / `ReportErrors` / `PendingPartitions` | Cross-tier log start offset reporting to the control plane. |
| `io.aiven.inkless.cache:type=CrossTierLogStartCache` | `CacheHits` / `CacheMisses` / `CacheSize` | Cross-tier earliest-offset cache. |
| controller | `DisklessWithoutRemoteStorageCount` | Switched topics with remote storage off (invariant violation; surfaces `Failed` reconciler state). |
| controller | `DisklessWithoutRemoteStorageCount` | Diskless topics where `remote.storage.enable` is `false` or unset, including born-diskless topics that never enter the `Failed` reconciler state. |

## Compatibility

Expand Down
2 changes: 1 addition & 1 deletion docs/inkless/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ New JMX metrics are available on the active controller (`kafka.controller:type=K
| `DisklessTopicCount` | Total number of diskless topics |
| `DisklessPartitionCount` | Total number of partitions in diskless topics |
| `DisklessOfflinePartitionCount` | Diskless partitions without a leader (leader=-1), typically because no eligible assigned replica is available to lead |
| `DisklessWithoutRemoteStorageCount` | Diskless topics with `remote.storage.enable` explicitly set to `false` — a misconfiguration. Useful during the transition to remote storage consolidation to detect legacy topics that need remediation. Only counts explicit `false`, not absent config. Refreshed on metadata snapshot (default: every hour or 20 MB of metadata records) |
| `DisklessWithoutRemoteStorageCount` | Diskless topics where `remote.storage.enable` is `false` or unset. These topics don't consolidate until an operator sets `remote.storage.enable=true` while consolidation is on. |

These metrics are tracked separately from classic partition metrics to avoid false alerts — diskless topics may show offline replicas in KRaft metadata while remaining fully available via the metadata transformer.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,10 @@ public void setDisklessWithoutRemoteStorageCount(int count) {
this.disklessWithoutRemoteStorageCount.set(count);
}

public void addToDisklessWithoutRemoteStorageCount(int delta) {
this.disklessWithoutRemoteStorageCount.addAndGet(delta);
}

public int disklessWithoutRemoteStorageCount() {
return this.disklessWithoutRemoteStorageCount.get();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,11 @@
import org.apache.kafka.metadata.PartitionRegistration;
import org.apache.kafka.server.fault.FaultHandler;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.ArrayList;
import java.util.List;
import java.util.HashSet;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;


/**
Expand All @@ -50,8 +47,6 @@
*
*/
public class ControllerMetadataMetricsPublisher implements MetadataPublisher {
private static final Logger log = LoggerFactory.getLogger(ControllerMetadataMetricsPublisher.class);

private final ControllerMetadataMetrics metrics;
private final FaultHandler faultHandler;
private MetadataImage prevImage = MetadataImage.EMPTY;
Expand Down Expand Up @@ -101,6 +96,7 @@ public void onMetadataUpdate(

private void publishDelta(MetadataDelta delta, MetadataImage newImage) {
ControllerMetricsChanges changes = new ControllerMetricsChanges();
Set<String> changedTopicNames = new HashSet<>();
if (delta.clusterDelta() != null) {
for (Entry<Integer, Optional<BrokerRegistration>> entry :
delta.clusterDelta().changedBrokers().entrySet()) {
Expand All @@ -118,10 +114,12 @@ private void publishDelta(MetadataDelta delta, MetadataImage newImage) {
throw new RuntimeException("Unable to find deleted topic id " + topicId +
" in previous topics image.");
}
changedTopicNames.add(prevTopic.name());
// For deleted topics, check isDiskless from prevImage since config is already removed from newImage
changes.handleDeletedTopic(prevTopic, isDisklessTopic(prevImage.configs(), prevTopic.name()));
}
for (Entry<Uuid, TopicDelta> entry : delta.topicsDelta().changedTopics().entrySet()) {
changedTopicNames.add(entry.getValue().name());
// Use prevImage configs intentionally: diskless.enable transitions
// are handled by the configsDelta loop below, not here.
changes.handleTopicChange(
Expand All @@ -135,6 +133,7 @@ private void publishDelta(MetadataDelta delta, MetadataImage newImage) {
if (delta.configsDelta() != null) {
for (ConfigResource resource : delta.configsDelta().changes().keySet()) {
if (resource.type() != ConfigResource.Type.TOPIC) continue;
changedTopicNames.add(resource.name());
boolean wasDiskless = isDisklessTopic(prevImage.configs(), resource.name());
boolean isDiskless = isDisklessTopic(newImage.configs(), resource.name());
if (wasDiskless != isDiskless) {
Expand All @@ -146,6 +145,23 @@ private void publishDelta(MetadataDelta delta, MetadataImage newImage) {
}
}
changes.apply(metrics);
publishDisklessWithoutRemoteStorageDelta(changedTopicNames, newImage);
}

private void publishDisklessWithoutRemoteStorageDelta(
Set<String> changedTopicNames,
MetadataImage newImage
) {
int disklessWithoutRemoteStorageChange = 0;
for (String topicName : changedTopicNames) {
disklessWithoutRemoteStorageChange += ControllerMetricsChanges.delta(
isDisklessWithoutRemoteStorage(prevImage, topicName),
isDisklessWithoutRemoteStorage(newImage, topicName)
);
}
if (disklessWithoutRemoteStorageChange != 0) {
metrics.addToDisklessWithoutRemoteStorageCount(disklessWithoutRemoteStorageChange);
}
}

private void publishSnapshot(MetadataImage newImage) {
Expand Down Expand Up @@ -174,17 +190,13 @@ private void publishSnapshot(MetadataImage newImage) {
int disklessPartitions = 0;
int disklessOfflinePartitions = 0;
int disklessWithoutRemoteStorage = 0;
List<String> disklessTopicsWithoutRemoteStorage = new ArrayList<>();
for (TopicImage topicImage : newImage.topics().topicsById().values()) {
// Check diskless from newImage configs directly for consistency with delta path
final boolean isDiskless = isDisklessTopic(newImage.configs(), topicImage.name());
if (isDiskless) {
disklessTopics++;
if (hasRemoteStorageExplicitlyDisabled(newImage.configs(), topicImage.name())) {
if (!isRemoteStorageEnabled(newImage.configs(), topicImage.name())) {
disklessWithoutRemoteStorage++;
if (disklessTopicsWithoutRemoteStorage.size() < 20) {
disklessTopicsWithoutRemoteStorage.add(topicImage.name());
}
}
}
for (PartitionRegistration partition : topicImage.partitions().values()) {
Expand All @@ -210,11 +222,13 @@ private void publishSnapshot(MetadataImage newImage) {
metrics.setDisklessTopicCount(disklessTopics);
metrics.setDisklessPartitionCount(disklessPartitions);
metrics.setDisklessOfflinePartitionCount(disklessOfflinePartitions);
// Only refreshed on snapshot (not delta) — advisory metric for legacy topic detection,
// not a real-time alert. Snapshots occur on controller failover or periodically
// (default: every hour or 20MB of metadata records, whichever comes first).
metrics.setDisklessWithoutRemoteStorageCount(disklessWithoutRemoteStorage);
warnDisklessWithoutRemoteStorage(disklessWithoutRemoteStorage, disklessTopicsWithoutRemoteStorage);
}

private static boolean isDisklessWithoutRemoteStorage(MetadataImage image, String topicName) {
return image.topics().getTopic(topicName) != null
&& isDisklessTopic(image.configs(), topicName)
&& !isRemoteStorageEnabled(image.configs(), topicName);
}

private static boolean isDisklessTopic(ConfigurationsImage configsImage, String topicName) {
Expand All @@ -223,23 +237,10 @@ private static boolean isDisklessTopic(ConfigurationsImage configsImage, String
return Boolean.parseBoolean(configMap.getOrDefault(TopicConfig.DISKLESS_ENABLE_CONFIG, "false"));
}

private static boolean hasRemoteStorageExplicitlyDisabled(ConfigurationsImage configsImage, String topicName) {
private static boolean isRemoteStorageEnabled(ConfigurationsImage configsImage, String topicName) {
ConfigResource resource = new ConfigResource(ConfigResource.Type.TOPIC, topicName);
Map<String, String> configMap = configsImage.configMapForResource(resource);
// Only flag topics where remote.storage.enable is explicitly stored as false.
// Absent means "never configured" — the controller will auto-enable on next interaction.
String value = configMap.get(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG);
return value != null && !Boolean.parseBoolean(value);
}

private void warnDisklessWithoutRemoteStorage(int count, List<String> topicNames) {
if (count > 0 && count <= 20) {
log.warn("Found {} diskless topic(s) with remote.storage.enable explicitly set to false: {}. "
+ "Set remote.storage.enable=true on these topics.", count, topicNames);
} else if (count > 20) {
log.warn("Found {} diskless topic(s) with remote.storage.enable explicitly set to false (truncated, first 20): {}. "
+ "Set remote.storage.enable=true on these topics.", count, topicNames.stream().limit(20).toList());
}
return Boolean.parseBoolean(configMap.getOrDefault(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG, "false"));
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ public void testLoadSnapshot() {
assertEquals(0, env.metrics.disklessTopicCount());
assertEquals(0, env.metrics.disklessPartitionCount());
assertEquals(0, env.metrics.disklessOfflinePartitionCount());
assertEquals(0, env.metrics.disklessWithoutRemoteStorageCount());
}
}

Expand All @@ -212,14 +213,13 @@ public void testLoadSnapshotWithDisklessTopics() {
assertEquals(7, env.metrics.disklessPartitionCount());
// 3 partitions from "quux" topic are offline (leader=-1)
assertEquals(3, env.metrics.disklessOfflinePartitionCount());
// remote.storage.enable absent (never configured) — not counted; only explicit false triggers the metric
assertEquals(0, env.metrics.disklessWithoutRemoteStorageCount());
// All three topics have remote.storage.enable unset, so its effective value is false.
assertEquals(3, env.metrics.disklessWithoutRemoteStorageCount());
}
}

@Test
public void testDisklessWithoutRemoteStorageCountsOnlyExplicitFalse() {
// Only diskless topics with remote.storage.enable explicitly stored as "false" are counted.
public void testDisklessWithoutRemoteStorageCountsFalseAndUnset() {
try (TestEnv env = new TestEnv()) {
// Build image with 3 diskless topics: remote.storage.enable=true, =false, and absent
Map<ConfigResource, ConfigurationImage> configs = new HashMap<>();
Expand All @@ -242,7 +242,46 @@ public void testDisklessWithoutRemoteStorageCountsOnlyExplicitFalse() {
env.publisher.onMetadataUpdate(delta, image, fakeManifest(true));

assertEquals(3, env.metrics.disklessTopicCount());
// Only "bar" (remote.storage.enable explicitly false) is counted
// "bar" has remote storage explicitly disabled, and "quux" inherits the false default.
assertEquals(2, env.metrics.disklessWithoutRemoteStorageCount());
}
}

@Test
public void testDisklessWithoutRemoteStorageTracksConfigDeltas() {
try (TestEnv env = new TestEnv()) {
MetadataDelta snapshotDelta = new MetadataDelta(MetadataImage.EMPTY);
ImageReWriter writer = new ImageReWriter(snapshotDelta);
IMAGE1_DISKLESS.write(writer, new ImageWriterOptions.Builder(MetadataVersion.MINIMUM_VERSION).build());
env.publisher.onMetadataUpdate(snapshotDelta, IMAGE1_DISKLESS, fakeManifest(true));
assertEquals(3, env.metrics.disklessWithoutRemoteStorageCount());

MetadataDelta enableRemoteStorageDelta = new MetadataDelta(IMAGE1_DISKLESS);
enableRemoteStorageDelta.replay(new ConfigRecord()
.setResourceType(ConfigResource.Type.TOPIC.id())
.setResourceName("foo")
.setName(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG)
.setValue("true"));
MetadataImage remoteStorageEnabledImage =
enableRemoteStorageDelta.apply(MetadataProvenance.EMPTY);
env.publisher.onMetadataUpdate(
enableRemoteStorageDelta,
remoteStorageEnabledImage,
fakeManifest(false));
assertEquals(2, env.metrics.disklessWithoutRemoteStorageCount());

MetadataDelta disableDisklessDelta = new MetadataDelta(remoteStorageEnabledImage);
disableDisklessDelta.replay(new ConfigRecord()
.setResourceType(ConfigResource.Type.TOPIC.id())
.setResourceName("bar")
.setName(TopicConfig.DISKLESS_ENABLE_CONFIG)
.setValue("false"));
MetadataImage disklessDisabledImage =
disableDisklessDelta.apply(MetadataProvenance.EMPTY);
env.publisher.onMetadataUpdate(
disableDisklessDelta,
disklessDisabledImage,
fakeManifest(false));
assertEquals(1, env.metrics.disklessWithoutRemoteStorageCount());
}
}
Expand Down Expand Up @@ -283,6 +322,7 @@ public void testCreateDisklessTopicViaDeltaDoesNotDoubleCount() {
"Diskless partition count should equal global partition count");
assertEquals(0, env.metrics.offlinePartitionCount());
assertEquals(0, env.metrics.disklessOfflinePartitionCount());
assertEquals(1, env.metrics.disklessWithoutRemoteStorageCount());
}
}

Expand All @@ -304,6 +344,7 @@ public void testDeleteDisklessTopicViaDelta() {
env.publisher.onMetadataUpdate(snapshotDelta, baseImage, fakeManifest(true));
assertEquals(1, env.metrics.globalTopicCount());
assertEquals(1, env.metrics.disklessTopicCount());
assertEquals(1, env.metrics.disklessWithoutRemoteStorageCount());

// Now delete via delta
MetadataDelta deleteDelta = new MetadataDelta(baseImage);
Expand All @@ -316,6 +357,7 @@ public void testDeleteDisklessTopicViaDelta() {
assertEquals(0, env.metrics.disklessTopicCount());
assertEquals(0, env.metrics.globalPartitionCount());
assertEquals(0, env.metrics.disklessPartitionCount());
assertEquals(0, env.metrics.disklessWithoutRemoteStorageCount());
}
}

Expand Down Expand Up @@ -367,6 +409,7 @@ public void testDisklessConfigChangeOnExistingTopicWithNewPartitions() {
assertEquals(2, env.metrics.disklessPartitionCount(),
"Both partitions (old + new) should be counted as diskless exactly once");
assertEquals(0, env.metrics.offlinePartitionCount());
assertEquals(1, env.metrics.disklessWithoutRemoteStorageCount());
}
}
}
Loading
Loading