Repository navigation
fix(inkless:retention): size retention enforcement to keep up with produce - #744
Merged
Merged
Conversation
jeqo
force-pushed
the
jeqo/fix-retention-enforcer-defaults
branch
2 times, most recently
from
August 12, 2026 16:04
c0a9979 to
924893c
Compare
jeqo
force-pushed
the
jeqo/kill-background-on-restart
branch
7 times, most recently
from
August 19, 2026 07:07
b5a369c to
c9e1a69
Compare
jeqo
force-pushed
the
jeqo/fix-retention-enforcer-defaults
branch
from
August 19, 2026 07:10
924893c to
71633c5
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Updates retention defaults to increase deletion throughput for continuously written diskless partitions.
Changes:
- Reduces enforcement interval from five minutes to one minute.
- Raises deletion cap from 1,000 to 2,000 batches.
- Decouples WAL cleanup cadence and updates configuration documentation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
InklessConfig.java |
Updates retention and cleanup defaults. |
InklessConfigTest.java |
Updates the batch-cap default assertion. |
configs.rst |
Documents the new defaults and sizing guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…oduce The defaults couldn't keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per writing broker, so a single writer already exceeds 4 batches/s, and in diskless any broker can write to any partition. The batches table grew without bound on busy partitions while data outlived its retention. Raise capacity by ~10x: - retention.enforcement.interval.ms: 5 min -> 1 min - retention.enforcement.max.batches.per.request: 1000 -> 2000 That's 33 batches/s per partition, covering a partition written by ~8 brokers at the default commit interval. The interval was the better lever. Capacity is cap/interval, so a 10x cap would also work, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22), and the delete holds the log's row: fewer, heavier passes hold locks longer, which V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The tradeoff is transaction rate: every due partition costs one transaction regardless of expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions). consolidation.cleanup.interval.ms no longer derives from the retention interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost profile differs (whole-file scans, not per-partition scans), so it keeps 5 min explicitly. Not addressed: these defaults assume ~8 brokers writing to a partition; nothing scales with cluster size. RetentionEnforcementScheduler already throttles per-broker cadence by brokerCount, but that bounds enforcement transactions, not delete throughput, which stays capped at cap/interval regardless of cluster size while producer fan-in per partition can grow with it. Scaling the cap by brokerCount would close that gap but reintroduce the heavier locked passes this change avoided; relaxing the cadence throttle instead grows transaction rate the same way. Both just move the cost this change traded away, so it is a scheduling design question, not a follow-up tweak: today every broker checks every partition independently with no coordination. Once managed-replica leaders are the norm, as tiered-storage consolidation becomes standard, a partition gets one natural scheduling owner instead of N uncoordinated brokers, which changes the shape of this problem. Left for that.
jeqo
force-pushed
the
jeqo/fix-retention-enforcer-defaults
branch
from
August 19, 2026 07:48
71633c5 to
0aa4595
Compare
jeqo
marked this pull request as ready for review
August 19, 2026 07:53
tvainika
approved these changes
Aug 19, 2026
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
…oduce (#744) The defaults couldn't keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per writing broker, so a single writer already exceeds 4 batches/s, and in diskless any broker can write to any partition. The batches table grew without bound on busy partitions while data outlived its retention. Raise capacity by ~10x: - retention.enforcement.interval.ms: 5 min -> 1 min - retention.enforcement.max.batches.per.request: 1000 -> 2000 That's 33 batches/s per partition, covering a partition written by ~8 brokers at the default commit interval. The interval was the better lever. Capacity is cap/interval, so a 10x cap would also work, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22), and the delete holds the log's row: fewer, heavier passes hold locks longer, which V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The tradeoff is transaction rate: every due partition costs one transaction regardless of expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions). consolidation.cleanup.interval.ms no longer derives from the retention interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost profile differs (whole-file scans, not per-partition scans), so it keeps 5 min explicitly. Not addressed: these defaults assume ~8 brokers writing to a partition; nothing scales with cluster size. RetentionEnforcementScheduler already throttles per-broker cadence by brokerCount, but that bounds enforcement transactions, not delete throughput, which stays capped at cap/interval regardless of cluster size while producer fan-in per partition can grow with it. Scaling the cap by brokerCount would close that gap but reintroduce the heavier locked passes this change avoided; relaxing the cadence throttle instead grows transaction rate the same way. Both just move the cost this change traded away, so it is a scheduling design question, not a follow-up tweak: today every broker checks every partition independently with no coordination. Once managed-replica leaders are the norm, as tiered-storage consolidation becomes standard, a partition gets one natural scheduling owner instead of N uncoordinated brokers, which changes the shape of this problem. Left for that.
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
…oduce (#744) The defaults couldn't keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per writing broker, so a single writer already exceeds 4 batches/s, and in diskless any broker can write to any partition. The batches table grew without bound on busy partitions while data outlived its retention. Raise capacity by ~10x: - retention.enforcement.interval.ms: 5 min -> 1 min - retention.enforcement.max.batches.per.request: 1000 -> 2000 That's 33 batches/s per partition, covering a partition written by ~8 brokers at the default commit interval. The interval was the better lever. Capacity is cap/interval, so a 10x cap would also work, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22), and the delete holds the log's row: fewer, heavier passes hold locks longer, which V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The tradeoff is transaction rate: every due partition costs one transaction regardless of expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions). consolidation.cleanup.interval.ms no longer derives from the retention interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost profile differs (whole-file scans, not per-partition scans), so it keeps 5 min explicitly. Not addressed: these defaults assume ~8 brokers writing to a partition; nothing scales with cluster size. RetentionEnforcementScheduler already throttles per-broker cadence by brokerCount, but that bounds enforcement transactions, not delete throughput, which stays capped at cap/interval regardless of cluster size while producer fan-in per partition can grow with it. Scaling the cap by brokerCount would close that gap but reintroduce the heavier locked passes this change avoided; relaxing the cadence throttle instead grows transaction rate the same way. Both just move the cost this change traded away, so it is a scheduling design question, not a follow-up tweak: today every broker checks every partition independently with no coordination. Once managed-replica leaders are the norm, as tiered-storage consolidation becomes standard, a partition gets one natural scheduling owner instead of N uncoordinated brokers, which changes the shape of this problem. Left for that.
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
jeqo
added a commit
that referenced
this pull request
Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
jeqo
added a commit
that referenced
this pull request
Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defaults couldn't keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms: 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per writing broker, so a single writer already exceeds 4 batches/s, and in diskless any broker can write to any partition. The batches table grew without bound on busy partitions while data outlived its retention.
Raise capacity by ~10x:
That's 33 batches/s per partition, covering a partition written by ~8 brokers at the default commit interval.
The interval was the better lever. Capacity is cap/interval, so a 10x cap would also work, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22), and the delete holds the log's row: fewer, heavier passes hold locks longer, which V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The tradeoff is transaction rate: every due partition costs one transaction regardless of expiry, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions).
consolidation.cleanup.interval.ms no longer derives from the retention interval, so WAL pruning doesn't silently get 5x more frequent too. Its cost profile differs (whole-file scans, not per-partition scans), so it keeps 5 min explicitly.
Not addressed: these defaults assume ~8 brokers writing to a partition; nothing scales with cluster size. RetentionEnforcementScheduler already throttles per-broker cadence by brokerCount, but that bounds enforcement transactions, not delete throughput, which stays capped at cap/interval regardless of cluster size while producer fan-in per partition can grow with it. Scaling the cap by brokerCount would close that gap but reintroduce the heavier locked passes this change avoided; relaxing the cadence throttle instead grows transaction rate the same way. Both just move the cost this change traded away, so it is a scheduling design question, not a follow-up tweak: today every broker checks every partition independently with no coordination. Once managed-replica leaders are the norm, as tiered-storage consolidation becomes standard, a partition gets one natural scheduling owner instead of N uncoordinated brokers, which changes the shape of this problem. Left for that.