Repository navigation
feat(inkless:consume): default the lagging request-rate limit off, cap batches per partition - #769
Conversation
There was a problem hiding this comment.
Pull request overview
Updates Inkless fetch safeguards by disabling the default lagging-consumer rate limit and introducing a finite per-partition batch cap.
Changes:
- Disables the lagging-consumer request-rate limit by default.
- Raises the optional rate-limit ceiling to 1,000,000 requests/second.
- Defaults per-partition batch discovery to 1,024 batches and updates documentation.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java |
Changes fetch defaults, limits, and descriptions. |
storage/inkless/src/test/java/io/aiven/inkless/config/InklessConfigTest.java |
Updates rate-limit default and validation tests. |
docs/inkless/PERFORMANCE.md |
Updates consumer tuning guidance. |
docs/inkless/configs.rst |
Regenerates configuration reference values. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
30f5bee to
fb2e993
Compare
fb2e993 to
e2c5296
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
Suppressed comments (1)
storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java:252
- This omits the consolidation fetcher's unbudgeted readiness probe.
DelayedConsolidationFetchcallsReplicaManager.findDisklessBatches, which passes this shared setting tofindBatches; only the subsequent data read uses the consolidation override. The current wording can lead operators to set this value to0believing consolidation remains capped.
+ "The consolidation fetcher uses its own override, diskless.consolidation.fetch.find.batches.max.per.partition.";
e2c5296 to
d0a7ad0
Compare
d0a7ad0 to
d960ad3
Compare
d960ad3 to
e1a183d
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java:254
- This default doesn't cap the actual consolidation fetch path described in the PR.
ReplicaManager.scala:355passesdisklessConsolidationFindBatchesMaxPerPartitionto the consolidationReader, and that separate setting still defaults to0(unlimited) inServerConfigs.java:183-188. The shared setting only reaches the best-effort delayed probe, not the authoritative fetch. Set the consolidation override to a finite default too, or revise the PR's consolidation rationale and scope.
+ "Consolidation has separate settings under diskless.consolidation.";
private static final int FETCH_FIND_BATCHES_MAX_BATCHES_PER_PARTITION_DEFAULT = 1024;
e1a183d to
6e40692
Compare
…p batches per partition The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth defense. Recent incidents were control-plane load, not bandwidth, and mitigating them meant tuning fetch.find.batches.max.per.partition down by hand. The find_batches request-level budget fix (KC-407) plus batch coalescing now let the consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which lever governs what: - The lagging consumer thread pool is the steady-state GET-rate governor: pool size / GET latency. That replaces the request-rate limit as the default throttle, so the limit defaults to 0 (disabled) and becomes opt-in for operators who want a hard request/sec cap independent of storage latency, for example to bound object storage request cost. The upper bound is raised to 1000000: even at the ceiling the request cost is bounded, and it is left to the operator to set. - fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes already bound bytes, so it bounds the control-plane scan and object GET fan-out when batches are small. It is the only per-partition bound on the consolidation fetch path, which calls find_batches with no byte budget, so this is where the cap does most of its work. 1024 covers a full partition response (about 16 MiB at 16 KiB batches); deeper backlogs drain over successive fetches. These move together: dropping the rate-limit default is only safe once the batch cap is finite, otherwise both governors would be off at once. Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by pool concurrency (pool size x range / latency). Raise the lagging thread pool with that in mind. Testing: InklessConfigTest; :storage:inkless and :core compile.
6e40692 to
8898fe0
Compare
…p batches per partition (#769) The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth defense. Recent incidents were control-plane load, not bandwidth, and mitigating them meant tuning fetch.find.batches.max.per.partition down by hand. The find_batches request-level budget fix (KC-407) plus batch coalescing now let the consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which lever governs what: - The lagging consumer thread pool is the steady-state GET-rate governor: pool size / GET latency. That replaces the request-rate limit as the default throttle, so the limit defaults to 0 (disabled) and becomes opt-in for operators who want a hard request/sec cap independent of storage latency, for example to bound object storage request cost. The upper bound is raised to 1000000: even at the ceiling the request cost is bounded, and it is left to the operator to set. - fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes already bound bytes, so it bounds the control-plane scan and object GET fan-out when batches are small. It is the only per-partition bound on the consolidation fetch path, which calls find_batches with no byte budget, so this is where the cap does most of its work. 1024 covers a full partition response (about 16 MiB at 16 KiB batches); deeper backlogs drain over successive fetches. These move together: dropping the rate-limit default is only safe once the batch cap is finite, otherwise both governors would be off at once. Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by pool concurrency (pool size x range / latency). Raise the lagging thread pool with that in mind. Testing: InklessConfigTest; :storage:inkless and :core compile.
…p batches per partition (#769) The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth defense. Recent incidents were control-plane load, not bandwidth, and mitigating them meant tuning fetch.find.batches.max.per.partition down by hand. The find_batches request-level budget fix (KC-407) plus batch coalescing now let the consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which lever governs what: - The lagging consumer thread pool is the steady-state GET-rate governor: pool size / GET latency. That replaces the request-rate limit as the default throttle, so the limit defaults to 0 (disabled) and becomes opt-in for operators who want a hard request/sec cap independent of storage latency, for example to bound object storage request cost. The upper bound is raised to 1000000: even at the ceiling the request cost is bounded, and it is left to the operator to set. - fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes already bound bytes, so it bounds the control-plane scan and object GET fan-out when batches are small. It is the only per-partition bound on the consolidation fetch path, which calls find_batches with no byte budget, so this is where the cap does most of its work. 1024 covers a full partition response (about 16 MiB at 16 KiB batches); deeper backlogs drain over successive fetches. These move together: dropping the rate-limit default is only safe once the batch cap is finite, otherwise both governors would be off at once. Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by pool concurrency (pool size x range / latency). Raise the lagging thread pool with that in mind. Testing: InklessConfigTest; :storage:inkless and :core compile.
The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth defense. Recent incidents were control-plane load, not bandwidth, and mitigating them meant tuning fetch.find.batches.max.per.partition down by hand. The find_batches request-level budget fix (KC-407) plus batch coalescing now let the consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which lever governs what:
These move together: dropping the rate-limit default is only safe once the batch cap is finite, otherwise both governors would be off at once.
Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by pool concurrency (pool size x range / latency). Raise the lagging thread pool with that in mind.
Out of scope: consolidation keeps its own diskless.consolidation.fetch.find.batches.max.per.partition (default 0); it's already byte-budgeted via KC-407, so a count cap there is a separate, measured decision.
KC-453
[KC-455]