Skip to content

feat(inkless:consume): default the lagging request-rate limit off, cap batches per partition - #769

Merged
jeqo merged 1 commit into
mainfrom
jeqo/rate-limit-self-tune
Aug 26, 2026
Merged

jeqo merged 1 commit into
mainfrom
jeqo/rate-limit-self-tune

Conversation

@jeqo

@jeqo jeqo commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth defense. Recent incidents were control-plane load, not bandwidth, and mitigating them meant tuning fetch.find.batches.max.per.partition down by hand. The find_batches request-level budget fix (KC-407) plus batch coalescing now let the consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which lever governs what:

  • The lagging consumer thread pool is the steady-state GET-rate governor: pool size / GET latency. That replaces the request-rate limit as the default throttle, so the limit defaults to 0 (disabled) and becomes opt-in for operators who want a hard request/sec cap independent of storage latency, for example to bound object storage request cost. The upper bound is raised to 1000000: even at the ceiling the request cost is bounded, and it is left to the operator to set.
  • fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes already bound bytes, so it bounds the control-plane scan and object GET fan-out when batches are small. It is the only per-partition bound on the consolidation fetch path, which calls find_batches with no byte budget, so this is where the cap does most of its work. 1024 covers a full partition response (about 16 MiB at 16 KiB batches); deeper backlogs drain over successive fetches.

These move together: dropping the rate-limit default is only safe once the batch cap is finite, otherwise both governors would be off at once.

Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by pool concurrency (pool size x range / latency). Raise the lagging thread pool with that in mind.

Out of scope: consolidation keeps its own diskless.consolidation.fetch.find.batches.max.per.partition (default 0); it's already byte-budgeted via KC-407, so a count cap there is a separate, measured decision.

KC-453
[KC-455]

@jeqo
jeqo marked this pull request as ready for review August 25, 2026 08:58
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 08:58
@jeqo
jeqo marked this pull request as draft August 25, 2026 08:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Inkless fetch safeguards by disabling the default lagging-consumer rate limit and introducing a finite per-partition batch cap.

Changes:

  • Disables the lagging-consumer request-rate limit by default.
  • Raises the optional rate-limit ceiling to 1,000,000 requests/second.
  • Defaults per-partition batch discovery to 1,024 batches and updates documentation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java Changes fetch defaults, limits, and descriptions.
storage/inkless/src/test/java/io/aiven/inkless/config/InklessConfigTest.java Updates rate-limit default and validation tests.
docs/inkless/PERFORMANCE.md Updates consumer tuning guidance.
docs/inkless/configs.rst Regenerates configuration reference values.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/inkless/PERFORMANCE.md Resolved
@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from 30f5bee to fb2e993 Compare August 25, 2026 09:09
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 09:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Comment thread docs/inkless/configs.rst Resolved
@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from fb2e993 to e2c5296 Compare August 25, 2026 09:45
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 09:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Suppressed comments (1)

storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java:252

  • This omits the consolidation fetcher's unbudgeted readiness probe. DelayedConsolidationFetch calls ReplicaManager.findDisklessBatches, which passes this shared setting to findBatches; only the subsequent data read uses the consolidation override. The current wording can lead operators to set this value to 0 believing consolidation remains capped.
        + "The consolidation fetcher uses its own override, diskless.consolidation.fetch.find.batches.max.per.partition.";

Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/configs.rst Resolved
@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from e2c5296 to d0a7ad0 Compare August 25, 2026 10:15
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 10:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/configs.rst Resolved
Comment thread docs/inkless/configs.rst Resolved
@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from d0a7ad0 to d960ad3 Compare August 25, 2026 11:06
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 11:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from d960ad3 to e1a183d Compare August 25, 2026 11:24
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 11:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java:254

  • This default doesn't cap the actual consolidation fetch path described in the PR. ReplicaManager.scala:355 passes disklessConsolidationFindBatchesMaxPerPartition to the consolidation Reader, and that separate setting still defaults to 0 (unlimited) in ServerConfigs.java:183-188. The shared setting only reaches the best-effort delayed probe, not the authoritative fetch. Set the consolidation override to a finite default too, or revise the PR's consolidation rationale and scope.
        + "Consolidation has separate settings under diskless.consolidation.";
    private static final int FETCH_FIND_BATCHES_MAX_BATCHES_PER_PARTITION_DEFAULT = 1024;

@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from e1a183d to 6e40692 Compare August 25, 2026 11:32
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 11:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/configs.rst Resolved
Comment thread docs/inkless/PERFORMANCE.md Resolved
…p batches per partition

The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth
defense. Recent incidents were control-plane load, not bandwidth, and mitigating
them meant tuning fetch.find.batches.max.per.partition down by hand. The
find_batches request-level budget fix (KC-407) plus batch coalescing now let the
consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which
lever governs what:

- The lagging consumer thread pool is the steady-state GET-rate governor:
  pool size / GET latency. That replaces the request-rate limit as the default
  throttle, so the limit defaults to 0 (disabled) and becomes opt-in for
  operators who want a hard request/sec cap independent of storage latency, for
  example to bound object storage request cost. The upper bound is raised to
  1000000: even at the ceiling the request cost is bounded, and it is left to
  the operator to set.
- fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It
  is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes
  already bound bytes, so it bounds the control-plane scan and object GET
  fan-out when batches are small. It is the only per-partition bound on the
  consolidation fetch path, which calls find_batches with no byte budget, so
  this is where the cap does most of its work. 1024 covers a full partition
  response (about 16 MiB at 16 KiB batches); deeper backlogs drain over
  successive fetches.

These move together: dropping the rate-limit default is only safe once the
batch cap is finite, otherwise both governors would be off at once.

Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by
pool concurrency (pool size x range / latency). Raise the lagging thread pool
with that in mind.

Testing: InklessConfigTest; :storage:inkless and :core compile.
@jeqo
jeqo force-pushed the jeqo/rate-limit-self-tune branch from 6e40692 to 8898fe0 Compare August 25, 2026 11:45
@jeqo
jeqo requested a balanced review from Copilot August 25, 2026 11:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@jeqo
jeqo marked this pull request as ready for review August 25, 2026 11:54
@jeqo
jeqo requested a review from tvainika August 25, 2026 12:22
@jeqo
jeqo merged commit 4925b82 into main Aug 26, 2026
11 checks passed
@jeqo
jeqo deleted the jeqo/rate-limit-self-tune branch August 26, 2026 07:04
jeqo added a commit that referenced this pull request Sep 9, 2026
…p batches per partition (#769)

The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth
defense. Recent incidents were control-plane load, not bandwidth, and mitigating
them meant tuning fetch.find.batches.max.per.partition down by hand. The
find_batches request-level budget fix (KC-407) plus batch coalescing now let the
consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which
lever governs what:

- The lagging consumer thread pool is the steady-state GET-rate governor:
  pool size / GET latency. That replaces the request-rate limit as the default
  throttle, so the limit defaults to 0 (disabled) and becomes opt-in for
  operators who want a hard request/sec cap independent of storage latency, for
  example to bound object storage request cost. The upper bound is raised to
  1000000: even at the ceiling the request cost is bounded, and it is left to
  the operator to set.
- fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It
  is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes
  already bound bytes, so it bounds the control-plane scan and object GET
  fan-out when batches are small. It is the only per-partition bound on the
  consolidation fetch path, which calls find_batches with no byte budget, so
  this is where the cap does most of its work. 1024 covers a full partition
  response (about 16 MiB at 16 KiB batches); deeper backlogs drain over
  successive fetches.

These move together: dropping the rate-limit default is only safe once the
batch cap is finite, otherwise both governors would be off at once.

Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by
pool concurrency (pool size x range / latency). Raise the lagging thread pool
with that in mind.

Testing: InklessConfigTest; :storage:inkless and :core compile.
jeqo added a commit that referenced this pull request Sep 9, 2026
…p batches per partition (#769)

The lagging request-rate limit shipped defaulting to 200 req/s as a bandwidth
defense. Recent incidents were control-plane load, not bandwidth, and mitigating
them meant tuning fetch.find.batches.max.per.partition down by hand. The
find_batches request-level budget fix (KC-407) plus batch coalescing now let the
consumer's fetch.max.bytes drive per-fetch fan-out fairly, which changes which
lever governs what:

- The lagging consumer thread pool is the steady-state GET-rate governor:
  pool size / GET latency. That replaces the request-rate limit as the default
  throttle, so the limit defaults to 0 (disabled) and becomes opt-in for
  operators who want a hard request/sec cap independent of storage latency, for
  example to bound object storage request cost. The upper bound is raised to
  1000000: even at the ceiling the request cost is bounded, and it is left to
  the operator to set.
- fetch.find.batches.max.per.partition defaults to 1024 (was 0, unbounded). It
  is a per-partition batch count: fetch.max.bytes and max.partition.fetch.bytes
  already bound bytes, so it bounds the control-plane scan and object GET
  fan-out when batches are small. It is the only per-partition bound on the
  consolidation fetch path, which calls find_batches with no byte budget, so
  this is where the cap does most of its work. 1024 covers a full partition
  response (about 16 MiB at 16 KiB batches); deeper backlogs drain over
  successive fetches.

These move together: dropping the rate-limit default is only safe once the
batch cap is finite, otherwise both governors would be off at once.

Tradeoff: with no byte-rate limiter yet, cold-path bandwidth is bounded only by
pool concurrency (pool size x range / latency). Raise the lagging thread pool
with that in mind.

Testing: InklessConfigTest; :storage:inkless and :core compile.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants