Skip to content

fix(inkless:retention): Limit the size of AWS delete error messages - #733

Merged
jeqo merged 1 commit into
mainfrom
delete-error-message
Aug 6, 2026
Merged

jeqo merged 1 commit into
mainfrom
delete-error-message

Conversation

@gharris1727

Copy link
Copy Markdown
Contributor

The number of keys passed to the ObjectDeleter:delete method may be extremely large. If any key fails to delete, an error message including every key is assembled. This error message may be several megabytes in size, putting excessive memory pressure on the broker node.

Instead, the error message should be scoped to just the batch of keys that has just been deleted. This is limited by MAX_DELETE_KEYS_LIMIT (1000) which should keep the size of the message bounded.

This same failure mode appears in the other cloud implementations, those bugs/fixes are out of scope but will be addressed in an immediate follow-up. We will also investigate limiting the number of keys distributed to nodes by the control plane.

@gharris1727
gharris1727 requested review from ivanyu and jeqo August 6, 2026 19:43
@jeqo
jeqo merged commit f751436 into main Aug 6, 2026
@jeqo
jeqo deleted the delete-error-message branch August 6, 2026 19:51
jeqo added a commit that referenced this pull request Aug 11, 2026
Two error paths interpolated an unbounded value into the message, the same
class of bug as the S3 bulk-delete fix (#733):

- FetchCompleter: the catch-all wrapped the failure with the full
  fetchInfos.keySet(), which spans every client-requested partition
  (up to thousands). Report only the partition count.
- FileCommitter: logged the whole ClosedFile record, whose toString dumps
  every aggregated produce request and commit batch. Log a bounded summary
  (start, request count, batch count, bytes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 11, 2026
A delete pass that fails for every key it submitted logged one line per key
(with a stack trace per key on Azure) and repeated it on every cleanup cycle,
since undeleted keys stay marked for deletion. GCS still interpolated the whole
key set into the exception message, the bug #733 fixed for S3.

Aggregate per-key failures into DeleteErrorSummary: a count by error code, at
most 3 sampled hard failures, and the first hard cause. One line per call,
bounded by the distinct error codes rather than the key count, at INFO when
every failure was a throttle and WARN otherwise. GCS cannot report per-key
results, so its message just carries the key count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 11, 2026
Two error paths interpolated an unbounded value into the message, the same
class of bug as the S3 bulk-delete fix (#733):

- FetchCompleter: the catch-all wrapped the failure with the full
  fetchInfos.keySet(), which spans every client-requested partition
  (up to thousands). Report only the partition count.
- FileCommitter: logged the whole ClosedFile record, whose toString dumps
  every aggregated produce request and commit batch. Log a bounded summary
  (start, request count, batch count, bytes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 12, 2026
Two error paths interpolated an unbounded value into the message, the same
class of bug as the S3 bulk-delete fix (#733):

- FetchCompleter: the catch-all wrapped the failure with the full
  fetchInfos.keySet(), which spans every client-requested partition
  (up to thousands). Report only the partition count.
- FileCommitter: logged the whole ClosedFile record, whose toString dumps
  every aggregated produce request and commit batch. Log a bounded summary
  (start, request count, batch count, bytes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 12, 2026
A delete pass that fails for every key it submitted logged one line per key
(with a stack trace per key on Azure) and repeated it on every cleanup cycle,
since undeleted keys stay marked for deletion. GCS still interpolated the whole
key set into the exception message, the bug #733 fixed for S3.

Aggregate per-key failures into DeleteErrorSummary: a count by error code, at
most 3 sampled hard failures, and the first hard cause. One line per call,
bounded by the distinct error codes rather than the key count, at INFO when
every failure was a throttle and WARN otherwise. GCS cannot report per-key
results, so its message just carries the key count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 13, 2026
…741)

Two error paths interpolated an unbounded value into the message, the same
class of bug as the S3 bulk-delete fix (#733):

- FetchCompleter: the catch-all wrapped the failure with the full
  fetchInfos.keySet(), which spans every client-requested partition
  (up to thousands). Report only the partition count.
- FileCommitter: logged the whole ClosedFile record, whose toString dumps
  every aggregated produce request and commit batch. Log a bounded summary
  (start, request count, batch count, bytes).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 18, 2026
A delete pass that fails for every key it submitted logged one line per key
(with a stack trace per key on Azure) and repeated it on every cleanup cycle,
since undeleted keys stay marked for deletion. GCS still interpolated the whole
key set into the exception message, the bug #733 fixed for S3.

Aggregate per-key failures into DeleteErrorSummary: a count by error code, at
most 3 sampled hard failures, and the first hard cause. One line per call,
bounded by the distinct error codes rather than the key count, at INFO when
every failure was a throttle and WARN otherwise. GCS cannot report per-key
results, so its message just carries the key count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 18, 2026
A delete pass that fails for every key it submitted logs one line per key, with a
stack trace each on Azure, and repeats that on every cleanup cycle, since undeleted
keys stay marked for deletion. GCS also interpolates the whole key set into the
exception message, which is the bug #733 fixed for S3.

Aggregate the per-key failures of one call into `DeleteErrorSummary`: a count by
error code, at most 3 sampled hard failures, and the first hard cause. One line per
call, bounded by the distinct error codes rather than by the key count, at INFO when
every failure is a throttle and at WARN otherwise. GCS can't report per-key results,
so its message carries only the key count.

A whole-request `DeleteObjects` failure joins the same summary and claims the keys
the pass neither deleted nor already counted, so one line still accounts for every
undeleted key. It counts as hard even under a throttling code: the adaptive retry
has spent its budget by then.

Azure counts only `ServerBusy` as throttling, so a pass that fails entirely on
`InternalError` or `OperationTimedOut` keeps the WARN and the stack trace instead of
dropping to INFO. Those are transient but don't self-heal like backpressure.

## Testing

`DeleteErrorSummaryTest` covers the sample cap, the per-code breakdown, cause
retention, whole-request attribution, and the INFO-versus-WARN choice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 18, 2026
A delete pass that fails for every key it submitted logs one line per key, with a
stack trace each on Azure, and repeats that on every cleanup cycle, since undeleted
keys stay marked for deletion. GCS interpolates the whole key set into the exception
message, which is the bug #733 fixed for S3.

Count the failures of a pass by error code and log them once, at INFO when every code
is a throttle and at WARN otherwise, so the output is bounded by the distinct codes
rather than by the worklist. Report an exception that carries no service code in full
instead, since the stack trace is then the only diagnosis: a whole-request
`SdkException` on S3, any non-`BlobStorageException` on Azure. Both stop the pass,
because neither is attributable to a single key and the rest retry on the next cycle.
GCS logs only the key count, since it can't report per-key results.

The counts describe one pass and never the worklist: `FileCleaner` reports how many
candidates it deleted and derives `FileCleanerFilesFailedRate` from the same numbers.

Both throttle sets keep only the codes the provider calls throttling, since the set
picks the log level and a transient server error doesn't self-heal like backpressure.
S3 drops `ServiceUnavailable`, which AWS classifies as a retryable 503. Azure keeps
`ServerBusy` alone, sourced from `BlobErrorCode`.

[KC-430]

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 19, 2026
A delete pass that fails for every key it submitted logs one line per key, with a
stack trace each on Azure, and repeats that on every cleanup cycle, since undeleted
keys stay marked for deletion. GCS interpolates the whole key set into the exception
message, which is the bug #733 fixed for S3.

Count the failures of a pass by error code and log them once, at INFO when every code
is a throttle and at WARN otherwise, so the output is bounded by the distinct codes
rather than by the worklist. Report an exception that carries no service code in full
instead, since the stack trace is then the only diagnosis: a whole-request
`SdkException` on S3, any non-`BlobStorageException` on Azure. Both stop the pass,
because neither is attributable to a single key and the rest retry on the next cycle.
GCS logs only the key count, since it can't report per-key results.

The counts describe one pass and never the worklist: `FileCleaner` reports how many
candidates it deleted and derives `FileCleanerFilesFailedRate` from the same numbers.

Both throttle sets keep only the codes the provider calls throttling, since the set
picks the log level and a transient server error doesn't self-heal like backpressure.
S3 drops `ServiceUnavailable`, which AWS classifies as a retryable 503. Azure keeps
`ServerBusy` alone, sourced from `BlobErrorCode`.

[KC-430]

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 19, 2026
…741)

Two error paths interpolated an unbounded value into the message, the same
class of bug as the S3 bulk-delete fix (#733):

- FetchCompleter: the catch-all wrapped the failure with the full
  fetchInfos.keySet(), which spans every client-requested partition
  (up to thousands). Report only the partition count.
- FileCommitter: logged the whole ClosedFile record, whose toString dumps
  every aggregated produce request and commit batch. Log a bounded summary
  (start, request count, batch count, bytes).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 19, 2026
A delete pass that fails for every key it submitted logs one line per key, with a
stack trace each on Azure, and repeats that on every cleanup cycle, since undeleted
keys stay marked for deletion. GCS interpolates the whole key set into the exception
message, which is the bug #733 fixed for S3.

Count the failures of a pass by error code and log them once, at INFO when every code
is a throttle and at WARN otherwise, so the output is bounded by the distinct codes
rather than by the worklist. Report an exception that carries no service code in full
instead, since the stack trace is then the only diagnosis: a whole-request
`SdkException` on S3, any non-`BlobStorageException` on Azure. Both stop the pass,
because neither is attributable to a single key and the rest retry on the next cycle.
GCS logs only the key count, since it can't report per-key results.

The counts describe one pass and never the worklist: `FileCleaner` reports how many
candidates it deleted and derives `FileCleanerFilesFailedRate` from the same numbers.

Both throttle sets keep only the codes the provider calls throttling, since the set
picks the log level and a transient server error doesn't self-heal like backpressure.
S3 drops `ServiceUnavailable`, which AWS classifies as a retryable 503. Azure keeps
`ServerBusy` alone, sourced from `BlobErrorCode`.

[KC-430]

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 19, 2026
…741)

Two error paths interpolated an unbounded value into the message, the same
class of bug as the S3 bulk-delete fix (#733):

- FetchCompleter: the catch-all wrapped the failure with the full
  fetchInfos.keySet(), which spans every client-requested partition
  (up to thousands). Report only the partition count.
- FileCommitter: logged the whole ClosedFile record, whose toString dumps
  every aggregated produce request and commit batch. Log a bounded summary
  (start, request count, batch count, bytes).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jeqo added a commit that referenced this pull request Aug 19, 2026
A delete pass that fails for every key it submitted logs one line per key, with a
stack trace each on Azure, and repeats that on every cleanup cycle, since undeleted
keys stay marked for deletion. GCS interpolates the whole key set into the exception
message, which is the bug #733 fixed for S3.

Count the failures of a pass by error code and log them once, at INFO when every code
is a throttle and at WARN otherwise, so the output is bounded by the distinct codes
rather than by the worklist. Report an exception that carries no service code in full
instead, since the stack trace is then the only diagnosis: a whole-request
`SdkException` on S3, any non-`BlobStorageException` on Azure. Both stop the pass,
because neither is attributable to a single key and the rest retry on the next cycle.
GCS logs only the key count, since it can't report per-key results.

The counts describe one pass and never the worklist: `FileCleaner` reports how many
candidates it deleted and derives `FileCleanerFilesFailedRate` from the same numbers.

Both throttle sets keep only the codes the provider calls throttling, since the set
picks the log level and a transient server error doesn't self-heal like backpressure.
S3 drops `ServiceUnavailable`, which AWS classifies as a retryable 503. Azure keeps
`ServerBusy` alone, sourced from `BlobErrorCode`.

[KC-430]

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants