Skip to content

Cancel unfinished BigQuery read streams on close - #1

Closed
akmhmgc wants to merge 2 commits into
masterfrom
bigquery-cancel-read-streams
Closed

akmhmgc wants to merge 2 commits into
masterfrom
bigquery-cancel-read-streams

Conversation

@akmhmgc

@akmhmgc akmhmgc commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Description

Cancel the BigQuery Storage ReadRows stream when a page source is closed before the stream is fully read.

close() cancelled the future that fetches the next response and then closed the BigQueryReadClient. Closing the client only shuts the gRPC channel down gracefully, which lets in-flight calls continue, and cancelling the future does not stop the call either. So whenever splits are closed before their streams end, the streams stay open on the BigQuery side. This happens when a query fails or is cancelled, and also when a query finishes early, for example once a LIMIT is satisfied, because the Storage API cannot push the limit down. A thread waiting for the next response also stays blocked.

Each open stream counts against the MaxConcurrentReadConnectionsPerProjectRegion quota (400 by default) until BigQuery ends it. While the quota is used up by these streams, new reads in the same project fail with RESOURCE_EXHAUSTED.

ReadRowsHelper now keeps the current ServerStream, and the page sources cancel it first in close(). A stream opened by a retry after close() is cancelled as well.

Cancelling the stream is preferred over bigQueryReadClient.shutdownNow(): a call stopped by shutdownNow() ends with UNAVAILABLE, which both BigQueryUtil.isRetryable and the gax ReadRows retry settings retry, while a cancelled call ends with CANCELLED and is not retried.

Additional context and related issues

We saw this in production with a query that opens many streams. After one failed query, the quota usage (read/max_concurrent_connections_region) stayed at 360 to 398 of 400 for about an hour with nothing running, until BigQuery ended the streams with CANCELLED. Every query in that hour failed, even one reading only one hour of data.

The new tests read the first page from a stream that never completes, close the page source, and check that the server sees the call cancelled. They fail without the change.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## BigQuery connector
* Fix query failures with `RESOURCE_EXHAUSTED` errors caused by read streams left open by earlier queries that failed, were cancelled, or finished early.

@akmhmgc akmhmgc closed this Nov 1, 2026
@akmhmgc akmhmgc reopened this Nov 1, 2026
@akmhmgc akmhmgc closed this Nov 1, 2026
@akmhmgc akmhmgc reopened this Nov 1, 2026
Closing the BigQueryReadClient only shuts the gRPC channel down
gracefully, which lets in-flight calls continue, and cancelling the
future that fetches the next response does not stop the call either.
When splits are closed before their streams end, because the query
failed, was cancelled, or finished early, the ReadRows streams stay
open on the BigQuery side. Each one counts against the concurrent read
connections quota until BigQuery ends it, so later reads in the same
project fail with RESOURCE_EXHAUSTED.

Keep the current ServerStream in ReadRowsHelper and cancel it first
when the page source is closed.
@akmhmgc
akmhmgc force-pushed the bigquery-cancel-read-streams branch from 0aa68a4 to c57fe32 Compare November 1, 2026 05:26
@akmhmgc

akmhmgc commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Opened upstream as trinodb#31945.

@akmhmgc akmhmgc closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant