Repository navigation
Conversation
Closing the BigQueryReadClient only shuts the gRPC channel down gracefully, which lets in-flight calls continue, and cancelling the future that fetches the next response does not stop the call either. When splits are closed before their streams end, because the query failed, was cancelled, or finished early, the ReadRows streams stay open on the BigQuery side. Each one counts against the concurrent read connections quota until BigQuery ends it, so later reads in the same project fail with RESOURCE_EXHAUSTED. Keep the current ServerStream in ReadRowsHelper and cancel it first when the page source is closed.
akmhmgc
force-pushed
the
bigquery-cancel-read-streams
branch
from
November 1, 2026 05:26
0aa68a4 to
c57fe32
Compare
Owner
Author
|
Opened upstream as trinodb#31945. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cancel the BigQuery Storage
ReadRowsstream when a page source is closed before the stream is fully read.close()cancelled the future that fetches the next response and then closed theBigQueryReadClient. Closing the client only shuts the gRPC channel down gracefully, which lets in-flight calls continue, and cancelling the future does not stop the call either. So whenever splits are closed before their streams end, the streams stay open on the BigQuery side. This happens when a query fails or is cancelled, and also when a query finishes early, for example once aLIMITis satisfied, because the Storage API cannot push the limit down. A thread waiting for the next response also stays blocked.Each open stream counts against the
MaxConcurrentReadConnectionsPerProjectRegionquota (400 by default) until BigQuery ends it. While the quota is used up by these streams, new reads in the same project fail withRESOURCE_EXHAUSTED.ReadRowsHelpernow keeps the currentServerStream, and the page sources cancel it first inclose(). A stream opened by a retry afterclose()is cancelled as well.Cancelling the stream is preferred over
bigQueryReadClient.shutdownNow(): a call stopped byshutdownNow()ends withUNAVAILABLE, which bothBigQueryUtil.isRetryableand the gaxReadRowsretry settings retry, while a cancelled call ends withCANCELLEDand is not retried.Additional context and related issues
We saw this in production with a query that opens many streams. After one failed query, the quota usage (
read/max_concurrent_connections_region) stayed at 360 to 398 of 400 for about an hour with nothing running, until BigQuery ended the streams withCANCELLED. Every query in that hour failed, even one reading only one hour of data.The new tests read the first page from a stream that never completes, close the page source, and check that the server sees the call cancelled. They fail without the change.
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: