You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417
A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.
HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.
This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.
FYI, this fix is not sufficient for Kube, and possibly other projects. We are working on this in kubernetes/kubernetes#121120. pprof svg from a single client with a single connection attempting to DOS a Kube API server that has this fix and max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):
@enj from the pprof attached the bulk of the memory consumed comes from the 4kb buffer allocated to the responseWriter that is created after the Headers frame is received. The buffer is also obtained from the sync.Pool that will make the memory usage more efficient.
In this case, since that the attack is based only on headers frames, postponing the allocation of the responseWriter will reduce a lot the impact on memory usage, however, I've been checking around the code and I don't know how feasible is to do that, based on the rfc7540
Sending or receiving a HEADERS frame causes the stream to become "open"
A stream in the "open" state may be used by both peers to send frames of any type. In this state, sending peers observe
advertised stream-level flow-control limits (Section 5.2)
The issue with HTTP/2 rapid reset is still active. When an HTTP/2 server is attacked using rapid resets, it results in memory leaks (up to peak memory usage). This problem is especially noticeable when using net/http/httputil (reverse proxy). IMPORTANT: The memory leak occurs ONLY during rapid reset attacks.
Activity
neild commentedon Oct 6, 2023
@gopherbot please open backport issues
gopherbot commentedon Oct 6, 2023
Backport issue(s) opened: #63426 (for 1.20), #63427 (for 1.21).
Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.
gopherbot commentedon Oct 10, 2023
Change https://go.dev/cl/534215 mentions this issue:
http2: limit maximum handler goroutines to MaxConcurrentStreamsgopherbot commentedon Oct 10, 2023
Change https://go.dev/cl/534235 mentions this issue:
[release-branch.go1.21] net/http: regenerate h2_bundle.gogopherbot commentedon Oct 10, 2023
Change https://go.dev/cl/534255 mentions this issue:
[release-branch.go1.20] net/http: regenerate h2_bundle.godmitshur commentedon Oct 10, 2023
Reopening to track bundling into the main tree, for Go 1.22.
[release-branch.go1.21] net/http: regenerate h2_bundle.go
gopherbot commentedon Oct 10, 2023
Change https://go.dev/cl/534295 mentions this issue:
all: pull in x/net v0.17.0 and its dependencies35 remaining items
fix: disable HTTP2 connections by default
net/http: regenerate h2_bundle.go
aojea commentedon Dec 26, 2023
@enj from the pprof attached the bulk of the memory consumed comes from the 4kb buffer allocated to the
responseWriterthat is created after the Headers frame is received. The buffer is also obtained from the sync.Pool that will make the memory usage more efficient.https://github.com/golang/net/blob/f12db26b1c9293fa3eb95c936e548d2c1fba4ba9/http2/server.go#L69-L75
In this case, since that the attack is based only on headers frames, postponing the allocation of the responseWriter will reduce a lot the impact on memory usage, however, I've been checking around the code and I don't know how feasible is to do that, based on the rfc7540
chore(deps): update ⬆️ gomod to v0.17.0 (#112)
chore(deps): update ⬆️ gomod to v0.17.0 (#51)
[release-branch.go1.21] net/http: regenerate h2_bundle.go
[release-branch.go1.21] all: tidy dependency versioning after release
[release-branch.go1.21] net/http: regenerate h2_bundle.go
[release-branch.go1.21] all: tidy dependency versioning after release
ilyanaum commentedon Aug 20, 2024
The issue with HTTP/2 rapid reset is still active. When an HTTP/2 server is attacked using rapid resets, it results in memory leaks (up to peak memory usage). This problem is especially noticeable when using net/http/httputil (reverse proxy). IMPORTANT: The memory leak occurs ONLY during rapid reset attacks.
ilyanaum commentedon Aug 20, 2024
@bradfitz
Disable HTTP/2 by Default for Webhooks to Mitigate CVE Risks
Disable HTTP/2 by Default for Webhooks to Mitigate CVE Risks