Skip to content

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

Closed

Description

@neild

A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the
http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.

HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.

This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.

The default stream concurrency limit is 250 streams (requests)
per HTTP/2 connection. This value may be adjusted using the
golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
setting and the ConfigureServer function.

This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
This is also tracked by CVE-2023-44487.


This is a PRIVATE issue for CVE-2023-39325, tracked in http://b/303836512.

/cc @golang/security and @golang/release

Activity

  1. self-assigned this
    on Oct 6, 2023
  2. added this to the Go1.22 milestone on Oct 6, 2023
  3. neild commented on Oct 6, 2023

    @neild
    ContributorAuthor

    @gopherbot please open backport issues

  4. gopherbot commented on Oct 6, 2023

    @gopherbot
    Contributor

    Backport issue(s) opened: #63426 (for 1.20), #63427 (for 1.21).

    Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

  5. gopherbot commented on Oct 10, 2023

    @gopherbot
    Contributor

    Change https://go.dev/cl/534215 mentions this issue: http2: limit maximum handler goroutines to MaxConcurrentStreams

  6. gopherbot commented on Oct 10, 2023

    @gopherbot
    Contributor

    Change https://go.dev/cl/534235 mentions this issue: [release-branch.go1.21] net/http: regenerate h2_bundle.go

  7. gopherbot commented on Oct 10, 2023

    @gopherbot
    Contributor

    Change https://go.dev/cl/534255 mentions this issue: [release-branch.go1.20] net/http: regenerate h2_bundle.go

  8. dmitshur commented on Oct 10, 2023

    @dmitshur
    Member

    Reopening to track bundling into the main tree, for Go 1.22.

  9. gopherbot commented on Oct 10, 2023

    @gopherbot
    Contributor

    Change https://go.dev/cl/534295 mentions this issue: all: pull in x/net v0.17.0 and its dependencies

  10. 35 remaining items

  11. added a commit that references this issue on Dec 7, 2023
  12. aojea commented on Dec 26, 2023

    @aojea
    Contributor

    FYI, this fix is not sufficient for Kube, and possibly other projects. We are working on this in kubernetes/kubernetes#121120. pprof svg from a single client with a single connection attempting to DOS a Kube API server that has this fix and max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):

    @enj from the pprof attached the bulk of the memory consumed comes from the 4kb buffer allocated to the responseWriter that is created after the Headers frame is received. The buffer is also obtained from the sync.Pool that will make the memory usage more efficient.

    https://github.com/golang/net/blob/f12db26b1c9293fa3eb95c936e548d2c1fba4ba9/http2/server.go#L69-L75

    In this case, since that the attack is based only on headers frames, postponing the allocation of the responseWriter will reduce a lot the impact on memory usage, however, I've been checking around the code and I don't know how feasible is to do that, based on the rfc7540

    Sending or receiving a HEADERS frame causes the stream to become "open"

    A stream in the "open" state may be used by both peers to send frames of any type. In this state, sending peers observe
    advertised stream-level flow-control limits (Section 5.2)

  13. ilyanaum commented on Aug 20, 2024

    @ilyanaum

    The issue with HTTP/2 rapid reset is still active. When an HTTP/2 server is attacked using rapid resets, it results in memory leaks (up to peak memory usage). This problem is especially noticeable when using net/http/httputil (reverse proxy). IMPORTANT: The memory leak occurs ONLY during rapid reset attacks.

    photo_2024-08-20_23-26-23
    profile002
    Screenshot 2024-08-20 234726

  14. ilyanaum commented on Aug 20, 2024

    @ilyanaum
  15. locked and limited conversation to collaborators on Aug 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions