Skip to content

A pipeline sent before reading hangs its connection (200,000 SETs on macOS, 939,480 on Linux io_uring); Redis takes 2M #865

Closed

Description

@azeemshaik025

A client that sends a long pipeline before it reads any reply hangs its connection for good. redis-py's pipeline does that: it sends every command (send_packed_command) and only then reads the replies. Redis keeps reading while replies wait to be written, so it serves the same pipeline.

Reproduce

The script below sends a pipeline without reading and says whether the server took all of it. "Hangs" is no progress for 3 s.

macOS (M4), master 1db384a4, tokio, default settings, against Redis 8.8.0 on the same machine:

Sent before reading KiviDB Redis 8.8.0
170,000 SETs ok ok
200,000 SETs hangs after 6,352,616 of 6,488,890 bytes ok, 0.05 s
2,097,152 SETs hangs after 5,326,236 of 70,192,058 bytes ok, 0.78 s
10 GETs of a 1 MB value, then 16 SETs of 1 MB hangs after 1,044,492 of 16,777,980 bytes ok

Linux (Docker, master 2468f226; both loops read and write the same way today), the fewest SETs that hang:

Hangs at
io_uring, default socket buffers 939,480
io_uring, buffers capped at 128 KiB 134,759
tokio, buffers capped at 128 KiB 133,374
Redis 8.10.2 none up to 2,097,152

The 1 MB mix hangs under both loops on Linux with default buffers too. Other connections carry on: a new client gets PONG throughout.

pipe.py
#!/usr/bin/env python3
"""Send a whole pipeline before reading any reply, as redis-py's pipeline
does, and say whether the server kept up.

    pipe.py <port> set <n>...   n SETs of a small value
    pipe.py <port> big          10 GETs of a 1 MB value, then 16 SETs of 1 MB

"hung" means the server took no more bytes for 3 s.
"""
import select
import socket
import sys
import time


def sets(n):
    return b"".join(b"*3\r\n$3\r\nSET\r\n$%d\r\n%s\r\n$1\r\nv\r\n" % (len(k), k)
                    for k in (b"k%d" % i for i in range(n)))


def big():
    mb = b"x" * (1 << 20)
    one = b"*3\r\n$3\r\nSET\r\n$3\r\nbig\r\n$%d\r\n%s\r\n" % (len(mb), mb)
    return b"*2\r\n$3\r\nGET\r\n$3\r\nbig\r\n" * 10 + one * 16


def run(port, data, setup=b""):
    s = socket.create_connection(("127.0.0.1", port))
    if setup:
        s.sendall(setup)
        s.recv(64)
    s.setblocking(False)
    view, sent, start = memoryview(data), 0, time.monotonic()
    while sent < len(data):
        if not select.select([], [s], [], 3)[1]:
            s.close()
            return f"hung: sent {sent:,} of {len(data):,} bytes"
        try:
            sent += s.send(view[sent:])
        except BlockingIOError:
            pass
    s.close()
    return f"ok in {time.monotonic() - start:.2f} s"


port, mode = int(sys.argv[1]), sys.argv[2]
if mode == "set":
    for n in map(int, sys.argv[3:]):
        print(f"{n:>9,} SETs: {run(port, sets(n))}", flush=True)
else:
    mb = b"x" * (1 << 20)
    setup = b"*3\r\n$3\r\nSET\r\n$3\r\nbig\r\n$%d\r\n%s\r\n" % (len(mb), mb)
    print(f"10 GET + 16 SET of 1 MB: {run(port, big(), setup)}", flush=True)

python3 pipe.py 6380 set 170000 200000 2097152, then python3 pipe.py 6380 big.

Cause

Both loops read, run what they read, and then wait until every reply is written before they read again:

  • tokio: handle_client reads into a 4 KiB buffer (src/server.rs:703), then awaits write_tcp_batch (src/server.rs:2757). park_blocking flushes the same way (src/server.rs:198).
  • io_uring: reads at src/server_uring.rs:1027 and writes at src/server_uring.rs:2973, with no read in flight while it writes.

While the client is still sending, the replies fill the server's send buffer and the client's receive buffer, and the requests fill the other way. The server waits to write, the client waits to write, and neither reads. In one stall on macOS the server's socket held 1,545,828 bytes of requests not yet read beside 146,923 bytes of replies not yet sent.

keysizes_counts_every_key_of_a_large_database (tests/info_keyspace.rs:258) sends 200,000 SETs before it reads, which is why it hangs on macOS and passes on Linux.

What Redis does

Redis reads a client whenever it is readable, whatever is waiting to be written to it. client-query-buffer-limit (1 GB) bounds what it holds of the requests, and client-output-buffer-limit the replies.

A way to fix it

  • tokio: read into carried while the batch is written, as until_gone (src/server.rs:371) already does for a parked command, up to client-query-buffer-limit. A patch along these lines on 2468f226 took 2,097,152 SETs on macOS and passed info_keyspace, output_buffer_limit, blocked_client_gone, pause and config. park_blocking needs the same.
  • io_uring: the read can't be raced against the write, because tokio_uring can't cancel it (src/server_uring.rs:845). It needs a read kept in flight across the write, or a reader of its own.

The 1 MB mix makes a regression test that hangs on Linux as well as macOS, under both loops.

This changes the same read in handle_client as #432, so the two should agree on its shape.

I'd like to work on this, the tokio loop first.

Activity

  1. murtazayusuf commented on Nov 17, 2026

    @murtazayusuf
    Contributor

    Thanks @azeemshaik025, this is a very good report: the cause is right, and the 1 MB mix as the test that hangs on both loops is exactly what it needs. Yes, please take it, tokio first; I've assigned you.

    A few things that will make the PR go through quickly:

    • This is the hottest code in the server. handle_client's read → run → write turn is what every GET and SET goes through, and the common case (the write finishes at once, nothing more to read) must come out with the same instructions as today. Reading while writing should be the path taken only when the write can't complete. Please put instruction counts in the PR: scripts/perf/run.sh (callgrind, GET/SET at 100 B and 1 KB, pipelined) with MODEL=tokio, before and after; if you can't run it where you are, say so and I'll measure. A change that looked like one field read cost 50 instructions per GET here this week, so we measure everything on this path.
    • Bounds. What is read ahead counts against client-query-buffer-limit, as you say, and the replies still against client-output-buffer-limit: a client that never reads must still be dropped by the output limit rather than have the server hold its whole pipeline. Tests for both limits being reached during a read-while-writing.
    • Order and pausing. Commands read ahead must not run before the batch being written is done with (replies in order; CLIENT PAUSE, SHUTDOWN, a blocked command's line and MULTI state all sit in that loop), only be buffered.
    • perf: Tokio 4 KiB reads limit pipelined throughput for larger values #432 (4 KiB reads on tokio) is assigned to @Parikalp-Bhardwaj and touches the same read. Simplest is that yours doesn't change the read size and his doesn't change when it is read; I'll make sure whichever lands second is merged over the first.
    • io_uring as a second PR is fine. A read kept in flight across the write is the shape I'd expect there too; say in this issue what you plan before writing it, since that loop has more to get wrong.
    • A first PR here needs the CLA comment on it ("I have read the CLA Document and I hereby sign the CLA", CLA.md).

    keysizes_counts_every_key_of_a_large_database hanging on macOS for this reason is a good catch as well; the fix should let it pass there unchanged.

  2. azeemshaik025 commented on Nov 17, 2026

    @azeemshaik025
    ContributorAuthor

    The io_uring half, before I write it. #868 has the tokio one.

    What is different on this loop

    • A write is never done on its first poll: it completes when the ring is reaped. So there is no "poll it once, and read only if it has not finished" as on tokio.
    • A submitted read cannot be taken back (src/server_uring.rs:1008), so racing a uring read against the write would leave an op in flight, owning its buffer, after the write is done.

    The plan

    1. Submit the batch's next read before its write rather than after it. It is the read the loop makes next anyway: into parse_buf, after the same READ_MIN reserve the loop top does (src/server_uring.rs:1020). The write follows under write_guard as now, and the loop top awaits that read instead of submitting one. On the common path that is the same two ops in the other order; I'll measure it rather than assume it costs nothing.
    2. Not when the batch ends the connection or hands it on: QUIT, a SHUTDOWN that goes ahead, and a replica's handoff after PSYNC/SYNC. Those write as today and submit nothing early, so no read is left to take bytes that belong to the replication link.
    3. If that read completes while the write is still pending, the client is sending what it will read the replies to later, and the server has to keep reading. From there, out of line (boxed, as PeerWatch::gone is): wait for the socket to be readable through PeerWatch's registration, and recv(MSG_DONTWAIT) into parse_buf's spare capacity, reserving READ_BUFFER at a time, until the write completes. No uring read is submitted on this path, so nothing is left in flight when the write ends, and the loop top parses what was read before it reads again.
    4. What is read ahead is only buffered, as on tokio: it runs after the batch is written. Past client-query-buffer-limit the client is closed, with the socket shut down as the soft-limit timeout does, which also ends the pending write.
    5. The soft output limit's timeout covers the write and the reading together.
    6. park_blocking_uring's flush is mid-batch, so there is no next read to submit early: it races its write with the same readability wait from the start, and keeps the bytes for the loop to put in parse_buf before its next read.

    Tests

    • The three tests in tests/output_buffer_limit.rs run on both loops (io_models()), on Linux in the kividb-ci image.
    • pipe.py from this issue against both models.
    • Instruction counts with MODEL=uring (run.sh's default), LTO=off and LTO=fat, three runs a side.

    Open questions

    • An early read in the other states the loop goes through (MONITOR, a subscriber, a client whose socket CLIENT KILL shuts mid-write): I'll go through each, but say if you know one where reading before the write is wrong.
    • Step 3 only needs the read to finish before the write does. I'll check on the CI kernel that the 1 MB mix takes that path whether a write into a full socket comes back short or waits for room.
  3. murtazayusuf commented on Nov 17, 2026

    @murtazayusuf
    Contributor

    Thanks for laying it out first, @azeemshaik025. The plan is sound: one read submitted early, and after it nothing but readiness plus recv(MSG_DONTWAIT), so no op is ever left in flight. Go ahead once #868's bound is settled (see the review there), because the same rule has to hold on this loop. Points to build in:

    • The bound from server: a pipeline sent before its replies are read no longer hangs the connection (#865) #868 applies here too. What is read ahead is consumed a read's worth per turn, and reading ahead stops (and the client is dropped as the limit drops it) once what is waiting could not be delivered within the hard output limit. On this loop parse_buf is where it accumulates, so the parse loop must not be handed the whole of it in one turn.
    • Step 1, the order of the two ops. Instructions alone won't show what matters here. Please also report, before and after: io_uring_enter calls per turn (strace -c -f or perf stat -e syscalls:sys_enter_io_uring_enter over a fixed number of commands), and latency for one command per round trip (redis-benchmark -P 1 -c 1 and -c 50: p50/p99), since that is where a reply delayed behind a read submission would show. If the read and the write don't reach the kernel in the same submission, the unpipelined case pays a syscall; they must.
    • The early read and a batch that parks. A command that blocks (BLPOP, WAIT, a paused client) leaves the turn without writing: the early read is then in flight while PeerWatch is watching the same socket for the client going away. Say how the two coexist: a read in flight will itself see EOF, so the park may need to take its "gone" signal from that read rather than register a second interest.
    • PeerWatch's registration used for step 3 must not disturb its use for parked clients: one registration per socket, and whichever path is active owns it.
    • Order of bytes. Step 3 relies on the early read having completed before any recv. Assert it (a debug assertion that no read is in flight when recv is called); a recv overtaking an in-flight uring read would reorder the client's commands, which is the one failure here that corrupts rather than hangs.
    • Subscribers and MONITOR: their pushes are written by other tasks under write_guard; an early read is fine there (their input is only (P)(UN)SUBSCRIBE/PING/QUIT), but check that a subscriber's loop also awaits the early read rather than submitting a second one. CLIENT KILL mid-write: the shutdown must end both the pending write and the readiness wait; and a killed client's early read completes with an error that must not be reported as a protocol error.
    • Kernel. CI's runner kernel and the oldest we support for io_uring behave differently on short writes to a full socket; run the 1 MB mix on both paths you describe (short write, and a write that waits) by capping the socket buffers, as your first table did.
    • TLS connections on an io_uring server already go through the tokio loop, so server: a pipeline sent before its replies are read no longer hangs the connection (#865) #868 covers them; say so in the CHANGELOG entry of this half.

    Same as for #868: instruction counts on x86 I'll re-run here in any case.

  4. added 2 commits that reference this issue on Nov 17, 2026
  5. added 2 commits that reference this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions