Repository navigation
Description
A client that sends a long pipeline before it reads any reply hangs its connection for good. redis-py's pipeline does that: it sends every command (send_packed_command) and only then reads the replies. Redis keeps reading while replies wait to be written, so it serves the same pipeline.
Reproduce
The script below sends a pipeline without reading and says whether the server took all of it. "Hangs" is no progress for 3 s.
macOS (M4), master 1db384a4, tokio, default settings, against Redis 8.8.0 on the same machine:
| Sent before reading | KiviDB | Redis 8.8.0 |
|---|---|---|
170,000 SETs |
ok | ok |
200,000 SETs |
hangs after 6,352,616 of 6,488,890 bytes | ok, 0.05 s |
2,097,152 SETs |
hangs after 5,326,236 of 70,192,058 bytes | ok, 0.78 s |
10 GETs of a 1 MB value, then 16 SETs of 1 MB |
hangs after 1,044,492 of 16,777,980 bytes | ok |
Linux (Docker, master 2468f226; both loops read and write the same way today), the fewest SETs that hang:
| Hangs at | |
|---|---|
| io_uring, default socket buffers | 939,480 |
| io_uring, buffers capped at 128 KiB | 134,759 |
| tokio, buffers capped at 128 KiB | 133,374 |
| Redis 8.10.2 | none up to 2,097,152 |
The 1 MB mix hangs under both loops on Linux with default buffers too. Other connections carry on: a new client gets PONG throughout.
pipe.py
#!/usr/bin/env python3
"""Send a whole pipeline before reading any reply, as redis-py's pipeline
does, and say whether the server kept up.
pipe.py <port> set <n>... n SETs of a small value
pipe.py <port> big 10 GETs of a 1 MB value, then 16 SETs of 1 MB
"hung" means the server took no more bytes for 3 s.
"""
import select
import socket
import sys
import time
def sets(n):
return b"".join(b"*3\r\n$3\r\nSET\r\n$%d\r\n%s\r\n$1\r\nv\r\n" % (len(k), k)
for k in (b"k%d" % i for i in range(n)))
def big():
mb = b"x" * (1 << 20)
one = b"*3\r\n$3\r\nSET\r\n$3\r\nbig\r\n$%d\r\n%s\r\n" % (len(mb), mb)
return b"*2\r\n$3\r\nGET\r\n$3\r\nbig\r\n" * 10 + one * 16
def run(port, data, setup=b""):
s = socket.create_connection(("127.0.0.1", port))
if setup:
s.sendall(setup)
s.recv(64)
s.setblocking(False)
view, sent, start = memoryview(data), 0, time.monotonic()
while sent < len(data):
if not select.select([], [s], [], 3)[1]:
s.close()
return f"hung: sent {sent:,} of {len(data):,} bytes"
try:
sent += s.send(view[sent:])
except BlockingIOError:
pass
s.close()
return f"ok in {time.monotonic() - start:.2f} s"
port, mode = int(sys.argv[1]), sys.argv[2]
if mode == "set":
for n in map(int, sys.argv[3:]):
print(f"{n:>9,} SETs: {run(port, sets(n))}", flush=True)
else:
mb = b"x" * (1 << 20)
setup = b"*3\r\n$3\r\nSET\r\n$3\r\nbig\r\n$%d\r\n%s\r\n" % (len(mb), mb)
print(f"10 GET + 16 SET of 1 MB: {run(port, big(), setup)}", flush=True)python3 pipe.py 6380 set 170000 200000 2097152, then python3 pipe.py 6380 big.
Cause
Both loops read, run what they read, and then wait until every reply is written before they read again:
- tokio:
handle_clientreads into a 4 KiB buffer (src/server.rs:703), then awaitswrite_tcp_batch(src/server.rs:2757).park_blockingflushes the same way (src/server.rs:198). - io_uring: reads at
src/server_uring.rs:1027and writes atsrc/server_uring.rs:2973, with no read in flight while it writes.
While the client is still sending, the replies fill the server's send buffer and the client's receive buffer, and the requests fill the other way. The server waits to write, the client waits to write, and neither reads. In one stall on macOS the server's socket held 1,545,828 bytes of requests not yet read beside 146,923 bytes of replies not yet sent.
keysizes_counts_every_key_of_a_large_database (tests/info_keyspace.rs:258) sends 200,000 SETs before it reads, which is why it hangs on macOS and passes on Linux.
What Redis does
Redis reads a client whenever it is readable, whatever is waiting to be written to it. client-query-buffer-limit (1 GB) bounds what it holds of the requests, and client-output-buffer-limit the replies.
A way to fix it
- tokio: read into
carriedwhile the batch is written, asuntil_gone(src/server.rs:371) already does for a parked command, up toclient-query-buffer-limit. A patch along these lines on2468f226took 2,097,152SETs on macOS and passedinfo_keyspace,output_buffer_limit,blocked_client_gone,pauseandconfig.park_blockingneeds the same. - io_uring: the read can't be raced against the write, because tokio_uring can't cancel it (
src/server_uring.rs:845). It needs a read kept in flight across the write, or a reader of its own.
The 1 MB mix makes a regression test that hangs on Linux as well as macOS, under both loops.
This changes the same read in handle_client as #432, so the two should agree on its shape.
I'd like to work on this, the tokio loop first.
Activity
murtazayusuf commentedon Nov 17, 2026
Thanks @azeemshaik025, this is a very good report: the cause is right, and the 1 MB mix as the test that hangs on both loops is exactly what it needs. Yes, please take it, tokio first; I've assigned you.
A few things that will make the PR go through quickly:
handle_client's read → run → write turn is what every GET and SET goes through, and the common case (the write finishes at once, nothing more to read) must come out with the same instructions as today. Reading while writing should be the path taken only when the write can't complete. Please put instruction counts in the PR:scripts/perf/run.sh(callgrind, GET/SET at 100 B and 1 KB, pipelined) withMODEL=tokio, before and after; if you can't run it where you are, say so and I'll measure. A change that looked like one field read cost 50 instructions per GET here this week, so we measure everything on this path.client-query-buffer-limit, as you say, and the replies still againstclient-output-buffer-limit: a client that never reads must still be dropped by the output limit rather than have the server hold its whole pipeline. Tests for both limits being reached during a read-while-writing.CLIENT PAUSE,SHUTDOWN, a blocked command's line andMULTIstate all sit in that loop), only be buffered.keysizes_counts_every_key_of_a_large_databasehanging on macOS for this reason is a good catch as well; the fix should let it pass there unchanged.azeemshaik025 commentedon Nov 17, 2026
The io_uring half, before I write it. #868 has the tokio one.
What is different on this loop
src/server_uring.rs:1008), so racing a uring read against the write would leave an op in flight, owning its buffer, after the write is done.The plan
parse_buf, after the sameREAD_MINreserve the loop top does (src/server_uring.rs:1020). The write follows underwrite_guardas now, and the loop top awaits that read instead of submitting one. On the common path that is the same two ops in the other order; I'll measure it rather than assume it costs nothing.QUIT, aSHUTDOWNthat goes ahead, and a replica's handoff afterPSYNC/SYNC. Those write as today and submit nothing early, so no read is left to take bytes that belong to the replication link.PeerWatch::goneis): wait for the socket to be readable throughPeerWatch's registration, andrecv(MSG_DONTWAIT)intoparse_buf's spare capacity, reservingREAD_BUFFERat a time, until the write completes. No uring read is submitted on this path, so nothing is left in flight when the write ends, and the loop top parses what was read before it reads again.client-query-buffer-limitthe client is closed, with the socket shut down as the soft-limit timeout does, which also ends the pending write.park_blocking_uring's flush is mid-batch, so there is no next read to submit early: it races its write with the same readability wait from the start, and keeps the bytes for the loop to put inparse_bufbefore its next read.Tests
tests/output_buffer_limit.rsrun on both loops (io_models()), on Linux in thekividb-ciimage.pipe.pyfrom this issue against both models.MODEL=uring(run.sh's default),LTO=offandLTO=fat, three runs a side.Open questions
MONITOR, a subscriber, a client whose socketCLIENT KILLshuts mid-write): I'll go through each, but say if you know one where reading before the write is wrong.murtazayusuf commentedon Nov 17, 2026
Thanks for laying it out first, @azeemshaik025. The plan is sound: one read submitted early, and after it nothing but readiness plus
recv(MSG_DONTWAIT), so no op is ever left in flight. Go ahead once #868's bound is settled (see the review there), because the same rule has to hold on this loop. Points to build in:parse_bufis where it accumulates, so the parse loop must not be handed the whole of it in one turn.io_uring_entercalls per turn (strace -c -forperf stat -e syscalls:sys_enter_io_uring_enterover a fixed number of commands), and latency for one command per round trip (redis-benchmark -P 1 -c 1and-c 50: p50/p99), since that is where a reply delayed behind a read submission would show. If the read and the write don't reach the kernel in the same submission, the unpipelined case pays a syscall; they must.BLPOP,WAIT, a paused client) leaves the turn without writing: the early read is then in flight whilePeerWatchis watching the same socket for the client going away. Say how the two coexist: a read in flight will itself see EOF, so the park may need to take its "gone" signal from that read rather than register a second interest.PeerWatch's registration used for step 3 must not disturb its use for parked clients: one registration per socket, and whichever path is active owns it.recv. Assert it (a debug assertion that no read is in flight whenrecvis called); arecvovertaking an in-flight uring read would reorder the client's commands, which is the one failure here that corrupts rather than hangs.write_guard; an early read is fine there (their input is only(P)(UN)SUBSCRIBE/PING/QUIT), but check that a subscriber's loop also awaits the early read rather than submitting a second one.CLIENT KILLmid-write: the shutdown must end both the pending write and the readiness wait; and a killed client's early read completes with an error that must not be reported as a protocol error.Same as for #868: instruction counts on x86 I'll re-run here in any case.
server: what is read ahead runs a read at a time and stops at the har…
server: a pipeline sent before its replies are read no longer hangs t…
Merge pull request #868 from azeemshaik025/fix/865-pipeline-deadlock
server_uring: no read ahead on a hand-off turn; a failed write ends t…
server_uring: a pipeline sent before its replies are read no longer h…
Merge pull request #877 from azeemshaik025/fix/865-uring-read-ahead