Skip to content

Commit be71172

Browse files
norrietaylorclaude
andcommitted
fix(minimald): frame gvproxy forwarder replies by Content-Length
The forwarder control verbs (POST /services/forwarder/expose and the :7654/:7655 proxy-publish) sent HTTP/1.0 with `Connection: close` and read the reply to EOF. Over the KVM libkrun `add_vsock_port2(listen = false)` shuttle, gvproxy's response is dropped when it closes the socket, so the guest reads 0 bytes ("malformed gvproxy status line") even though the forward was applied. apply_ingress then rolls back a *successful* own-ip ingress attach, so the session's shell never starts (gap G-N8); own-ip without ingress is unaffected. Send HTTP/1.1 keep-alive (no `Connection: close`) and frame the reply by Content-Length instead of read-to-EOF, so the server never closes first and the response drains to the guest, which closes from its own side. This also repairs the in-VM :7654/:7655 proxy-publish (same leg). The native unix path (DM2) is unaffected. Verified live on DM3 (KVM): own-ip + --ingress attach starts the shell and publishes the lease; TC3/TC4/TC7/TC8 = 200/200/(401+200)/200, with no startup proxy-publish warnings. Adds a regression test that hangs if the framing reverts to read-to-EOF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 3238918 commit be71172

1 file changed

Lines changed: 129 additions & 17 deletions

File tree

‎crates/minimald/src/net/policy.rs‎

Lines changed: 129 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -174,25 +174,33 @@ pub async fn remove_ingress(control: &ControlChannel, exposed: &[ExposedMapping]
174174
}
175175

176176
/// `POST`s `body` as JSON to `path` on gvproxy's control socket over a fresh
177-
/// HTTP/1.0 connection, succeeding on a 2xx status.
177+
/// HTTP/1.1 keep-alive connection, succeeding on a 2xx status.
178178
///
179179
/// gvproxy's control socket speaks HTTP; the relay's `POST /connect` upgrade in
180180
/// [`switch`](super::switch) is the data-plane verb, while the forwarder verbs
181-
/// here are ordinary request/response. HTTP/1.0 with `Connection: close` keeps
182-
/// the exchange to one write + read-to-EOF, with no need to parse chunked or
183-
/// keep-alive framing.
181+
/// here are ordinary request/response. The exchange is framed by `Content-Length`
182+
/// (not read-to-EOF) with **no** `Connection: close`, so the server never closes
183+
/// the socket first — required for the response to survive the KVM vsock shuttle
184+
/// (see [`exchange`] and the request-builder comment below for the G-N8 detail).
184185
pub(crate) async fn post_json<T: Serialize>(
185186
control: &ControlChannel,
186187
path: &str,
187188
body: &T,
188189
) -> io::Result<()> {
189190
let body = serde_json::to_vec(body).map_err(io::Error::other)?;
190191
let mut request = Vec::with_capacity(128 + body.len());
191-
request.extend_from_slice(format!("POST {path} HTTP/1.0\r\n").as_bytes());
192+
// HTTP/1.1 keep-alive (no `Connection: close`): gvproxy must respond *without*
193+
// closing the connection. On the KVM libkrun `add_vsock_port2(listen=false)`
194+
// shuttle, a server-initiated close drops the still-buffered response bytes as
195+
// it propagates to the guest (the guest reads an immediate EOF → 0 bytes →
196+
// `malformed gvproxy status line: ""`, even though the forward *was* applied —
197+
// G-N8). Keeping the connection open lets the response drain to the guest,
198+
// which then reads exactly `Content-Length` bytes and closes from its side.
199+
request.extend_from_slice(format!("POST {path} HTTP/1.1\r\n").as_bytes());
192200
request.extend_from_slice(b"Host: localhost\r\n");
193201
request.extend_from_slice(b"Content-Type: application/json\r\n");
194202
request.extend_from_slice(format!("Content-Length: {}\r\n", body.len()).as_bytes());
195-
request.extend_from_slice(b"Connection: close\r\n\r\n");
203+
request.extend_from_slice(b"\r\n");
196204
request.extend_from_slice(&body);
197205

198206
// Bound the whole exchange: a gvproxy that accepts the socket and then
@@ -229,25 +237,63 @@ pub(crate) async fn post_json<T: Serialize>(
229237
}
230238
}
231239

232-
/// Writes `request` to a freshly-connected control stream, half-closes the write
233-
/// side (so an HTTP/1.0 server responds without waiting for more body), and reads
234-
/// the (bounded) response. Transport-agnostic so the unix (DM2) and vsock
240+
/// Writes `request` to a freshly-connected control stream and reads the response
241+
/// framed by its `Content-Length`. Transport-agnostic so the unix (DM2) and vsock
235242
/// (DM1/3/4) control channels share one exchange.
243+
///
244+
/// We must **not** read to EOF: the request is HTTP/1.1 keep-alive (see
245+
/// [`post_json`]), so gvproxy holds the connection open after responding and an
246+
/// EOF-terminated read would block until the [`GVPROXY_CONTROL_TIMEOUT`]. Instead
247+
/// we read the status line + headers, take `Content-Length` body bytes, and let
248+
/// the caller drop the stream (closing from the *guest* side). This avoids the
249+
/// server-initiated close that the KVM libkrun vsock splice mishandles by dropping
250+
/// the buffered response (G-N8). We deliberately do not `shutdown()` the write
251+
/// side for the same reason.
236252
async fn exchange<S: AsyncRead + AsyncWrite + Unpin>(
237253
mut stream: S,
238254
request: &[u8],
239255
) -> io::Result<Vec<u8>> {
240256
stream.write_all(request).await?;
241-
stream.shutdown().await?;
242257

258+
// Read incrementally until the header terminator is in hand, capping total
259+
// buffered bytes: gvproxy's forwarder replies are tiny, so a stream that never
260+
// completes its headers within the bound is misbehaving.
243261
let mut response = Vec::with_capacity(256);
244-
// Cap the buffered response: gvproxy's forwarder replies are tiny (`200 OK`
245-
// with a minimal body), so a stream that keeps sending past this bound is
246-
// misbehaving and must not be allowed to grow memory unbounded.
247-
(&mut stream)
248-
.take(MAX_CONTROL_RESPONSE)
249-
.read_to_end(&mut response)
250-
.await?;
262+
let mut buf = [0u8; 512];
263+
let header_end = loop {
264+
if let Some(i) = find_header_end(&response) {
265+
break i;
266+
}
267+
if response.len() as u64 >= MAX_CONTROL_RESPONSE {
268+
return Err(io::Error::other(
269+
"gvproxy control response exceeded the size cap before its headers ended",
270+
));
271+
}
272+
let n = stream.read(&mut buf).await?;
273+
if n == 0 {
274+
// EOF before headers completed — the empty-reply symptom the
275+
// keep-alive framing is meant to avoid; surface it as-is.
276+
return Ok(response);
277+
}
278+
response.extend_from_slice(&buf[..n]);
279+
};
280+
281+
// Body length from `Content-Length` (0 if absent). Read until the body is
282+
// fully buffered, so we never depend on a server close to signal completion.
283+
let want = content_length(&response[..header_end]).unwrap_or(0);
284+
let body_have = response.len() - header_end;
285+
let mut remaining = want.saturating_sub(body_have);
286+
while remaining > 0 {
287+
if response.len() as u64 >= MAX_CONTROL_RESPONSE {
288+
break;
289+
}
290+
let n = stream.read(&mut buf).await?;
291+
if n == 0 {
292+
break;
293+
}
294+
response.extend_from_slice(&buf[..n]);
295+
remaining = remaining.saturating_sub(n);
296+
}
251297
Ok(response)
252298
}
253299

@@ -266,6 +312,27 @@ fn parse_status_code(response: &[u8]) -> io::Result<u16> {
266312
.ok_or_else(|| io::Error::other(format!("malformed gvproxy status line: {line:?}")))
267313
}
268314

315+
/// Index just past the `\r\n\r\n` header terminator, or `None` if the headers are
316+
/// not yet fully buffered.
317+
fn find_header_end(response: &[u8]) -> Option<usize> {
318+
response
319+
.windows(4)
320+
.position(|w| w == b"\r\n\r\n")
321+
.map(|i| i + 4)
322+
}
323+
324+
/// The `Content-Length` header value parsed from a response's header block, or
325+
/// `None` if the header is absent or unparseable (treated as a zero-length body).
326+
fn content_length(headers: &[u8]) -> Option<usize> {
327+
std::str::from_utf8(headers).ok()?.lines().find_map(|line| {
328+
let (name, value) = line.split_once(':')?;
329+
name.trim()
330+
.eq_ignore_ascii_case("content-length")
331+
.then(|| value.trim().parse::<usize>().ok())
332+
.flatten()
333+
})
334+
}
335+
269336
/// Returns the body bytes following the blank line that ends the headers, or the
270337
/// whole buffer if no header terminator is found (best-effort, for diagnostics).
271338
fn body_after_headers(response: &[u8]) -> &[u8] {
@@ -441,4 +508,49 @@ mod tests {
441508
assert_eq!(Direction::Egress.to_string(), "egress");
442509
assert_eq!(Direction::Ingress.to_string(), "ingress");
443510
}
511+
512+
#[test]
513+
fn content_length_parses_case_insensitively_or_absent() {
514+
assert_eq!(
515+
content_length(b"HTTP/1.1 200 OK\r\nContent-Length: 6\r\n\r\n"),
516+
Some(6)
517+
);
518+
assert_eq!(
519+
content_length(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n"),
520+
Some(0)
521+
);
522+
assert_eq!(content_length(b"HTTP/1.1 200 OK\r\n\r\n"), None);
523+
}
524+
525+
#[tokio::test]
526+
async fn exchange_frames_by_content_length_without_a_server_close() {
527+
// Regression for G-N8: the response must be framed by `Content-Length`,
528+
// never by the server closing the socket. On the KVM libkrun vsock shuttle
529+
// a server-initiated close drops the still-buffered response, so `exchange`
530+
// must return the full reply from a peer that stays open. If it ever
531+
// reverts to read-to-EOF, this test hangs (the peer never closes).
532+
let (client, mut server) = tokio::io::duplex(1024);
533+
let server = tokio::spawn(async move {
534+
let mut buf = [0u8; 256];
535+
// Drain the request (headers + empty body arrive together here).
536+
let _ = server.read(&mut buf).await.unwrap();
537+
server
538+
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 6\r\n\r\nWEB_OK")
539+
.await
540+
.unwrap();
541+
// Hold the connection open — do NOT close — until the caller is done.
542+
server
543+
});
544+
545+
let resp = exchange(
546+
client,
547+
b"POST /services/forwarder/expose HTTP/1.1\r\nHost: localhost\r\nContent-Length: 0\r\n\r\n",
548+
)
549+
.await
550+
.expect("exchange returns the framed response without waiting for EOF");
551+
552+
assert_eq!(parse_status_code(&resp).unwrap(), 200);
553+
assert_eq!(body_after_headers(&resp), b"WEB_OK");
554+
drop(server.await.unwrap());
555+
}
444556
}

0 commit comments

Comments
 (0)