Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 50 additions & 27 deletions crates/minimald/src/box_actor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,6 @@ enum BoxMessage {
/// The attaching connection's liveness, see [`BoxActor::creator`].
Option<std::sync::Weak<()>>,
),
GetHostAttrs(oneshot::Sender<Option<HostAttrs>>),
/// Hand back this session's host, minting one — with no channel bound to
/// it — if the session has none running. Used by the exec path, which runs
/// a command inside the session's sandbox and so needs the sandbox up, but
Expand Down Expand Up @@ -246,6 +245,12 @@ enum BoxMessage {
/// would strand it) or an `Active` with a minted host. A `Draft` that
/// was merely created isn't busy — nothing is in flight to strand.
IsBusy(oneshot::Sender<bool>),
/// Test-only: park the actor for the duration, holding its mailbox the
/// way a long inline message does. The sender is answered as the park
/// begins, so a test knows every earlier message, and the publish that
/// follows it, is done.
#[cfg(test)]
Park(std::time::Duration, oneshot::Sender<()>),
/// Shutdown-stop: kill the host (if any), withdraw the hostname, and stop
/// the actor — the on-disk record is kept.
Stop(oneshot::Sender<()>),
Expand Down Expand Up @@ -399,6 +404,8 @@ enum BoxMessage {
#[derive(Debug)]
pub struct BoxActor {
receiver: crate::traced::Receiver<BoxMessage>,
/// The slot this actor's handle reads its host from; see [`HostSlot`].
host_slot: HostSlot,
minimal_state_dir: DaemonAbsPath,
minimal_cache_dir: DaemonAbsPath,
daemon_ctx: Arc<mctx::DaemonContext>,
Expand Down Expand Up @@ -633,6 +640,7 @@ impl BoxActor {
workspace_uploads_in_flight: 0,
creator,
manager,
host_slot: Arc::clone(&weak_self.2),
weak_self,
destroying,
// No host yet; the first launch sets this. `Interactive` is the
Expand Down Expand Up @@ -781,7 +789,12 @@ impl BoxActor {
}
// A weak self-handle so the actor can hand its own mailbox to the
// runtime objects it spawns without a caller threading it in.
let weak_self = WeakBoxHandle(sender.downgrade(), destroying.clone());
let host_slot = HostSlot::default();
let weak_self = WeakBoxHandle(
sender.downgrade(),
destroying.clone(),
Arc::clone(&host_slot),
);
let actor = Self::assemble(conf, receiver, inner, weak_self, destroying.clone());

// Register the PTask hostname before the actor goes live, so the
Expand All @@ -808,7 +821,28 @@ impl BoxActor {
session_id = %actor.record.id(),
);
crate::traced::spawn_detached(root, actor.mainloop());
Ok(BoxHandle(sender, destroying))
Ok(BoxHandle(sender, destroying, host_slot))
}

/// Publishes the host this actor holds to the slot its handle reads
/// ([`HostSlot`]). Called after every message, so between messages, and
/// for as long as one parks the actor, the slot holds the host the actor
/// last had, and a listing's probe never waits on this mailbox. Also
/// called where a host is installed ([`Self::ensure_host`],
/// [`Self::mint_session_host`], [`Self::launch_host_for_hooks`]), which
/// is before the message that launched it replies: a caller that awaits
/// an attach or `ensure_host` and then reads attrs sees the new host.
pub(super) fn publish_host(&self) {
let host = match &self.inner {
BoxInner::Active {
host: Some((h, _)), ..
} => Some(AttrsProbe(h.clone())),
_ => None,
};
*self
.host_slot
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner) = host;
}
Comment thread
minimal-foundry[bot] marked this conversation as resolved.
Resolved

/// Tears the session down for good: its `on_destroy` hooks (failures
Expand Down Expand Up @@ -870,6 +904,8 @@ impl BoxActor {
/// instead of deadlocking against a full manager mailbox.
async fn mainloop(mut self) {
let mut teardown = Teardown::ManagerInitiated;
let _clear_slot = ClearSlotOnExit(Arc::clone(&self.host_slot));
self.publish_host();
while let Some((msg, caller)) = self.receiver.recv().await {
// Opened now, a child of the sender's ids (see
// `traced::Receiver::recv`), so compose, graph resolution and
Expand All @@ -889,6 +925,7 @@ impl BoxActor {
teardown = t;
break;
}
self.publish_host();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two points about when the slot changes. Neither blocks the merge.

  1. handle_message sends its reply before publish_host runs on this line. A caller that awaits ensure_host or attach and then reads attrs can still see the old slot on the multi-thread runtime. No production caller does this today. The test ensure_host_launches_an_unattached_host_and_then_reuses_it asserts is_some right after ensure_host, and it passes only because #[tokio::test] uses the current-thread runtime. Publish before the reply, or document the window.

  2. The break above this line exits without clearing the slot, and a panic in the actor does the same. The slot keeps a sender to the host's mailbox, so the host loop never sees its channel close, and min ls keeps reporting attrs for a session whose actor is gone. A drop guard that sets the slot to None fixes both cases.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixed in e36f6cf.

  1. The slot is now also published where a host is installed (mint_session_host, launch_host_for_hooks). That runs before the message replies, so a caller that awaits ensure_host or an attach and then reads attrs sees the new host. ensure_host_launches_an_unattached_host_and_then_reuses_it now runs on a multi-thread runtime, so it pins that order.
  2. A guard in the actor's loop empties the slot however the loop ends, a teardown's break or a panic. The host's mailbox then closes, and min ls reports no attrs for an actor that is gone. an_ended_actors_slot_is_emptied covers the break.

Comment thread
minimal-foundry[bot] marked this conversation as resolved.
Resolved
}
if matches!(teardown, Teardown::SelfInitiated) {
let session_id = *self.record.id();
Expand Down Expand Up @@ -922,29 +959,6 @@ impl BoxActor {
BoxMessage::EnsureHost(r, session_hnd, conn_username) => {
let _ = r.send(self.ensure_host(session_hnd, conn_username).await);
}
BoxMessage::GetHostAttrs(r) => match &self.inner {
BoxInner::Active {
host: Some((h, _)), ..
} => {
// Forwarded off-actor for the same reason as
// `GetWorkspaceDelta`: the host loop can be mid-attach or
// mid-teardown, and awaiting it from here parks this
// actor — which the manager awaits in turn, from its own
// mainloop, where a park costs every session RPC.
//
// Bounded *inside* the task: dropping the caller's
// receiver does not cancel a spawned future, so an
// unbounded probe would strand one task per poll against
// a host that never answers.
let h = h.clone();
crate::traced::spawn(async move {
let _ = r.send(probe_host(h.get_attrs()).await);
});
}
_ => {
let _ = r.send(None);
}
},
BoxMessage::GetWorkspaceDelta(r) => match &self.inner {
BoxInner::Active {
host: Some((h, _)), ..
Expand All @@ -966,7 +980,7 @@ impl BoxActor {
host: Some((h, _)), ..
} => {
// Off-actor and bounded for the same reasons as
// `GetHostAttrs`.
// `GetWorkspaceDelta`.
let h = h.clone();
crate::traced::spawn(async move {
let _ = r.send(probe_host(h.get_screen()).await);
Expand Down Expand Up @@ -1027,6 +1041,15 @@ impl BoxActor {
BoxMessage::Rename(new_name, r) => {
let _ = r.send(self.rename(new_name).await);
}
#[cfg(test)]
BoxMessage::Park(d, entered) => {
#[expect(
clippy::let_underscore_must_use,
reason = "the parker may already be gone; the park still runs"
)]
let _ = entered.send(());
tokio::time::sleep(d).await;
}
BoxMessage::IsBusy(r) => {
let _ = r.send(match &self.inner {
// Awaiting a verdict: a client is mid create flow.
Expand Down
99 changes: 88 additions & 11 deletions crates/minimald/src/box_actor/handle.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,56 @@
use super::*;

/// The host a box actor last published, read by its handle: a listing's
/// attrs probe goes to the host through this slot rather than through the
/// actor's mailbox, so an actor that is parked (a `ConfigureLoadout` inside
/// its package-graph clone, an inline attach) costs the listing nothing, and
/// a box with no host reads as `None` at once instead of after
/// [`HOST_PROBE_TIMEOUT`]. The actor refreshes it after every message it
/// handles ([`BoxActor::publish_host`]).
pub(crate) type HostSlot = Arc<std::sync::Mutex<Option<AttrsProbe>>>;

/// Empties a [`HostSlot`] when the actor that publishes it ends, however it
/// ends: a teardown's `break` (which publishes nothing after it), or a panic
/// unwinding the actor's task. A listing then reads no host for an actor
/// that is gone, and the slot holds no sender that would keep the host's
/// mailbox open.
pub(super) struct ClearSlotOnExit(pub(super) HostSlot);

impl Drop for ClearSlotOnExit {
fn drop(&mut self) {
*self
.0
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner) = None;
}
}

/// What the [`HostSlot`] hands out: a host's attrs, and nothing else. The
/// slot is read by every holder of a [`BoxHandle`], so it carries no way to
/// kill, attach to or rename the host: those stay with the actor, which
/// serializes them and checks an attach before binding a channel.
#[derive(Debug, Clone)]
pub(crate) struct AttrsProbe(pub(super) crate::box_runtime::HostHandle);

impl AttrsProbe {
async fn get_attrs(&self) -> Result<HostAttrs, ()> {
self.0.get_attrs().await
}
}

/// The handle to the session.
#[derive(Debug, Clone)]
pub struct BoxHandle(
pub(super) crate::traced::Sender<BoxMessage>,
pub(super) CancellationToken,
pub(super) HostSlot,
);

impl BoxHandle {
/// Returns a non-owning handle to this session.
#[must_use]
pub fn downgrade(&self) -> WeakBoxHandle {
WeakBoxHandle(self.0.downgrade(), self.1.clone())
WeakBoxHandle(self.0.downgrade(), self.1.clone(), Arc::clone(&self.2))
}

/// Whether this session is being destroyed.
Expand Down Expand Up @@ -194,14 +233,47 @@ impl BoxHandle {
std::io::Error::new(std::io::ErrorKind::NotConnected, "session actor is gone")
})
}
/// Returns the host attributes of the running session, if any. A dead
/// actor (self-terminated or mid-teardown) reads as `None` rather than a
/// panic — the manager polls this while actors come and go.
/// Returns the host attributes of the running session, if any: `None`
/// for a session without a host, a dead host, or a host that misses
/// [`HOST_PROBE_TIMEOUT`]. Read through the host slot, not the actor's
/// mailbox (see [`HostSlot`]): the manager polls this for every running
/// session on each listing, and an actor parked in a long message held
/// every `min ls` for the probe's full deadline (710d: 2 s per call
/// through a stalled clone, on every distro).
pub async fn get_attrs(&self) -> Option<HostAttrs> {
let (send, recv) = oneshot::channel();
// Ignore send errors - the recv will also fail.
let _ = self.0.send(BoxMessage::GetHostAttrs(send)).await;
recv.await.ok().flatten()
let host = self
.2
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.clone();
probe_host(host?.get_attrs()).await
}

/// Test-only: puts `host` in the slot a listing reads, as the actor does
/// after a message; the actor's next message overwrites it.
#[cfg(test)]
pub(crate) fn put_host_for_test(&self, host: crate::box_runtime::HostHandle) {
*self
.2
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner) = Some(AttrsProbe(host));
}

/// Test-only: parks the actor for `d`, the shape of a message that holds
/// its mailbox (a clone inside `ConfigureLoadout`, an inline attach).
/// Returns once the actor is inside the park, so every earlier message
/// and the slot publish after it are done.
#[cfg(test)]
pub(crate) async fn park_for(&self, d: std::time::Duration) {
let (entered, entered_recv) = oneshot::channel();
#[expect(
clippy::let_underscore_must_use,
reason = "a gone actor cannot be parked, and the recv below says so"
)]
let _ = self.0.send(BoxMessage::Park(d, entered)).await;
entered_recv
.await
.expect("the session actor should enter the park");
}

/// Returns the workspace's at-risk report (what a destroy would lose),
Expand Down Expand Up @@ -734,14 +806,19 @@ impl BoxHandle {
pub struct WeakBoxHandle(
pub(super) crate::traced::WeakSender<BoxMessage>,
pub(super) CancellationToken,
pub(super) HostSlot,
);

impl WeakBoxHandle {
/// Promotes to a strong [`BoxHandle`], or `None` if the session actor
/// has already shut down (all strong senders dropped).
#[must_use]
pub fn upgrade(&self) -> Option<BoxHandle> {
Some(BoxHandle(self.0.upgrade()?, self.1.clone()))
Some(BoxHandle(
self.0.upgrade()?,
self.1.clone(),
Arc::clone(&self.2),
))
}

/// A dangling handle whose actor is already gone (`upgrade` always yields
Expand All @@ -754,7 +831,7 @@ impl WeakBoxHandle {
let weak = tx.downgrade();
// Drop the only strong sender so `upgrade()` returns `None`.
drop(tx);
Self(weak, CancellationToken::new())
Self(weak, CancellationToken::new(), HostSlot::default())
}

/// A handle to a stand-in "actor" that answers each expose request with
Expand Down Expand Up @@ -792,6 +869,6 @@ impl WeakBoxHandle {
}
}
});
Self(weak, CancellationToken::new())
Self(weak, CancellationToken::new(), HostSlot::default())
}
}
7 changes: 7 additions & 0 deletions crates/minimald/src/box_actor/host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,10 @@ impl BoxActor {
BoxInner::Active { host, sops, .. } => Some((host.take(), std::mem::take(sops))),
BoxInner::Draft { .. } => None,
};
// The host is out of the actor's hands now: out of the slot too, so a
// handle that outlives a `Stop` reads no host rather than probing the
// one being stopped.
self.publish_host();
if let Some((host, mut sops)) = inner {
for s in sops.drain(..) {
s.shutdown().await;
Expand Down Expand Up @@ -750,6 +754,7 @@ impl BoxActor {
// Minted for an exec: a command is about to run in this sandbox, so a
// later attach must not replace it. See [`HostOrigin::Exec`].
self.host_origin = HostOrigin::Exec;
self.publish_host();
Ok(host)
}

Expand Down Expand Up @@ -828,6 +833,7 @@ impl BoxActor {
// Minted by an attach: its environment describes the terminal that is
// here, so nothing may replace it out from under that client.
self.host_origin = HostOrigin::Interactive;
self.publish_host();
outcome
}

Expand Down Expand Up @@ -983,6 +989,7 @@ impl BoxActor {
// attach may replace this shell rather than inherit its blank `TERM`.
// See [`HostOrigin::Hooks`].
self.host_origin = HostOrigin::Hooks;
self.publish_host();
Ok(())
}
}
3 changes: 2 additions & 1 deletion crates/minimald/src/box_actor/kind.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ impl BoxMessage {
Self::GetPaths { .. } => "GetPaths",
Self::MakeContext { .. } => "MakeContext",
Self::Attach { .. } => "Attach",
Self::GetHostAttrs { .. } => "GetHostAttrs",
Self::EnsureHost { .. } => "EnsureHost",
Self::GetWorkspaceDelta { .. } => "GetWorkspaceDelta",
Self::GetHostScreen { .. } => "GetHostScreen",
Expand Down Expand Up @@ -42,6 +41,8 @@ impl BoxMessage {
#[cfg(test)]
Self::PeekPendingAsks(_) => "PeekPendingAsks",
#[cfg(test)]
Self::Park(..) => "Park",
#[cfg(test)]
Self::PeekComposition { .. } => "PeekComposition",
#[cfg(test)]
Self::PeekIngressCells(_) => "PeekIngressCells",
Expand Down
8 changes: 5 additions & 3 deletions crates/minimald/src/box_actor/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ const GIVE_UP: Duration = Duration::from_secs(600);

/// A probe of a host that never answers has to end on its own deadline.
///
/// The session actor forwards `GetHostAttrs`/`GetHostScreen` to a spawned
/// The session actor forwards `GetHostScreen` (and, through the host slot,
/// the listing's attrs probe takes the same bounded path) to a spawned
/// task so the actor stays responsive, but nothing cancels that task:
/// dropping the caller's receiver does not stop a spawned future. Left
/// unbounded, every poll of a wedged host stranded one task forever — and
Expand Down Expand Up @@ -1866,7 +1867,7 @@ async fn missing_sidecar_falls_back_to_baseline() {
/// session sitting idle with nobody attached must still be able to launch
/// a host, and a second request must reuse it rather than starting a
/// second shell in the same session.
#[tokio::test]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn ensure_host_launches_an_unattached_host_and_then_reuses_it() {
let server = TestServer::new().await;
let mut client = server.connect().await;
Expand All @@ -1891,7 +1892,8 @@ async fn ensure_host_launches_an_unattached_host_and_then_reuses_it() {
assert!(host.is_alive());
assert!(
handle.get_attrs().await.is_some(),
"the session should now be holding the host it launched"
"the session should now be holding the host it launched: the slot is \
published before `ensure_host` replies, on a multi-thread runtime too"
);

let again = handle
Expand Down
Loading
Loading