Multiplexer backends
cafleet hosts every coding-agent member inside a terminal-multiplexer pane.
The multiplexer is abstracted behind the Multiplexer Protocol, so the
spawn, keystroke-delivery, capture, and teardown paths are backend-neutral. Two backends ship today: tmux and
herdr (herdr.dev). Both satisfy the same Protocol, so
every member * path behaves identically regardless of which one is active.
Pane ids are treated as opaque strings end to end — tmux ids look like %7,
herdr ids look like w1:p1; cafleet stores and passes them verbatim and never
parses them.
The pane is also cafleet's only push channel: message delivery stays pull-based
(recipients drain the persisted queue with cafleet message poll), and the
broker keystrokes an inline preview into the recipient's pane after
persisting a message — see Push notifications.
Backend matrix
Each backend invokes its CLI from PATH. Their behavior differs as follows:
TmuxError and HerdrError are both subclasses of MultiplexerError. Each
behavior's full contract stays under its own heading below.
Backend selection
Every call site resolves its backend through one shared resolver rather than hardcoding one. Resolution precedence:
- Explicit override. If
CAFLEET_MULTIPLEXERis set, it must name a supported backend (tmuxorherdr); an unknown value fails loudly. - Auto-detect from the environment.
HERDR_ENVtruthy signals a herdr session;TMUXset signals a tmux session. - Ambiguity is a hard error. Both
HERDR_ENVandTMUXset → error (setCAFLEET_MULTIPLEXERto disambiguate). Neither set → error (run cafleet inside a tmux or herdr session, or setCAFLEET_MULTIPLEXER). Exactly one present → that backend.
The outcomes of that order:
Auto-detect (an unset CAFLEET_MULTIPLEXER) is the default. cafleet doctor
reports the resolved backend and its
identifiers (see CLI options).
Error taxonomy
Backend failures share a base MultiplexerError, with the per-backend
subclasses in the backend matrix. CLI boundaries catch
MultiplexerError, so both backends' failures are handled uniformly while each
backend keeps its own message text.
Pane ownership during creation
split_window owns a newly created pane until it returns successfully. As
soon as herdr extracts the pane id from the split response, it arms a pane
guard. If the subsequent pane run fails, that guard calls
kill_pane(id, true) before returning the original run error. A failed close
retains both the pane id and the run error and adds the cleanup failure.
Layout equalization remains best-effort and does not fail an otherwise
successful spawn.
Creation errors carry internal cleanup metadata:
On success, ownership transfers immediately to the CLI's creation guard. A
fleet callback installs that guard and returns the id without another
fallible operation between those steps. Guard finish/rollback explicitly
disarms ownership; Drop is a last defense for unhandled ownership, so an
explicit cleanup attempt cannot cause a second kill. CLI registration and
transaction compensation follow the
creation failure order.
Creation compensation uses kill_pane, because send_exit submits a command
to the pane and does not guarantee shell or pane termination. Normal
member delete and notification keystrokes keep their existing behavior.
Subprocess output and deadlines
The shared subprocess runner captures stdout and stderr without truncation. With a timeout, it drains both pipes concurrently from spawn using nonblocking I/O. Each loop checks the monotonic deadline and the direct child's exit status, then reads at most 64 KiB per stream; polling waits at most 20 ms or the remaining deadline, whichever is shorter. Interrupted operations retry against the same deadline. Completion requires both the child's exit and EOF on both streams. Success returns stdout as lossy UTF-8; a nonzero exit reports stderr as lossy UTF-8. Calls without a timeout retain the unbounded output-collection path.
At the deadline, the runner kills and reaps the direct child, closes both read pipes, and reports a timeout. This also applies when the direct child has exited but a descendant still holds a pipe open. Descendant termination is outside this contract. FD setup, read, poll, and child-status failures also release the pipes and kill/reap the child; cleanup failures accompany the primary error instead of replacing it. The deadline bounds observation and the start of cleanup; an unresponsive operating system can delay cleanup itself.
Native agent-state (herdr only)
herdr natively tracks each agent's lifecycle state
(working/blocked/done/idle/unknown), exposed through a separate
optional capability Protocol, AgentStateAware, that only the herdr backend
implements — the base Multiplexer Protocol stays clean and tmux implements
nothing new.
No DB column backs the native status; supervision does not consume it — the monitor loop's wake is unconditional and periodic. See Monitoring.
The monitor wake and the fixed direct ping
The monitor loop is one fixed-cadence scan → wake → sleep path on both
backends. Its only keystroke target is the monitor member's own pane; it
never calls send_poll_trigger and never keystrokes any other pane.
send_wake_trigger receives the fleet's wake roster plus a Director
descriptor and emits a pure trigger — the member list with
pending-delivery counts, the Director: segment, the pointer to the monitor
role protocol, and the resume clause; nothing else. Each rendered entry is
<member-id> (<name>; coding_agent=<agent>; unacked=<pending-count>), joined
by , , ordered by member_id ascending, excluding the Director and the
monitor member itself; <pending-count> counts that member's
input_required unicast deliveries. The Director: segment is always
present, in the same field grammar as an entry. Every <name> is passed
through sanitize_wake_field; an entry (roster or Director) whose
coding_agent is not a supported backend name fails the wake closed with
member <id> has invalid coding_agent '<agent>' and no keystroke is sent.
tmux and herdr emit the payload byte-identically:
With N == 1 the noun is singular (health-check your 1 member: …); with
N == 0 the clause becomes no members to health-check. and there is no
<entries> segment — the Director: segment stays.
Example:
The Director's <name> renders as stored — fleet create registers the root
Director as Director — with no case transformation.
cafleet member ping is a fixed manual primitive of the Director and the
monitor member, unchanged on both backends: Esc, then the literal payload
cafleet message poll <member-id> — then resume your work if something was still running., then Enter (a
pending-placement target skips the keystroke and succeeds). It cannot carry
arbitrary text.
Anything a member needs from the Director travels as a plain
cafleet message send — the same persisted queue and Esc-safeguarded
inline-preview path every fleet message uses.
Pane spawn working directory
A member pane spawned by cafleet member create starts in the invoking
process's working directory (the Director's pane cwd); each backend realizes
this differently, per the backend matrix.
On herdr, <dir> is the invoking process's current working directory. herdr's own
inheritance is not relied upon because herdr spawns /bin/sh instead of the
passwd login shell when SHELL is unset
(herdr discussion #1517).
An unresolvable cwd fails the spawn loudly with HerdrError; there is no
fallback directory.
Delete-time pane layout
Closing a member pane leaves the two backends asymmetric on layout reflow, per the backend matrix.
herdr's kill_pane reads the target pane's tab (herdr pane get) before the
close, runs herdr pane close, then rebalances. The scoping comes from the
layout read itself: the killed pane is gone, so the rebalance picks a pane
still open in that tab (herdr pane list) and anchors the geometry read on it
(herdr pane layout --pane <surviving>), which returns that tab's layout
regardless of which tab or pane holds focus. When no pane remains in the tab,
there is nothing to rebalance and the step is skipped. With ≥ 2 members
remaining, the member column is re-equalized to equal heights (the same
invariant the create path enforces); after the last member is deleted, the
Director pane is explicitly restored to full tab width when the layout read
shows a residual right split; a single remaining member needs no resize. The
rebalance silently skips on unexpected layout shapes. Any HerdrError during
the rebalance is swallowed: a layout failure never fails member delete — the
pane is closed and the member deregistered regardless.
Prompt dispatch (send_prompt)
cafleet member prompt delivers its keystrokes through the multiplexer
interface's send_prompt(target_pane_id, text, shell = false) operation.
Both backends validate fail-fast: text empty after strip →
send_prompt: text may not be empty; the original text containing \n or
\r → send_prompt: text may not contain newlines (raised as the backend's
native error type, TmuxError / HerdrError). Both forms use the same Esc
safeguard and failure semantics; the shell flag controls only whether the
payload has the ! prefix. The per-backend payloads are in the
backend matrix. Both herdr forms mirror
send_poll_trigger's esc-then-run shape.
Push notifications
CAFleet's delivery model is pull-based: recipients discover messages via
cafleet message poll. To cut latency, the broker keystrokes a 2-line inline
preview into the recipient's pane immediately after persisting a message, so
the recipient's coding agent consumes it as a fresh user-turn input:
The keystroke is dispatched through the resolved backend's
send_inline_preview helper; the contract — one Esc-safeguarded submit of the
whole 2-line payload — is identical on both, and the per-backend realization is
in the backend matrix.
Inline-preview error propagation
send_inline_preview is the one keystroke path that propagates its failure as
a Result carrying the raw backend error instead of a best-effort boolean.
A missing backend binary fails with exactly tmux binary not found on PATH or
herdr binary not found on PATH; a subprocess failure after that precheck
carries the backend's existing raw error formatting — the failed command, its
payload argv, and a newline-delimited stderr detail — from whichever Escape,
payload, or Enter operation failed. The string is preserved verbatim all the
way to the caller.
The other trigger keystrokes, send_poll_trigger and send_wake_trigger,
keep their best-effort boolean contract: they never raise, and any failure
returns false.
Callers consume the inline-preview Result asymmetrically:
The recipient pane is resolved from member_placements by member_id alone,
so Member → Director notifications work automatically. The recipient acks via
cafleet message ack <message_id> once it has consumed the message.
Body truncation in the preview (… at CAFLEET_MAX_TEXT_LEN codepoints) is
documented in CLI options.
The Esc safeguard
Every keystroke path presses Escape, lets the pane settle ~0.1 s, then types
the payload and Enter.
Delivery outcomes
The persisted queue remains authoritative. Notification is attempted at most
once; a failed push leaves the existing row available for polling and ACK.
The recipient pane comes from its placement, so member-to-Director messages
use the same path. Backend resolution follows Backend selection;
keystrokes target the opaque pane id on the same host with a reachable server.
Direct targeting uses tmux send-keys -t <pane> or herdr pane send-*.
Broadcast wrapper fields recipients and delivered count intended recipients
and successful previews. Neither count is persisted. CLI wrappers and exact
partial-failure output are owned by Output shapes
and message send.