← reports

#1363 · Provider processes need one host-daemon lease owner tied to active turns

Type: not set (author's characterization: architecture / bug) host Priority: not set Effort: not set (large) open on GitHub 2026-08-18 base 16ceb3a540f81c1189efaffb27a39b1d9443abf5 (main)

Verdict: PARTIALLY REPRODUCED · root-cause confidence: high (for the mechanisms below) / low (for the reporter's exact 17-process count on 0.36.1) · linked open PRs: none

TL;DR

The reporter saw, on bb-app 0.36.1, 17 resident codex app-server processes (1.26 GiB) plus 7 code-mode hosts for only 4 active threads, and asks for one host-daemon owner that leases every provider process to an active turn or a bounded idle TTL and kills the process group when the lease ends. I could not reproduce that exact ratio on main (the Codex plumbing was rewritten between 0.36.1 and main: per-thread bridge + per-thread app-server, #1640), but I verified the structural claim: there is no single lease owner. Provider lifetime is decided by three independent mechanisms with gaps between them:

  1. The idle reaper (apps/host-daemon/src/app.ts, every 5 min, 30 min idle) is the only automatic release, and by default it only applies to Codex. With the shipped default (providerSessionReaping experiment off) every idle Claude Code / ACP / Pi session stays resident until the thread is manually stopped or the daemon restarts. Verified live: 3 idle claude threads = 3 resident claude processes (~1.05 GB RSS) for >33 min while codex peers were reaped at 31 min. Unit repro test fails on main.
  2. The reaper only sees "idle" state the daemon itself tracks. A thread whose provider hangs mid-turn (I froze an app-server with SIGSTOP) is never a reap candidate: server shows stopping forever after bb thread stop times out, the daemon keeps the active turn, and the app-server + code-mode host stay resident indefinitely. There is no maximum turn duration anywhere; the server's own comment says settlement "rests on" the turn ending by itself.
  3. Archive releases Codex but not Claude Code: archiving an idle claude-code thread leaves its 350 MB claude process resident (archive only forwards to providers with threadArchive: true; the "stop active work" call skips idle threads).
  4. Two smaller items: RuntimeManager.evictIdleEnvironments() has had no caller since May 2026, so per-environment shared bridge processes never unload; and the codex bridge's SIGTERM/stdin-close path calls process.exit(0) right after sending SIGTERM to children, so its 4 s SIGKILL escalation never fires — a child that cannot act on SIGTERM survives daemon death as an orphan (observed: codex app-server + codex-code-mode-host reparented to PID 1 after kill -9 of the daemon).

What does work on main: Codex idle reap (verified at 31 min), archive of an idle Codex thread (bridge, app-server and code-mode host all exit), bb thread stop on an idle thread releases any provider (manual release verb from #1584), bridge/app-server crashes cascade cleanly (no orphans), and daemon restart cleans up everything that is not wedged.

Claims vs findings

ClaimStatusEvidence
17 Codex app-servers / 7 code-mode hosts for 4 active threads on 0.36.1UnverifiableReporter's production numbers; the pre-#1640 code path (packages/agent-runtime/src/codex/adapter.ts in 0.36.0) is gone on main. Consistent with the mechanisms below (idle <30 min, hung/pending turns, threads with in-flight ops are all non-reapable), but I could not measure his host.
Restart clears the processes (cleanup on restart works)VerifiedDaemon restart via bb-dev-app and via kill -9 removed every bridge / app-server / claude process except one wedged app-server (exp. G; the "wedged" child was an app-server I had frozen with SIGSTOP as an artificial stand-in for a hang, so this exception does not occur for a healthy child).
Steady-state provider ownership is not bounded by an invariantVerifiedExp. C (claude never reaped), exp. E (hung turn never reaped), exp. F (archive leaves claude resident); code walk in Root cause.
Reactive idle reaping existsVerifiedCodex sessions reaped 31 min after idle in exp. B (Reaped idle provider sessions log lines).
Operator cannot map PID → session → thread/turnVerifiedNo local-API/CLI diagnostic exists (bb machine show prints host row only). Codex app-servers carry no BB_THREAD_ID env (only claude CLIs do); daemon-side listActiveThreads() is only sent at session open.
124 host-daemon sessions (100 replaced, 21 daemon-disconnect)UnverifiableReporter's DB. Note: a same-instance reconnect (replaced) does not touch provider processes (closeDaemonSessionSocket, not closeDaemonSession) — so churn there neither leaks nor frees processes.
OOM kill of bb.service at 4 GiBUnverifiableNot reproduced; my instance held 2.6 GB RSS in provider processes with 4 codex + 2 claude threads (dev-mode tsx bridges are heavier than production bundles).

Environment

Glossary (process shapes you will see in procs.sh output)

Minimal reproduction

Precondition: from a bb worktree at the base commit run pnpm install --frozen-lockfile --prefer-offline && pnpm exec turbo run build && scripts/bb-dev-app current. Then, with your shell's cwd inside that worktree (or BB_WT=/path/to/bb exported), copy 1363/repro/*.sh anywhere and run them by absolute path; nothing in them is machine-specific. spawn.sh creates the qa project (local path /tmp/bb-1363-qa) on first use and passes --machine $(bb_host_id) because bb thread spawn without --machine can resolve a different host on a box with several bb instances (HTTP 404: Host not found). Its first output line prints server=… host=… project=…; the ad-hoc bb thread spawn commands below use those values as $HOST / $PROJECT. The bb function defined by env.sh is node $WT/packages/scripts/dist/commands/run-cli.js. Where the transcripts below show only $ bash 1363/repro/spawn.sh …, read it as bash /tmp/bb-reports/issues/1363/repro/spawn.sh … from inside the worktree.

A. Footprint per thread on main

$ bash 1363/repro/spawn.sh codex 3 c        # 3 codex threads, "Reply only with ok."
$ bash 1363/repro/procs.sh
daemon pid: 1493947  (05:22:17)
PID      PPID     RSS_KB    ETIME        ARGS
1558053  1493947  210388    00:20        [bridge] provider-codex        <- provider-maintenance runtime bridge (cwd provider-maintenance-workspace)
1558551  1493947  208524    00:19        [bridge] provider-codex        <- c-1
1558611  1558551  171796    00:19        codex app-server
1562935  1493947  207104    00:11        [bridge] provider-codex        <- c-2
1563195  1562935  176508    00:10        codex app-server
1563570  1493947  209292    00:10        [bridge] provider-codex        <- c-3
1563675  1563570  184264    00:10        codex app-server
--- counts:
bridges:      4
codex app-server: 3

Per Codex thread on main: one node bridge process (codex-thread:<threadId>#bridge:… process key) + one codex app-server; a turn that runs a shell command adds a codex-code-mode-host child (~20 MB). Per Claude Code thread: one claude CLI process (~350 MB) under a single per-environment bridge. Plus one permanent bridge per provider for the maintenance runtime. RSS at 05:37 with 4 codex + 2 claude sessions resident (procs-05-40.txt): codex-bridge n=6 780 MB · codex-app-server n=4 713 MB · claude-cli n=2 684 MB · claude-bridge n=2 382 MB · TOTAL 2649 MB (dev-mode tsx bridges; production bundles are ~50 MB each).

B. Codex idle reap works (31 min)

c-1..c-3 went idle at 05:22:17. Sweep ticks every 5 min from daemon start (05:18:18). The daemon log (dev-log-excerpt.txt):

[05:53:18] INFO: [host-daemon] Reaped idle provider sessions {"count":3,"sessions":[
  {"environmentId":"env_thjnh8daw7","idleForMs":1876254,"providerId":"codex","threadId":"thr_jxeat766jj"},
  {"environmentId":"env_thjnh8daw7","idleForMs":1868549,"providerId":"codex","threadId":"thr_sswktt9hb6"},
  {"environmentId":"env_thjnh8daw7","idleForMs":1862713,"providerId":"codex","threadId":"thr_jcj8w4e3b4"}]}
[05:58:18] INFO: [host-daemon] Reaped idle provider sessions {"count":1,"sessions":[
  {"environmentId":"env_thjnh8daw7","idleForMs":1966503,"providerId":"codex","threadId":"thr_dufwvzmwci"}]}

After the sweep the three bridges and app-servers were gone (procs-05-54-after-reap.txt). Expected per the issue: yes. Actual: yes, for Codex only.

C. Claude Code sessions are never reaped (default policy) — the monotonic-growth case

$ bash 1363/repro/spawn.sh claude-code 3 cl     # cl-1..cl-3, "Reply only with ok.", all idle by 05:30:15
$ bash 1363/repro/procs.sh | tail -6
1637115  1493947  194360    03:56        [bridge] provider-claude-code   <- maintenance runtime
1637298  1493947  196928    03:54        [bridge] provider-claude-code   <- env_thjnh8daw7 (shared by all claude threads)
1638262  1637298  336192    03:54        /home/sawyer/.local/bin/claude --output-format stream-json ... (BB_THREAD_ID=thr_arj57gdfup)
1693532  1637298  356000    00:12        /home/sawyer/.local/bin/claude --output-format stream-json ...
1693897  1637298  357172    00:11        /home/sawyer/.local/bin/claude --output-format stream-json ...
claude:       3

Same snapshot at 05:59:29, i.e. after the codex reap ticks at 05:53 and 05:58 (procs-before-daemon-kill.txt): the two remaining claude CLIs (cl-1 idle 33 min, cl-3 idle 29 min) are still resident; only cl-2, which I released manually with bb thread stop in exp. D, is gone. Expected (issue's acceptance evidence "zero unmatched provider processes after the lease TTL"): the claude sessions released at ~30 min like codex. Actual: never released; the reaper returns nothing for non-Codex providers unless the providerSessionReaping experiment is on (docs/configuration.md L622). N idle claude threads ⇒ N × ~350 MB resident until daemon restart.

Unit-level repro (fails on main): 1363/repro/issue-1363-provider-lease.test.ts, also at packages/agent-runtime/src/issue-1363-provider-lease.test.ts in my worktree. Run: cd packages/agent-runtime && pnpm exec vitest run src/issue-1363-provider-lease.test.ts. Output (vitest-issue-1363.out):

FAIL  src/issue-1363-provider-lease.test.ts > issue #1363: idle provider sessions are bounded by a lease
  > releases every idle non-Codex session after the idle TTL under the shipped default policy
AssertionError: expected [] to deeply equal [ 't1', 't2', 't3' ]
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { createAgentRuntimeWithAdapters } from "./runtime.js";
import { createFakeAdapter, fullRuntimeOptions } from "./test/runtime-test-harness.js";

// Issue #1363: "Provider processes need one host-daemon lease owner tied to
// active turns". The issue's proposed invariant is that every resident
// provider session maps to an active turn or a bounded lease, and that after
// the lease TTL there are zero unmatched sessions.
//
// This test encodes that invariant against the runtime's only lease-like
// mechanism, `reapIdleProviderSessions`, using the policy the host daemon
// ships by default (`providerSessionReapingEnabled: false`, i.e. the
// "Idle provider session release" experiment OFF) and a 60-minute idle
// window (twice IDLE_PROVIDER_SESSION_REAP_AFTER_MS).
//
// It FAILS on main: for every provider except Codex, the sweep returns []
// no matter how long the session has been idle, so N idle threads keep N
// resident provider sessions (for claude-code: N `claude` CLI processes)
// until the thread is archived/stopped or the daemon restarts.

const RESTORABLE_PROVIDER_SCRIPT = `const fs = require("fs");
const readline = require("readline");
const rl = readline.createInterface({ input: process.stdin });
const threads = new Map();
let nextTurn = 1;
function send(m) { process.stdout.write(JSON.stringify(m) + "\\n"); }
process.on("SIGTERM", () => process.exit(0));
rl.on("line", (line) => {
  const msg = JSON.parse(line);
  const p = msg.params || {};
  if (msg.method === "initialize" || msg.method === "skills/configure") {
    send({ jsonrpc: "2.0", id: msg.id, result: { ok: true } });
    return;
  }
  if (msg.method === "thread/start") {
    const providerThreadId = "prov-" + p.threadId;
    threads.set(p.threadId, providerThreadId);
    // A graduated bridge (claude-code, acp, pi) reports sessionRestorable.
    send({ jsonrpc: "2.0", id: msg.id, result: { providerThreadId, sessionRestorable: true } });
    send({ jsonrpc: "2.0", method: "thread/identity", params: { threadId: p.threadId, providerThreadId } });
    return;
  }
  if (msg.method === "turn/start") {
    const providerThreadId = threads.get(p.threadId);
    const turnId = "turn-" + nextTurn++;
    send({ jsonrpc: "2.0", id: msg.id, result: { ok: true } });
    send({ jsonrpc: "2.0", method: "turn/started", params: { threadId: p.threadId, providerThreadId, turnId } });
    send({ jsonrpc: "2.0", method: "turn/completed", params: { threadId: p.threadId, providerThreadId, turnId, status: "completed" } });
    return;
  }
  if (msg.method === "thread/stop") {
    threads.delete(p.threadId);
    send({ jsonrpc: "2.0", id: msg.id, result: {} });
    return;
  }
  if (msg.id !== undefined) {
    send({ jsonrpc: "2.0", id: msg.id, result: {} });
  }
});
`;

describe("issue #1363: idle provider sessions are bounded by a lease", () => {
  let tmpDir: string;

  beforeEach(() => {
    tmpDir = mkdtempSync(join(tmpdir(), "bb-issue-1363-"));
  });

  afterEach(() => {
    rmSync(tmpDir, { recursive: true, force: true });
  });

  it("releases every idle non-Codex session after the idle TTL under the shipped default policy", async () => {
    const scriptPath = join(tmpDir, "restorable-provider.cjs");
    writeFileSync(scriptPath, RESTORABLE_PROVIDER_SCRIPT);
    const runtime = createAgentRuntimeWithAdapters({
      workspacePath: tmpDir,
      onEvent: () => {},
      onToolCall: async () => ({
        contentItems: [{ type: "inputText", text: "ok" }],
        success: true,
      }),
      adapterFactory: () => ({
        ...createFakeAdapter(scriptPath),
        displayName: "Claude Code",
        id: "claude-code",
      }),
    });

    try {
      const threadIds = ["t1", "t2", "t3"];
      for (const threadId of threadIds) {
        await runtime.startThread({
          environmentId: "env-1",
          threadId,
          projectId: "p1",
          providerId: "claude-code",
          options: fullRuntimeOptions,
        });
      }
      for (const threadId of threadIds) {
        expect(runtime.hasThread(threadId)).toBe(true);
        expect(runtime.getActiveTurnId(threadId)).toBeNull();
      }

      // Same arguments the host daemon's reaper passes
      // (apps/host-daemon/src/app.ts startIdleProviderSessionReaper), with the
      // experiment OFF (default) and the sessions idle for 60 minutes.
      const result = await runtime.reapIdleProviderSessions({
        idleForMs: 30 * 60 * 1000,
        nowMs: Date.now() + 60 * 60 * 1000,
        providerSessionReapingEnabled: false,
      });

      // Issue #1363 acceptance evidence: "Zero unmatched provider processes
      // after the lease TTL." On main this is [] and all three sessions stay
      // resident.
      expect(result.reapedSessions.map((s) => s.threadId).sort()).toEqual(
        threadIds,
      );
      for (const threadId of threadIds) {
        expect(runtime.hasThread(threadId)).toBe(false);
      }
    } finally {
      await runtime.shutdown();
    }
  });
});

D. Manual release works for any provider (#1584 verb)

$ bb thread stop thr_ze4x9aemtz        # cl-2, idle
Thread thr_ze4x9aemtz stopped
$ bash 1363/repro/procs.sh | tail -3    # 5 s later
1638262  1637298  337996    07:00        /home/sawyer/.local/bin/claude ...
1693897  1637298  355584    03:17        /home/sawyer/.local/bin/claude ...
claude:       2                          # was 3

E. A hung provider turn pins its process group forever (no turn lease)

$ source 1363/repro/env.sh; HOST=$(bb_host_id); PROJECT=<id printed by spawn.sh>
$ bb thread spawn --project $PROJECT --provider codex --permission-mode accept-edits --machine $HOST --title c-7 --prompt "Run the shell command 'sleep 60' then reply only with ok." --json
  "id": "thr_3tjcy6jkye"
$ bash 1363/repro/procs.sh | grep -A6 1734125     # while the command runs
1734125  1493947  210732    00:25        [bridge] provider-codex
1734149  1734125  208940    00:25        codex app-server
1735139  1734149  21348     00:19        .../codex-code-mode-host
1735197  1734149  17288     00:19        .../codex-linux-sandbox --sandbox-policy-cwd ...
1735205  1735204  7520      00:19        sleep 60
$ kill -STOP 1734149                               # simulate a wedged app-server, 05:34:27
$ bb thread show thr_3tjcy6jkye --json | grep status   # 90 s later
    "status": "active",
$ bb thread stop thr_3tjcy6jkye                    # 05:36:04
Thread thr_3tjcy6jkye stopped                      # returned 05:36:35 (server log: WARN Awaited thread stop command failed {"intent":"interrupt"})
$ bb thread show thr_3tjcy6jkye --json | grep status
    "status": "stopping",                          # still "stopping" at 05:59; both reap ticks (05:53, 05:58) skipped it
$ bash 1363/repro/procs.sh | grep -A3 1734125      # 05:59:29
1734125  1493947  135132    25:35        [bridge] provider-codex
1734149  1734125  208916    25:34        codex app-server        (STAT Tl)
1735139  1734149  21344     25:28        .../codex-code-mode-host

Expected (issue): a turn that "disconnects, errors, or exceeds the reuse TTL" ends the lease and the daemon terminates the process group. Actual: the daemon has no notion of a maximum turn age. The reaper's first check rejects any thread with getActiveTurnId(threadId) !== null, the server's stop is "dispatched once — no inline retry, no durable timer" and relies on the turn ending by itself, so bridge + app-server + code-mode host stay resident until the daemon restarts (which is what finally settled c-7 to idle at 05:59:39 through reconcileDaemonReportedThreads). SIGSTOP is a stand-in for any real hang (deadlock, stuck network read); the observable state is identical.

F. Archiving an idle Claude Code thread does not release its process

$ bb thread spawn --project $PROJECT --provider claude-code --permission-mode accept-edits --machine $HOST --title cl-4 --prompt "Reply only with ok." --json   # (or: bash 1363/repro/spawn.sh claude-code 1 cl)  -> thr_pnndii523q, idle 06:00:46
$ bb thread archive thr_pnndii523q
Thread thr_pnndii523q archived
$ ps -o pid,ppid,etime,rss,args -p 1858952          # 53 s later
1858952 1858924    00:53 350788 /home/sawyer/.local/bin/claude --output-format stream-json ...
$ ps -o pid,ppid,etime,rss,args -p 1858952          # 5.5 min later, still resident
1858952 1858924    05:36 353008 /home/sawyer/.local/bin/claude --output-format stream-json ...

Server-level repro test (fails on main): 1363/repro/issue-1363-archive-idle-runtime-release.test.ts (also at apps/server/test/threads/ in my worktree; run cd apps/server && pnpm exec vitest run test/threads/issue-1363-archive-idle-runtime-release.test.ts). It archives an idle thread through POST /api/v1/threads/:id/archive with a real in-memory DB and lists the host commands the server enqueued: for codex a thread.archive is queued (passes); for claude-code neither thread.archive nor thread.stop is queued — AssertionError: expected 0 to be greater than 0 (vitest-issue-1363-archive.out).

Re-run with the portable helper scripts on a second dev instance (revision pass, worktree wf_debcf606-e4a-50, server :23519, host host_hvuqh8u6ut; raw: rerun-06-18-spawn.txt, rerun-06-18-procs-after-spawn.txt, rerun-06-19-procs-after-archive.txt). Same shape: the codex thread's bridge + app-server are gone within 20 s of archive; the claude CLI stays.

$ bash /tmp/bb-reports/issues/1363/repro/spawn.sh codex 1 c
server=http://localhost:23519 host=host_hvuqh8u6ut project=proj_fixrjmwu3c
== spawn c-1 (codex) 06:18:21
Thread spawned: thr_ep5aj224vw
...
all idle at 06:18:33
$ bash /tmp/bb-reports/issues/1363/repro/spawn.sh claude-code 1 cl
== spawn cl-1 (claude-code) 06:18:37
Thread spawned: thr_gb3snsw5up
all idle at 06:18:45
$ bash /tmp/bb-reports/issues/1363/repro/procs.sh | grep -E "bridge|app-server|claude "
1879976  1878453  261932    00:23        [bridge] provider-codex          <- maintenance runtime
1880248  1878453  214060    00:22        [bridge] provider-codex          <- c-1
1880259  1880248  164680    00:21        codex app-server
1881326  1878453  605444    00:07        [bridge] provider-claude-code    <- maintenance runtime
1881408  1878453  287376    00:05        [bridge] provider-claude-code    <- env_irxtwkf8y3
1881431  1881408  364428    00:05        /home/sawyer/.local/bin/claude --output-format stream-json ...   (BB_THREAD_ID=thr_gb3snsw5up)
$ bb thread archive thr_ep5aj224vw ; bb thread archive thr_gb3snsw5up      # 06:18:58
$ bash /tmp/bb-reports/issues/1363/repro/procs.sh | grep -E "bridge|app-server|claude "   # 06:19:19
1879976  1878453  261936    00:57        [bridge] provider-codex          <- maintenance runtime only
1881326  1878453  605456    00:41        [bridge] provider-claude-code
1881408  1878453  287376    00:39        [bridge] provider-claude-code
1881431  1881408  364472    00:39        /home/sawyer/.local/bin/claude --output-format stream-json ...   <- still resident
$ ps -o pid,ppid,etime,rss,args -p 1881431      # 06:21:02
1881431 1881408       02:22 363236 /home/sawyer/.local/bin/claude --output-format stream-json ...

Contrast with Codex: archiving idle c-4 (thr_k2tifuq2s7, which had spawned a code-mode host) removed bridge, app-server and code-mode host within 8 s. Reason: archiveThreadWithLifecycleEffects only calls requestActiveRuntimeThreadStopIfNeeded (no-op for idle) and dispatchSettledArchivedThreadProviderArchiveCommand, which returns early unless providerSupportsThreadArchiveForwarding — true for the codex bridge (threadArchive: true), false for claude-code.

G. Crash cascades and the one orphan path

Root cause

There is no lease; there are three uncoordinated release triggers, each with a hole.

1. The idle reaper is provider- and state-gated, and off by default for everything but Codex. app.ts#L66-L67 sweeps every 5 min for 30-min-idle sessions and reads the experiment flag from the server before each pass (app.ts#L714-L718). In the runtime, findReapableIdleProviderSession rejects a thread when any of these hold:

threadHasInFlightOperation(threadId) || pendingTurnStartThreadIds.has(threadId) || turnState.getActiveTurnId(threadId) !== null   // → never idle
!runtimeConfig || (providerSessionReapingEnabled ? !runtimeConfig.sessionRestorable : runtimeConfig.providerId !== CODEX_PROVIDER_ID)
!providerThreadId || idleSinceMs === undefined || nowMs - idleSinceMs < idleForMs

and reapIdleProviderSessions additionally skips a candidate that has open background work (experiment on) or is not a thread-scoped Codex process (experiment off). So with the shipped default the sweep is literally providerId === "codex". Every other provider's session lives until thread.stop, thread.archive (Codex only, see 3), or daemon shutdown. Exp. C and the failing unit test show this. #1604 reported the same gap; the experiment in 3bc9ce54b made it opt-in rather than closing it.

2. The reaper's inputs are the daemon's own turn state, and nothing bounds a turn. Idle time is only recorded on turn/completed / non-retry provider/error (observeProviderSessionIdleState) or after startThread/resumeThread/a failed turn/start send. A provider that stops emitting events keeps turnState active forever, so the process is un-reapable forever. The only watchdog is the 120 s turn-start watchdog (runtime.ts#L381-L400), which only emits a system/error. On the server side, thread-lifecycle.ts#L1302-L1314 states the assumption explicitly: "turns are bounded, so the daemon's turn-completed/failed/interrupted event drives stopping → idle/error regardless of the stop RPC". Exp. E shows what happens when that assumption is false: stopping on the server, active turn in the daemon, resident process group, and no timer anywhere that ends it. This is the reporter's "one observable invariant" gap: thread status (server), turn state (daemon), and process (bridge child) can disagree indefinitely.

3. Archive releases only providers that forward archive. thread-archive.ts#L55-L64 calls requestActiveRuntimeThreadStopIfNeeded (returns unless status === "active", L1607) and dispatchSettledArchivedThreadProviderArchiveCommand, which returns false unless providerSupportsThreadArchiveForwarding (thread-commands.ts L589-L596). Codex forwards (bridge.ts#L1302) and the runtime then runs forgetThreadRuntimeState + releaseIdleProviderProcess (runtime.ts#L1014-L1019); Claude Code does not, so an archived idle claude thread keeps its 350 MB CLI (exp. F).

4. Environment runtimes never unload. RuntimeManager.evictIdleEnvironments() exists and is tested, but git log -S evictIdleEnvironments shows its last production caller was removed in 68565f7e4 ("Clean up sandbox host handling", May 2026). Per-environment shared bridges (claude-code, ACP, pi: one node process per environment per provider) therefore persist after their last thread is released. Bounded by environment count, but with many worktrees it is the same shape as the reporter's "processes outnumber threads". (Consistent with the read-only census of the user's real daemon on this machine: 12 bb-claude-code-bridge.mjs processes, 2 with a live claude child.)

5. Kill is not process-group kill. Bridges spawn codex app-server with a plain spawn() (no detached/setsid) and kill() signals the child pid only (app-server-connection.ts#L332-L344); code-mode hosts and sandboxed commands die only because they watch their own parent/pipes. And bridge.ts#L1914-L1930 exits before the SIGKILL escalation can run (exp. G).

Why the reporter saw 17 for 4. On 0.36.1 (pre-#1640) each Codex thread was one directly-spawned app-server with the same reap rules (runtime.ts at desktop-v0.36.0 L1628-L1668). Anything idle <30 min, anything with a pending/hung turn, anything with an in-flight RPC, and every non-Codex session was resident. Without a PID→thread diagnostic (none exists; Codex children carry no BB_THREAD_ID) the operator cannot tell which bucket each of the 13 extra processes was in — which is exactly the issue's point.

Proposed fix (first principles)

  1. Make the runtime the lease owner and make the lease provider-agnostic. In createAgentRuntimeInternal keep one record per provider session: {threadId, providerThreadId, processKey, leaseKind: "turn"|"idle", leaseUntilMs}. Refresh on every provider event for the thread. Replace the providerSessionReapingEnabled ? !sessionRestorable : providerId !== codex gate with: restorable ⇒ release at idle TTL (default on; keep the experiment only as an opt-out or delete it and update docs/configuration.md); non-restorable ⇒ still release at a longer TTL and mark the thread so the next turn starts a fresh session with a visible notice (the ACP fallback already does this). Risk: providers whose resume loses in-agent context — that is why the flag exists; ship it default-on for claude-code/codex (both restorable) first.
  2. Bound turns. Add a per-thread "no provider event for N minutes" watchdog (N configurable, e.g. 30 min, on top of the existing 120 s turn-start watchdog): emit turn/completed {status:"failed", error: provider_stalled}, then stopThread with release so the server's stopping settles and the process group is torn down. This is the missing "turn finishes, disconnects, errors, or exceeds the TTL" arm of the invariant. Risk: legitimately silent long tool calls; make the threshold long and reset it on any event including background-work notifications.
  3. Kill process groups. Spawn app-server / CLI children with detached: true (own pgid) and kill with process.kill(-pid, sig); in the codex bridge, await the escalation (or send SIGKILL directly) before process.exit(0) in onClose/onSigterm. Same for the runtime's terminateProviderProcess.
  4. Release on archive for every provider. In archiveThreadWithLifecycleEffects, when the thread is idle and its runtime is loaded, dispatch thread.stop {intent:"release"} (the existing releaseIdleThreadRuntime) when archive forwarding is unsupported. Server-side, no wire change.
  5. Call evictIdleEnvironments() from the same 5-minute maintenance pass after the reap (an entry with no live threads, no terminals and no background work). Existing tests cover the method.
  6. Diagnostics. Add a daemon local-API endpoint (and bb machine show <id> --processes) listing pid → providerId → processKey → threadIds → lease, sourced from RuntimeProviderProcessManager; set BB_THREAD_ID on codex app-server children as claude does. This makes the invariant observable and turns the reporter's next audit into a query.

Items 1, 4 and 5 are daemon/server policy changes with no protocol change; item 2 adds a new failure code but no new wire fields; none of them require a HOST_DAEMON_PROTOCOL_VERSION bump unless a diagnostic RPC (item 6) is added to the host command set, in which case it must be bumped.

PR review

No open PRs are linked to this issue.

Related issues

Appendix

Timeline of the live session (UTC)

05:18:18  daemon started (pid 1493947); reaper ticks every 5 min from here
05:21:23  c-1..c-3 (codex) spawned; idle 05:22:17               -> 3 bridges + 3 app-servers (+1 maintenance bridge)
05:23:03  c-4 spawned with 'echo hi' -> code-mode host appears; archived 05:23:57 -> all 3 processes gone by 05:24:05
05:24:13  c-5 spawned; kill -9 its bridge 05:25:0x -> app-server + code-mode host gone; thread idle
05:25:2x  c-6 spawned; kill -9 its app-server -> code-mode host gone, bridge stays; reaped 05:58:18
05:26:2x  cl-1 (claude) spawned; 05:30:07 cl-2, cl-3 -> 3 claude CLIs (~350 MB each)
05:31:5x  bb thread stop cl-2 -> its claude CLI released
05:34:0x  c-7 spawned with 'sleep 60'; 05:34:27 SIGSTOP app-server; 05:36:04 bb thread stop -> WARN Awaited thread stop command failed; status stopping
05:53:18  reaper: c-1, c-2, c-3 released (idleForMs ~1.87e6)
05:58:18  reaper: c-6 released
05:59:29  snapshot: cl-1, cl-3 claude CLIs still resident (33 / 29 min idle); c-7 group resident
05:59:30  kill -9 daemon -> all bridges/claude exit; c-7 app-server + code-mode host orphaned to PID 1; supervisor restarts daemon 05:59:39; c-7 settles idle
06:00:46  cl-4 spawned; archived; claude CLI still resident 53 s later

Commands run

pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build
scripts/bb-dev-app current                                  # app :12352, server :20352, daemon :28352
BB=node\ packages/scripts/dist/commands/run-cli.js  BB_SERVER_URL=http://localhost:20352
$BB thread spawn --project proj_9tmcdwmi7t --provider codex --permission-mode accept-edits --machine host_vp62yvdkp4 --title c-1 --prompt "Reply only with ok." --json
# from inside the bb worktree (scripts derive URLs/host/project from `scripts/bb-dev-app status` + `bb machine list --json`):
bash /tmp/bb-reports/issues/1363/repro/spawn.sh codex 3 c ; bash /tmp/bb-reports/issues/1363/repro/spawn.sh claude-code 3 cl
bash /tmp/bb-reports/issues/1363/repro/procs.sh ; bash /tmp/bb-reports/issues/1363/repro/rss-sum.sh
$BB thread archive thr_k2tifuq2s7 ; $BB thread stop thr_ze4x9aemtz ; $BB thread stop thr_3tjcy6jkye
kill -9 <bridge pid> ; kill -9 <app-server pid> ; kill -STOP <app-server pid> ; kill -9 <daemon pid> ; kill -CONT <orphan>
grep "Reaped idle provider sessions" ~/.bb-dev/launchers/projects-bb-.claude-worktrees-wf_debcf606-e4a-16/dev.log
cd packages/agent-runtime && pnpm exec vitest run src/issue-1363-provider-lease.test.ts
git log -S evictIdleEnvironments ; git show desktop-v0.36.0:packages/agent-runtime/src/runtime.ts
pnpm dev:stop

Raw artifacts

Notes on the pre-#1640 code path (0.36.1)

git show desktop-v0.36.0:packages/agent-runtime/src/runtime.ts has the same reaper shape (findReapableIdleProviderSession with runtimeConfig?.providerId !== CODEX_PROVIDER_ID, reapIdleProviderSessions gated by isThreadScopedCodexProcess) with the app-server spawned directly by the daemon rather than through a per-thread bridge. The provider-session release experiment (3bc9ce54b) is in 0.38.0 but not 0.37.0.

Verification

An independent verifier set up their own worktree at 16ceb3a54 and dev instance (server :21825, host host_zm5j79zai4) and followed this report. Both unit repros reproduced exactly (expected [] to deeply equal [ 't1', 't2', 't3' ]; archive test codex passes / claude-code fails with expected 0 to be greater than 0). Live: one claude-code + one codex thread produced the process census described in exp. A; after archiving both, the codex bridge and app-server were gone within 19 s while the claude CLI (~360 MB) stayed resident (exp. F confirmed). The verifier did not wait for the 30-min reap ticks or repeat the SIGSTOP / kill -9 experiments (B, E, G), which therefore rest on the author's log excerpts (dev-log-excerpt.txt) and are consistent with the code walk. Verifier findings and what changed in this revision: