← reports

#1660 · bb host process grew to 77GB RSS and froze the machine; orphaned dev processes in deleted managed worktrees compound the leak

Type: Bug / feature request (3 asks) Priority: High Effort: Medium–Large perf workspaces host open on GitHub investigated 2026-08-18 (revised same day after independent verification; third verification pass 06:25 UTC re-ran the unit repro and re-checked every code permalink) base commit 16ceb3a540f81c1189efaffb27a39b1d9443abf5 (main)

Verdict: PARTIALLY REPRODUCED. The orphaned-process claim (request 2) is reproduced live and with a failing unit test: bb's environment teardown removes the managed worktree directory and kills only its own direct provider child; anything an agent left running from that directory survives with a (deleted) cwd. The lack of resource telemetry (request 3) is verified from code and CLI. The 77 GB RSS growth of "the bb host process" (request 1) is not reproduced — it needs a day of macOS multi-thread load I cannot replay — but code review shows no RSS ceiling, no RSS warning on macOS, and at least one deliberately unbounded in-memory queue in the daemon.
Root-cause confidence: high for the orphan mechanism, low for the specific memory-growth mechanism.

TL;DR

A user running many agent threads a day saw one bb process reach ~77 GB RSS and lock the Mac, and during recovery found 88 stray dev servers/watchers whose working directory pointed at worktrees bb had already deleted. Two different things are going on. (1) When bb archives the last thread of a managed worktree, the host daemon shuts down its provider process (the bridge it spawned) and then git worktree removes + rm -rfs the directory. Nothing enumerates or signals other processes rooted in that directory, so anything the agent started with &/nohup/a tool's background mode (dev servers, watchers, test runners) keeps running from a deleted inode forever. I reproduced this on main in six minutes with a real Claude Code turn. (2) Separately, the daemon and server have no memory ceiling and no memory telemetry: the only resource monitor in the daemon reads RSS but only warns when the Linux inotify count is high, so on macOS a leak is invisible until the machine swaps. I could not reproduce the 77 GB itself; the report gives no heap snapshot or process name, and V8's default heap cap makes a 77 GB JS heap unlikely, so it was more likely native/external memory (Buffers, node-pty, better-sqlite3) or an unbounded queue — the daemon's outbound event queue is explicitly unbounded by design ("These only warn — they never drop, fault, or bound the queue").

Claims vs findings

Claim (from issue)StatusEvidence
"The main bb host process grew to ~77GB RSS over a day of heavy use and froze the machine."UnverifiedNo heap profile, PID, or process name in the report ("host process" could be the host daemon, the server, or the bb-app launcher — they are three separate node processes; see startFullStackServerProcess/startFullStackDaemonProcess in packages/bb-app/src/launcher.ts). Not reproducible in the timebox. Code review below lists candidate growth vectors; none proven.
"88 orphaned development processes … whose cwd pointed into deleted managed worktrees had accumulated."Verified (mechanism reproduced)Live repro: agent-spawned sleep 100000 survives environment.destroy, cwd shows … (deleted). Unit test at packages/host-workspace/test/issue-1660-orphan-process.test.ts fails on main at the assertion "process is dead after destroy". Same class independently reported in #1647/#1769 (190 processes / 1.75 GB from 40 destroyed envs).
"bb's environment lifecycle reclaimed the worktrees but nothing reaped the processes started from them."VerifiedremoveWorktree() only runs git worktree remove --force and fs.rm; RuntimeManager.destroyEnvironment() only calls runtime.shutdown() (SIGTERM/SIGKILL to the direct bridge child via child.kill, no process-group or cwd scan). No code path in the repo lists processes by cwd.
Bug "recurs; twice previously traced machine-wide memory/CPU exhaustion to this pattern (orphaned dev stacks ~1GB each …)"UnverifiableHistorical; consistent with #1647/#1769 measurements on other machines.
Request 1: "A heap/RSS ceiling with graceful degradation, or periodic compaction"Verified absentNo --max-old-space-size anywhere in the repo (grep). Health monitor reads process.memoryUsage().rss but never warns on it. See also #1748 (server heap sized from host RAM).
Request 2: "reap (or at least surface) processes whose cwd resolves into the pruned path … bb doctor --reap-orphans"Verified absentbb doctor does not exist (CLI prints help). No reaping or surfacing anywhere in the destroy path.
Request 3: "Expose host memory telemetry (e.g. bb status --resources)"Verified absentbb status --help only has --json; the daemon's defaultReadResourceUsage() is never exposed over RPC or CLI.

Environment

Minimal reproduction

A. Live, end to end (one real Claude Code turn)

  1. Start a dev instance and export its env. Note: in every step below, bb … means the dev instance's CLI, i.e. pnpm bb:dev … (or node packages/scripts/dist/commands/run-cli.js … after the first pnpm bb:dev), run from the repo root with the exported env — a bare bb would talk to your real ~/.bb instance. 
    scripts/bb-dev-app current
eval "$(scripts/bb-dev-app env)"     # BB_SERVER_URL=http://localhost:23803 BB_HOST_DAEMON_PORT=31803 …
pnpm bb:dev machine list             # take the host id from here (mine: host_jkzcgxjdxs)
# Name  ID               Status     Last seen
# bee   host_jkzcgxjdxs  connected  just now
  2. Make a scratch repo and a project on it (substitute your host id from step 1): 
    mkdir -p /tmp/bb1660r-repo && cd /tmp/bb1660r-repo && git init -q -b main && echo "# qa" > README.md \
  && git add . && git -c user.email=qa@example.com -c user.name=qa commit -qm init
curl -s -X POST $BB_SERVER_URL/api/v1/projects -H 'content-type: application/json' \
  -d '{"name":"qa1660r","source":{"type":"local_path","path":"/tmp/bb1660r-repo","hostId":"host_jkzcgxjdxs"}}'
# → {"id":"proj_46fcadffg4", …}
  3. Spawn a thread in a new managed worktree whose only job is to leave a background process behind (stands in for pnpm dev &): 
    pnpm bb:dev thread spawn --project proj_46fcadffg4 --new-environment worktree --provider claude-code \
  --permission-mode full --title "orphan repro 1660" \
  --prompt "Run exactly this shell command and nothing else: nohup sleep 100000 >/dev/null 2>&1 & disown ; then reply only with ok." --json
# → thr_g3xmvwtd84; ~40 s later the thread is idle and environmentId = env_ig2nscwucq
sqlite3 <data>/bb.db "select id,status,environment_id from threads where id='thr_g3xmvwtd84';"
# thr_g3xmvwtd84|idle|env_ig2nscwucq
#   worktree: <data>/worktrees/env_ig2nscwucq/bb1660r-repo   (<data> = the data dir printed by scripts/bb-dev-app current)
  4. List processes rooted in the worktree (script: repro/find-worktree-procs.sh): 
    $ ./find-worktree-procs.sh <data>/worktrees/env_ig2nscwucq
pid=1591658 ppid=1582592 pgid=1581708 sid=1581708 cwd=…/env_ig2nscwucq/bb1660r-repo cmd=node … (provider bridge, child of host daemon 1582592)
pid=1591676 ppid=1591658 pgid=1581708 sid=1581708 cwd=…/env_ig2nscwucq/bb1660r-repo cmd=/home/sawyer/.local/bin/claude --output-format stream-json …
pid=1593823 ppid=1       pgid=1593823 sid=1593772 cwd=…/env_ig2nscwucq/bb1660r-repo cmd=sleep 100000   <-- agent's background process, ppid 1
    Note the pgid/sid columns: the bridge and claude live in the daemon's process group and session (1581708), but the agent's process is the leader of its own group (1593823) in its own session (1593772, whose leader — Claude Code's Bash-tool shell — has already exited). Claude Code spawns its Bash tool with detached: true (Node's setsid()), visible in the CLI bundle: 
    $ strings -n 8 ~/.local/share/claude/versions/2.1.234 | grep -o '.\{60\}type:"bash",shellPath:e,detached:!0.\{20\}'
!t?.skipSnapshot)KDp(e).catch(()=>{});let o,i=!1;return{type:"bash",shellPath:e,detached:!0,stdin:"pipe",
    This matters for the fix: a process-group kill aimed at the bridge cannot reach this process (see Proposed fix).
  5. Archive the thread. The environment enters retiring; after the 5-minute MANAGED_ENVIRONMENT_RETIRE_GRACE_MS the server dispatches environment.destroy to the daemon: 
    $ pnpm bb:dev thread archive thr_g3xmvwtd84
Thread thr_g3xmvwtd84 archived
$ sqlite3 <data>/bb.db "select id,status,retire_requested_at from environments where id='env_ig2nscwucq';"
env_ig2nscwucq|retiring|1787030673745
# ~1m40s later, still retiring: the bridge and claude are still alive (archive does not stop them)
$ ps -o pid,ppid,pgid,sid,stat,etime,cmd -p 1591658,1591676,1593823
    PID    PPID    PGID     SID STAT     ELAPSED CMD
1591658 1582592 1581708 1581708 Sl         01:42 node … (provider bridge)
1591676 1591658 1581708 1581708 Sl         01:42 /home/sawyer/.local/bin/claude …
1593823       1 1593823 1593772 S          01:37 sleep 100000
# … poll `select status from environments where id='env_ig2nscwucq'` until it reads destroyed (~5m12s after archive) …
$ sqlite3 <data>/bb.db "select id,status,path from environments where id='env_ig2nscwucq';"
env_ig2nscwucq|destroyed|
$ ls <data>/worktrees/env_ig2nscwucq
ls: cannot access …: No such file or directory
  6. Expected: no process is left with a cwd inside the destroyed worktree (or at least bb reports one). Actual: 
    $ ./find-worktree-procs.sh <data>/worktrees/env_ig2nscwucq
pid=1593823 ppid=1 pgid=1593823 sid=1593772 cwd=/home/sawyer/.bb-dev/projects-bb-.claude-worktrees-wf_debcf606-e4a-33-4322b8038a31/worktrees/env_ig2nscwucq/bb1660r-repo (deleted) cmd=sleep 100000
$ ps -o pid,ppid,pgid,sid,stat,etime,cmd -p 1591658,1591676,1593823
    PID    PPID    PGID     SID STAT     ELAPSED CMD
1593823       1 1593823 1593772 S          05:38 sleep 100000
    The bridge (1591658) and claude (1591676) did exit at destroy — RuntimeManager.destroyEnvironment calls runtime.shutdown() before workspace.destroy() (that ordering dates from b04593eb87, 2026-03-24) — but the process the agent started did not, and bb never looked at it. Transcripts: repro/live-repro-transcript-revision.txt (this run, with pgid/sid) and repro/live-repro-transcript.txt (original run: thr_dyb25avqn8 / env_tj4p2dg3ru, same outcome).

B. Unit-level (no provider, ~1 s)

File: repro/issue-1660-orphan-process.test.ts (copy into packages/host-workspace/test/). Run: cd packages/host-workspace && pnpm exec vitest run test/issue-1660-orphan-process.test.ts. It fails on main at the last assertion (expected true to be false — the child is still alive); the preceding assertion that /proc/<pid>/cwd reads <worktree> (deleted) passes. Output: repro/vitest-output.txt (re-run for this revision: repro/vitest-output-revision.txt, same failure at line 101; third pass: repro/vitest-output-verify2.txt, again AssertionError: expected true to be false at line 101).

// Repro for get-bb/bb#1660: destroying a managed worktree does not reap (or
// even notice) processes whose cwd is inside the worktree. The `sleep` below
// stands in for a dev server / watcher an agent left running with `&`.
//
// This test FAILS on main by design: the final assertion expects the process
// to be gone after `workspace.destroy()`, but bb only removes the directory.
import { spawn } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { provisionWorkspace } from "../src/index.js";
import { runGit } from "../src/git.js";

const tempDirs: string[] = [];
const children: number[] = [];

async function makeTempDir(prefix: string): Promise<string> {
  const dir = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
  tempDirs.push(dir);
  return dir;
}

async function initRepo(): Promise<string> {
  const repoPath = await makeTempDir("bb-1660-repo-");
  await runGit(["init", "-b", "main"], { cwd: repoPath });
  await runGit(["config", "user.name", "BB Tests"], { cwd: repoPath });
  await runGit(["config", "user.email", "bb@example.com"], { cwd: repoPath });
  await fs.writeFile(path.join(repoPath, "README.md"), "hello\n", "utf8");
  await runGit(["add", "."], { cwd: repoPath });
  await runGit(["commit", "-m", "Initial commit"], { cwd: repoPath });
  return repoPath;
}

function isAlive(pid: number): boolean {
  try { process.kill(pid, 0); return true; } catch { return false; }
}

afterEach(async () => {
  for (const pid of children.splice(0)) {
    try { process.kill(pid, "SIGKILL"); } catch { /* already gone */ }
  }
  await Promise.all(tempDirs.splice(0).map((dir) => fs.rm(dir, { recursive: true, force: true })));
});

describe("#1660 managed worktree destroy vs processes rooted in it", () => {
  it("leaves a background process whose cwd is the worktree running after destroy", async () => {
    const repoPath = await initRepo();
    const parentDir = await makeTempDir("bb-1660-parent-");
    const targetPath = path.join(parentDir, "env");

    const ws = await provisionWorkspace({
      workspaceProvisionType: "managed-worktree",
      sourcePath: repoPath,
      targetPath,
      branchName: "bb/issue-1660",
      baseBranch: "main",
      timeoutMs: 900000,
    });
    expect(ws.managed).toBe(true);

    // Simulate `nohup pnpm dev &` from an agent turn: a detached long-lived
    // process whose cwd is the managed worktree.
    const child = spawn("sleep", ["600"], { cwd: ws.path, detached: true, stdio: "ignore" });
    child.unref();
    if (child.pid === undefined) throw new Error("spawn failed");
    children.push(child.pid);
    expect(isAlive(child.pid)).toBe(true);

    await ws.destroy();

    // The directory is gone...
    await expect(fs.access(targetPath)).rejects.toThrow();

    // ...but the process rooted in it is still alive. On Linux its cwd now
    // reads "<path> (deleted)".
    if (process.platform === "linux") {
      const cwd = await fs.readlink(`/proc/${child.pid}/cwd`);
      expect(cwd).toBe(`${targetPath} (deleted)`);
    }

    // EXPECTED (per issue request #2): bb reaps or at least surfaces processes
    // rooted in a pruned managed worktree. ACTUAL on main: still running.
    expect(isAlive(child.pid)).toBe(false);
  });
});
❯ |@bb/host-workspace| test/issue-1660-orphan-process.test.ts (1 test | 1 failed) 61ms
   × leaves a background process whose cwd is the worktree running after destroy

AssertionError: expected true to be false // Object.is equality
- Expected  false
+ Received  true
 ❯ test/issue-1660-orphan-process.test.ts:101:32

Root cause

1. Environment teardown never considers processes rooted in the worktree (reproduced)

The destroy path is server → daemon environment.destroy → RuntimeManager.destroyEnvironment → workspace.destroy() → removeWorktree(). Each hop only touches what it directly owns:

Why the symptom follows: any long-lived process an agent starts in the worktree — dev servers are the common case because agents are asked to "run the app and check" — outlives the thread and the directory. Nothing in bb records that these processes belong to the environment (bb only knows the bridge PID), so no later sweep can find them either. Over a day of many threads created/archived this accumulates exactly the 88 orphans the reporter counted. Note the server-side 5-minute retire grace (constants.ts#L15, environment-cleanup-internal.ts#L402-L423) only delays the destroy; it does not change what destroy does.

Deeper issue: bb has no notion of "processes belonging to an environment". The only handle it holds is the bridge PID; the bridge is spawned in the daemon's own process group and session, and provider CLIs (Claude Code at least) deliberately start tool shells in new sessions, so no ancestry-, group- or session-based signal from bb can enumerate what an agent left running. On Linux a cgroup per environment would give true ownership; portably (macOS included) the only thing that finds the reproduced orphan is a scan by cwd. #1769 additionally shows a claude process alive 12h46m after its environment was destroyed on 0.38.0 — that is a different path from the one reproduced here (in both my runs the bridge and claude exited when destroyEnvironment ran) and remains unexplained; a crashed/restarted daemon that lost its RuntimeEntry map is the obvious candidate.

2. Host RSS growth (not reproduced; candidates only)

Next experiment that would settle it: run the daemon with --heapsnapshot-signal=SIGUSR2 (or --inspect) under a scripted load (N threads × archive loop with a 1 MB tool output each), sample process.memoryUsage() (rss/heapUsed/external/arrayBuffers) every minute, and diff snapshots. If external/arrayBuffers dominates, look at the event-sink queue and provider stdio; if heapUsed dominates, take a snapshot and look at retained RuntimeEntry/ThreadEvent graphs. Ask the reporter which PID (launcher, server, daemon) hit 77 GB and for a vmmap -summary.

Proposed fix (first principles)

Ordered by what actually covers the reproduced case. The verifier's and my own pgid/sid readings show that the orphan is in its own session, so a process-group fix alone would not have reaped it — the cwd scan is the load-bearing change.

  1. Reap by cwd on destroy (daemon; host-local primitive — correct layer per AGENTS.md). Covers the reproduced case. In RuntimeManager.destroyEnvironment, before workspace.destroy(), enumerate processes whose cwd is inside entry.workspace.path (Linux: readlink /proc/*/cwd, exactly what find-worktree-procs.sh does; macOS: lsof -d cwd -Fpn +D <path> or proc_pidinfo(PROC_PIDVNODEPATHINFO) via a tiny helper) and SIGTERM→SIGKILL them; log the PIDs/commands. Return the reaped list in the environment.destroy result so the server can record a thread event ("stopped 1 process left running in this workspace: sleep 100000") — that changes the RPC result shape, so bump HOST_DAEMON_PROTOCOL_VERSION. Risk: killing a process the user intentionally started from that directory in another terminal; the directory is about to be rm -rf'd anyway, so that process is doomed to a (deleted) cwd regardless — log loudly rather than skip. Also run the same scan in a periodic daemon sweep against already-deleted managed worktree paths (env rows with status destroyed whose former path is under the managed worktrees root, or simply any process whose cwd is under the worktrees root and ends in (deleted)) so orphans from before the fix and from daemon crashes get reaped — this is what the reporter's 10-minute automation does by hand. This is the change that would make the unit repro pass.
  2. Own the process tree (daemon). Defence in depth; does not cover the reproduced case. Spawn provider bridges and terminal shells with detached: true so each becomes its own process-group/session leader; on runtime.shutdown()/terminal close send process.kill(-pid, "SIGTERM") then SIGKILL, falling back to the direct child if the group is gone (the repo already has this exact pattern for setup scripts: killSetupScriptProcess in packages/host-workspace/src/provisioning.ts#L218-L229). This catches descendants that stayed in the bridge's group — bb terminal shells, providers that do not setsid their tool shells, a hung claude whose bridge died — but not the Claude Code Bash-tool case reproduced above, because that shell already runs in a new session (pgid 1593823 ≠ bridge pgid 1581708). Note also that today the bridge shares the daemon's group (in dev even the launcher's, 1581708), so a naive kill(-pgid) on the bridge's current group would kill the daemon itself; the detach must land together with the group kill. No wire change, so no protocol bump.
  3. Telemetry (daemon returns raw data, server/CLI present it). Add a host.resources RPC returning defaultReadResourceUsage() plus child-process count/RSS and the count of (deleted)-cwd processes under the worktrees root, surface it in bb status --resources and bb machine show, and have the health monitor warn on RSS growth (e.g. > 4 GB or > 2× the 1-hour minimum) on every platform, not only on the Linux inotify signal. Bump the protocol version for the new RPC.
  4. Ceiling. Set an explicit --max-old-space-size for server and daemon from the launcher (see #1748 for the cgroup-aware variant) and put a byte cap on the event-sink queue that switches to spilling to disk or refusing new turns rather than growing forever.

PR review

No open PRs are linked to this issue. However, PR #1696 "Kill processes left in a destroyed worktree environment" (open, Fixes #1647, not cross-referenced to #1660) implements essentially the fix proposed above for request 2: it adds listProcessesWithCwdUnder/killProcessesWithCwdUnder to @bb/process-utils (Linux /proc/*/cwd, macOS lsof -d cwd), makes RuntimeManager.destroyEnvironment sweep and SIGTERM→SIGKILL every process rooted in the workspace before removing the directory, and spawns provider bridges / ACP agents / ptys as process-group leaders that receive group signals. It does not touch requests 1 (RSS ceiling) or 3 (telemetry). It was reviewed hostilely in the #1647 report (verdict there: REQUEST CHANGES, minor — signal ordering and an explicit call-out that the sweep is ownership-agnostic). Note that the sweep lives in the daemon's destroyEnvironment, not in workspace.destroy(), so the unit repro in this report (which calls workspace.destroy() directly) would still fail on that branch; the PR's own runtime-manager.test.ts case covers the daemon path. If #1696 merges, #1660 request 2 should be closed as fixed-by-#1647 and requests 1 and 3 split into their own issues.

Related issues

Appendix

Commands run

gh issue view 1660 --repo get-bb/bb --json title,body,state,labels,comments
pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build
grep -rn "removeWorktree\|destroyEnvironment\|closeEnvironmentTerminals" apps packages --include=*.ts
grep -rn "max-old-space-size" apps packages scripts
grep -rn "memoryUsage\|rssBytes" apps/host-daemon/src apps/server/src apps/cli/src
scripts/bb-dev-app current ; scripts/bb-dev-app env
bb machine list ; bb provider list ; bb status --help ; bb doctor --help
curl -s -X POST $BB_SERVER_URL/api/v1/projects … (see repro)
bb thread spawn … --new-environment worktree --provider claude-code … (see repro)
./find-worktree-procs.sh <worktree parent>      # before archive
bb thread archive thr_dyb25avqn8
sqlite3 <data>/bb.db "select id,status,path,retire_requested_at from environments where id='env_tj4p2dg3ru';"
./find-worktree-procs.sh <worktree parent>      # after destroy
cd packages/host-workspace && NO_COLOR=1 pnpm exec vitest run test/issue-1660-orphan-process.test.ts
git merge-base --is-ancestor 1c3f3eff0 desktop-v0.38.0   # #1584 is in 0.38.0
# revision run
pnpm bb:dev machine list
curl -s -X POST $BB_SERVER_URL/api/v1/projects … hostId host_jkzcgxjdxs
pnpm bb:dev thread spawn --project proj_46fcadffg4 --new-environment worktree --provider claude-code … --json
./find-worktree-procs.sh <data>/worktrees/env_ig2nscwucq ; ps -o pid,ppid,pgid,sid,etime,cmd -p …
strings -n 8 ~/.local/share/claude/versions/2.1.234 | grep -o '.\{60\}type:"bash",shellPath:e,detached:!0.\{20\}'
pnpm bb:dev thread archive thr_g3xmvwtd84 ; poll sqlite3 until status=destroyed ; ./find-worktree-procs.sh … ; ps …
git blame -L 1117,1131 apps/host-daemon/src/runtime-manager.ts ; gh pr view 1584 --json title,mergedAt
cd packages/host-workspace && NO_COLOR=1 pnpm exec vitest run test/issue-1660-orphan-process.test.ts   # re-run, still fails at :101
gh issue list --repo get-bb/bb --state all --search "memory OR orphan OR RSS OR leak OR worktree process"
pnpm dev:stop

Artifacts

Notes on evidence quality

Verification

An independent verifier re-ran both reproductions on 16ceb3a54 (own dev instance, thread thr_6pqtx3mwdh / env env_ynzhqp6vd5; unit test in packages/host-workspace/test) and got the same results: bridge, claude and sleep 100000 rooted in the worktree; after archive → 5-min retire → destroy the directory is gone, bridge/claude exited, sleep alive with (deleted) cwd; vitest fails at line 101. All permalinked code excerpts and related-issue states matched the tree. The verifier raised three findings, all accepted and addressed in this revision: