#3407 · Bulk file replies delay daemon heartbeats

Bug · High priority · Medium effort · host · remote · inline-vis
2026-09-10 · Base 1bc80e1d57991fe60a3896e60e0159427194c7bb · GitHub issue

Verdict: PARTIALLY REPRODUCED · Root-cause confidence: medium (high for the measured transport mechanism; low for attribution of the historical incident).

1. TL;DR

Two synthetic HTML file responses queued on a bandwidth-limited connection prevented the daemon's application heartbeats from reaching the receiver before its acknowledgement lease expired. The actual main-branch daemon connection logged an acknowledgement timeout and reconnected while megabytes remained queued. The same file transferred successfully on the fast control connection. This reproduces a transport failure mechanism, not the complete production incident: no full preview UI, Tailscale deployment, HTTP 504 response, provider discovery failure, or repeated session replacement cycle was reproduced.

2. Claims vs findings

3. Environment

Trusted origin/main at the commit above; macOS 26.6.2; Node v22.22.3; repository-pinned pnpm 9.15.0 through Corepack. Both clean worktrees completed frozen dependency installation and Turbo build (20 tasks). No provider, credentials, production data, or real BB instance was used. Each execution allocates fresh loopback ports and a fresh temporary synthetic file directory, then closes all sockets and deletes that directory.

4. Minimal reproduction

This is a repeatable transport probe, not a full incident reproduction. Download repro-3407.ts and place it at apps/host-daemon/repro-3407.ts in a clean checkout at the recorded commit.

corepack pnpm install --frozen-lockfile --prefer-offline
corepack pnpm exec turbo run build
node --conditions=source --import tsx apps/host-daemon/repro-3407.ts

Ensure the pnpm executable on PATH works; the initial machine launcher was broken. A temporary Corepack shim fixed the launcher without changing repository dependencies. The probe reads a generated 3,145,743-byte HTML file through readHostFile. It starts the real ServerConnection with a synthetic session registrar and a local WebSocket receiver that acknowledges application heartbeats. After one successful fast transfer and heartbeat, a TCP relay limits daemon-to-receiver traffic to 16 KiB each 250 ms (64 KiB/s) and queues two file replies. The daemon retains its normal 5-second heartbeat interval and 30-second acknowledgement lease.

Expected resilience: control heartbeats remain responsive while file transfer is slow. Actual: no further heartbeat arrives before acknowledgement timeout, neither slow file reply finishes, and the relay still has approximately 4 MB queued. A subsequent heartbeat can arrive from the replacement connection. The probe exits successfully when this failure mechanism is observed; it is not a passing regression assertion for desired resilience.

Probe source

import assert from 'node:assert/strict';
import net from 'node:net';
import { once } from 'node:events';
import { mkdtemp, writeFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { WebSocketServer } from 'ws';
import { ServerConnection } from './src/server-connection.js';
import { readHostFile } from './src/command-handlers/host-files.js';

const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
const dir = await mkdtemp(path.join(tmpdir(), 'bb-3407-data-'));
const sockets = new Set<net.Socket>();
const intervals = new Set<ReturnType<typeof setInterval>>();
const server = new WebSocketServer({ host: '127.0.0.1', port: 0 });
await once(server, 'listening');
const address = server.address();
assert(address && typeof address !== 'string');
let slow = false;
let backlog = 0;
let replies = 0;
let heartbeats = 0;
let timeouts = 0;
let heartbeatsAtTimeout = 0;
server.on('connection', ws => ws.on('message', data => {
  const message = JSON.parse(data.toString());
  if (message.type === 'heartbeat') {
    heartbeats++;
    ws.send(JSON.stringify({ type: 'heartbeat-ack' }));
  }
  if (message.type === 'host-rpc.response') replies++;
}));
const relay = net.createServer(client => {
  const upstream = net.connect(address.port, '127.0.0.1');
  sockets.add(client); sockets.add(upstream);
  const queue: Buffer[] = [];
  client.on('error', () => {}); upstream.on('error', () => {});
  upstream.pipe(client);
  client.on('data', data => {
    if (!slow) upstream.write(data);
    else { queue.push(data); backlog += data.length; }
  });
  const timer = setInterval(() => {
    let budget = 16384;
    while (queue.length && budget > 0) {
      const chunk = queue[0];
      const length = Math.min(chunk.length, budget);
      upstream.write(chunk.subarray(0, length));
      backlog -= length; budget -= length;
      if (length === chunk.length) queue.shift();
      else queue[0] = chunk.subarray(length);
    }
  }, 250);
  intervals.add(timer);
});
relay.listen(0, '127.0.0.1');
await once(relay, 'listening');
const relayAddress = relay.address();
assert(relayAddress && typeof relayAddress !== 'string');
const connection = new ServerConnection({
  serverUrl: `http://127.0.0.1:${relayAddress.port}`, hostKey: 'synthetic-key',
  hostId: 'synthetic-host', hostName: 'synthetic-host', hostType: 'persistent',
  dataDir: dir, instanceId: 'synthetic-instance', localApiPort: null,
  logger: { debug() {}, info() {}, error() {}, warn(fields, message) {
    if (message === 'Server heartbeat acknowledgements stopped; reconnecting') { timeouts++; heartbeatsAtTimeout = heartbeats; }
  } },
  serverClient: {
    async openSession() { return {
      heartbeatIntervalMs: 5000, leaseTimeoutMs: 30000, sessionId: 'synthetic-session',
      retiredEnvironmentIds: [], connectShares: { generation: 0, ports: [] },
      pluginHostGenerations: [], watchSet: { generation: 0, threadStorageTargets: [], workspaceTargets: [] },
    }; },
    async fetchProjectAttachment() { throw Error('unused'); },
    async fetchSkillTree() { throw Error('unused'); },
    async fetchPluginHostArtifact() { throw Error('unused'); },
    async postEvents() { throw Error('unused'); },
    async callTool() { throw Error('unused'); },
    async registerInteractiveRequest() { throw Error('unused'); },
    async interruptInteractiveRequests() { throw Error('unused'); },
  },
});
try {
  const file = path.join(dir, 'synthetic.html');
  await writeFile(file, '<!doctype html>' + 'a'.repeat(3 * 1024 * 1024));
  const result = await readHostFile({ type: 'host.read_file', path: file, rootPath: dir });
  await connection.start();
  const send = (requestId: string) => connection.sendMessage({
    type: 'host-rpc.response', requestId, commandType: 'host.read_file', ok: true, result,
  });
  assert(send('control'));
  await sleep(6000);
  assert.equal(replies, 1); assert(heartbeats >= 1); assert.equal(timeouts, 0);
  console.log(JSON.stringify({ phase: 'fast-control', fileBytes: result.sizeBytes, replies, heartbeats, timeouts }));
  slow = true;
  assert(send('slow-one')); assert(send('slow-two'));
  const heartbeatBefore = heartbeats;
  await sleep(35000);
  console.log(JSON.stringify({ phase: '64-KiB-per-second', backlog, replies, heartbeats, heartbeatsAtTimeout, timeouts }));
  assert.equal(replies, 1); assert.equal(heartbeatsAtTimeout, heartbeatBefore); assert(timeouts >= 1); assert(backlog > 0);
} finally {
  await connection.shutdown();
  for (const timer of intervals) clearInterval(timer);
  for (const socket of sockets) socket.destroy();
  for (const socket of server.clients) socket.terminate();
  await new Promise<void>(resolve => relay.close(() => resolve()));
  await new Promise<void>(resolve => server.close(() => resolve()));
  await rm(dir, { recursive: true, force: true });
}

5. Root cause

The measured mechanism is head-of-line blocking on a shared ordered transport. apps/host-daemon/src/server-connection.ts:185 serializes each message and sends one complete WebSocket message. Its high-water handling covers terminal output, not file RPC replies. apps/host-daemon/src/app.ts:836 routes the file command response through that connection. The receiver cannot observe later application heartbeat messages until earlier file bytes have crossed the constrained connection.

apps/host-daemon/src/server-connection.ts:714 sends application heartbeats and reconnects after acknowledgement silence exceeds the lease. apps/server/src/ws/daemon-protocol.ts:222 acknowledges only a received application heartbeat. This explains the constrained probe timeout. It does not establish the initiating cause or bandwidth of the historical incident.

plugins/inline-vis/server.ts:125 performs a complete file read before returning only the relative filename. apps/host-daemon/src/command-handlers/file-read.ts:353 already enforces a MIME-dependent host file limit before reading, so the host reader is not entirely unbounded. apps/server/src/services/hosts/online-rpc.ts:178 maps a host-command timeout to HTTP 504; that mapping is supporting static evidence only.

6. Proposed fix / next experiment

Not enough evidence for a safe simple fix. A metadata-only preview preflight could reduce redundant traffic but would not keep the subsequent file transfer from blocking the same control channel. Preserving control responsiveness during arbitrary bulk transfer needs a transport design and compatibility assessment, which is outside this rule's simple-fix boundary. Next run the full preview path through the same bottleneck while measuring small RPC completion, socket backlog, session identities, and receiver acknowledgement latency; compare with the fast control and other transfer rates. Do not treat a larger heartbeat timeout as an established root-cause repair.

7. Related issues and PRs

GitHub cross-reference metadata and an open-PR search found no linked open pull request for #3407 at investigation time. The issue author's related-incident comparisons were not independently validated.

8. Verification

The same agent repeated the experiment in a second clean temporary worktree at the identical trusted commit, with a separate frozen install, Turbo build, ephemeral ports, and synthetic data directory. Both runs observed a successful fast control and an acknowledgement timeout with queued file data under the imposed bottleneck. No production fix was applied. The existing server-connection.test.ts suite passed all 13 tests through Turbo.

Harness correction: the initial assertion counted all heartbeats after reconnection and failed because a replacement connection received an acknowledgement. Both initial runs still recorded the congestion timeout. The corrected probe captures the heartbeat count at timeout; both final runs use that correction. No claim of an independent verifier is made.

9. Appendix

Final probe outputs: first.log and second.log. Build and test summary: checks.txt.

Issue content was treated as untrusted claims. No linked script, patch, branch, external evidence URL, or incident command was executed. No screenshots are supplied because this experiment tests transport behavior without rendering UI.