← reports

#2677 · Stale child failure outcome after late acceptance

Bug Medium Effort: Medium threads open on GitHub 2026-08-28 · base fc94f46c1

Verdict: REPRODUCED · Root-cause confidence: high

1. TL;DR

A command timeout can produce a terminal child failure before the server receives a later acceptance event.

The later event restores the child to active status, but the queued parent outcome remains unchanged.

The batch also accepts the same child more than once because it appends entries without an identity check.

Two clean checkouts reproduced both defects with focused server tests.

2. Claims vs findings

ClaimStatusEvidence
A parent can receive a failed outcome while the child has a live accepted turn.VerifiedThe test restored the child to active before the batch flush. The parent then received child-failed.
One outcome batch can contain the same child twice.VerifiedTwo queue calls for one child produced child-outcome-batch instead of one child-failed outcome.
The native event stream can have no terminal event for the false failure.VerifiedThe reproduction posted only turn/started and turn/input/accepted after the timeout result.
A timeout can enter the parent notification path.VerifiedAn RPC result with command_timeout reached the normal settlement path and queued the failure.
The exact reported child later completed normally.UnverifiedThe issue did not include a trusted artifact. The focused test stopped after it proved the live accepted state.

3. Environment

4. Minimal reproduction

  1. Create a clean checkout at the base commit.
  2. Install and build with the repository commands.
  3. Save the linked test at apps/server/test/services/threads/issue-2677-repro.test.ts.
  4. Run the focused test:
    pnpm exec turbo run test --filter=@bb/server --force -- test/services/threads/issue-2677-repro.test.ts

The expected result has no failure after late acceptance and one distinct outcome per child.

The actual result was:

Test Files  1 failed (1)
Tests       2 failed (2)

expected [ 'child-failed' ] to deeply equal []
expected [ 'child-outcome-batch' ] to deeply equal [ 'child-failed' ]

The second clean checkout produced the same two assertion failures.

Repro file: issue-2677-repro.test.ts.

Full reproduction test
import { and, eq } from "drizzle-orm";
import { events, getThread } from "@bb/db";
import { turnRequestEventDataSchema, turnScope } from "@bb/domain";
import {
  groupHostDaemonEvents,
  type HostDaemonEventEnvelope,
} from "@bb/host-daemon-contract";
import { describe, expect, it } from "vitest";
import { queueChildThreadTurnNotificationBestEffort } from "../../../src/services/threads/child-thread-notifications.js";
import { sendThreadMessage } from "../../../src/services/threads/thread-send.js";
import {
  internalAuthHeaders,
  reportQueuedCommandError,
  waitForQueuedCommand,
} from "../../helpers/commands.js";
import { textInput } from "../../helpers/prompt-input.js";
import {
  seedEnvironment,
  seedHostSession,
  seedProjectWithSource,
  seedThread,
  seedThreadRuntimeState,
} from "../../helpers/seed.js";
import { withTestHarness, type TestAppHarness } from "../../helpers/test-app.js";

interface FamilyFixture {
  child: ReturnType<typeof seedThread>;
  environment: ReturnType<typeof seedEnvironment>;
  parent: ReturnType<typeof seedThread>;
  sessionId: string;
}

function seedFamily(harness: TestAppHarness, suffix: string): FamilyFixture {
  const { host, session } = seedHostSession(harness.deps, { id: `host-${suffix}` });
  const { project } = seedProjectWithSource(harness.deps, { hostId: host.id });
  const environment = seedEnvironment(harness.deps, {
    hostId: host.id,
    projectId: project.id,
  });
  const parent = seedThread(harness.deps, {
    environmentId: environment.id,
    projectId: project.id,
    title: "Parent",
  });
  seedThreadRuntimeState(harness.deps, {
    environmentId: environment.id,
    providerThreadId: `provider-parent-${suffix}`,
    threadId: parent.id,
  });
  const child = seedThread(harness.deps, {
    environmentId: environment.id,
    parentThreadId: parent.id,
    projectId: project.id,
    title: "Child",
  });
  seedThreadRuntimeState(harness.deps, {
    environmentId: environment.id,
    providerThreadId: `provider-child-${suffix}`,
    threadId: child.id,
  });
  return { child, environment, parent, sessionId: session.id };
}

async function postEventBatch(args: {
  events: HostDaemonEventEnvelope[];
  harness: TestAppHarness;
  sessionId: string;
}): Promise<Response> {
  return args.harness.app.request("/internal/session/events", {
    method: "POST",
    headers: internalAuthHeaders(args.harness),
    body: JSON.stringify({
      sessionId: args.sessionId,
      eventGroups: groupHostDaemonEvents(args.events),
    }),
  });
}

function parentSystemMessageKinds(harness: TestAppHarness, parentThreadId: string): string[] {
  return harness.db
    .select()
    .from(events)
    .where(and(eq(events.threadId, parentThreadId), eq(events.type, "client/turn/requested")))
    .all()
    .flatMap((row) => {
      const data = turnRequestEventDataSchema.parse(JSON.parse(row.data));
      return data.initiator === "system" && data.systemMessageKind
        ? [data.systemMessageKind]
        : [];
    });
}

describe("child outcome reconciliation", () => {
  it("does not report a failed outcome after a later accepted turn becomes active", async () => {
    await withTestHarness(async (harness) => {
      const fixture = seedFamily(harness, "late-acceptance");
      await sendThreadMessage(harness.deps, {
        environment: fixture.environment,
        payload: {
          input: textInput("continue"),
          mode: "start",
          model: "gpt-5",
          permissionMode: "full",
          reasoningLevel: "medium",
          serviceTier: "default",
        },
        thread: fixture.child,
        trigger: "user",
      });
      const queued = await waitForQueuedCommand(
        harness,
        ({ command }) =>
          command.type === "turn.submit" && command.threadId === fixture.child.id,
      );
      if (queued.command.type !== "turn.submit") {
        throw new Error("Expected a turn submission");
      }
      await reportQueuedCommandError(harness, queued, {
        errorCode: "command_timeout",
        errorMessage: "Timed out waiting for command result",
      });
      const turnId = "turn-late-acceptance";
      const response = await postEventBatch({
        harness,
        sessionId: fixture.sessionId,
        events: [
          {
            threadId: fixture.child.id,
            event: {
              type: "turn/started",
              threadId: fixture.child.id,
              providerThreadId: "provider-child-late-acceptance",
              scope: turnScope(turnId),
            },
          },
          {
            threadId: fixture.child.id,
            event: {
              type: "turn/input/accepted",
              threadId: fixture.child.id,
              providerThreadId: "provider-child-late-acceptance",
              scope: turnScope(turnId),
              clientRequestId: queued.command.requestId,
            },
          },
        ],
      });
      expect(response.status).toBe(200);
      expect(getThread(harness.db, fixture.child.id)?.status).toBe("active");
      await new Promise((resolve) => setTimeout(resolve, 2_100));
      expect(getThread(harness.db, fixture.child.id)?.status).toBe("active");
      expect(parentSystemMessageKinds(harness, fixture.parent.id)).toEqual([]);
    });
  });

  it("keeps one outcome when the same child is queued twice", async () => {
    await withTestHarness(async (harness) => {
      const fixture = seedFamily(harness, "duplicate");
      for (let index = 0; index < 2; index += 1) {
        await queueChildThreadTurnNotificationBestEffort(harness.deps, {
          childThread: fixture.child,
          parentThreadId: fixture.parent.id,
          turnStatus: "failed",
        });
      }
      await new Promise((resolve) => setTimeout(resolve, 2_100));
      expect(parentSystemMessageKinds(harness, fixture.parent.id)).toEqual([
        "child-failed",
      ]);
    });
  });
});

5. Root cause

The server catches every live command error and creates a failure report.

It then applies the failure report before it returns the error to the caller.

See live-command.ts lines 232–257.

const failureReport = buildLiveHostCommandFailureReport(...);
await applyLiveHostCommandReport(deps, { command, execution, report: failureReport });

The thread settlement code does not distinguish an uncertain timeout from an authoritative command rejection.

Without an accepted event in the database, it appends a rejection, marks the run failed, and queues a parent failure.

See thread-lifecycle.ts lines 729–780.

A later root turn/started event can restore active state.

See events.ts lines 348–370.

However, the notification batch keeps the earlier result for two seconds and does not read the child again.

It also appends repeated child entries without an identity check.

See child-thread-notifications.ts lines 358–435.

The host router has a deeper order risk on errors.

It flushes events only after a successful dispatch, although the command contract requests a flush before each result.

See command-router.ts lines 213–220.

6. Proposed fix (first principles)

Treat command_timeout as uncertain delivery during thread command settlement.

Do not append a terminal rejection, apply run.failed, or notify a parent for that result alone.

Also replace an existing batch item when another outcome arrives for the same child.

Keep normal failure settlement for explicit non-timeout errors and native terminal events.

7. Related issues

These links show related areas. They do not prove the cause for this issue.

8. Verification

The first test ran in /tmp/bb-2677-repro-cLi3U1 at the recorded commit.

The second test ran in /tmp/bb-2677-verify-qUiVOl at the same commit.

Both runs used the same Turbo command and produced the same two failures.

The second run required no report correction.

9. Appendix

Commands

git fetch origin main
git worktree add --detach <clean-dir> fc94f46c13b89f54e9c8ba53600352df64b81798
pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build
pnpm exec turbo run test --filter=@bb/server --force -- test/services/threads/issue-2677-repro.test.ts
git log fc94f46c1..origin/main --oneline -- <affected paths>

Local test environment note

An unrelated native-module hardlink fault stopped the first Vitest launch in each checkout.

I rebuilt the isolated test binary and reran the same command.

Both final runs reached Vitest and reproduced the defect.

Trust note

The issue content was untrusted input.

I did not run issue code, follow issue links, or use issue attachments.