#2677 · Stale child failure outcome after late acceptance
Verdict: REPRODUCED · Root-cause confidence: high
1. TL;DR
A command timeout can produce a terminal child failure before the server receives a later acceptance event.
The later event restores the child to active status, but the queued parent outcome remains unchanged.
The batch also accepts the same child more than once because it appends entries without an identity check.
Two clean checkouts reproduced both defects with focused server tests.
2. Claims vs findings
| Claim | Status | Evidence |
|---|---|---|
| A parent can receive a failed outcome while the child has a live accepted turn. | Verified | The test restored the child to active before the batch flush. The parent then received child-failed. |
| One outcome batch can contain the same child twice. | Verified | Two queue calls for one child produced child-outcome-batch instead of one child-failed outcome. |
| The native event stream can have no terminal event for the false failure. | Verified | The reproduction posted only turn/started and turn/input/accepted after the timeout result. |
| A timeout can enter the parent notification path. | Verified | An RPC result with command_timeout reached the normal settlement path and queued the failure. |
| The exact reported child later completed normally. | Unverified | The issue did not include a trusted artifact. The focused test stopped after it proved the live accepted state. |
3. Environment
- BB commit:
fc94f46c13b89f54e9c8ba53600352df64b81798. - System: Linux 7.0.0-30-generic x86_64.
- Node: 24.18.0. pnpm: 9.15.0.
- Database: in-memory SQLite through the repository test harness.
- Provider: none. The test used the synthetic host RPC.
- Ports and persistent data directory: none.
4. Minimal reproduction
- Create a clean checkout at the base commit.
- Install and build with the repository commands.
- Save the linked test at
apps/server/test/services/threads/issue-2677-repro.test.ts. - Run the focused test:
pnpm exec turbo run test --filter=@bb/server --force -- test/services/threads/issue-2677-repro.test.ts
The expected result has no failure after late acceptance and one distinct outcome per child.
The actual result was:
Test Files 1 failed (1) Tests 2 failed (2) expected [ 'child-failed' ] to deeply equal [] expected [ 'child-outcome-batch' ] to deeply equal [ 'child-failed' ]
The second clean checkout produced the same two assertion failures.
Repro file: issue-2677-repro.test.ts.
Full reproduction test
import { and, eq } from "drizzle-orm";
import { events, getThread } from "@bb/db";
import { turnRequestEventDataSchema, turnScope } from "@bb/domain";
import {
groupHostDaemonEvents,
type HostDaemonEventEnvelope,
} from "@bb/host-daemon-contract";
import { describe, expect, it } from "vitest";
import { queueChildThreadTurnNotificationBestEffort } from "../../../src/services/threads/child-thread-notifications.js";
import { sendThreadMessage } from "../../../src/services/threads/thread-send.js";
import {
internalAuthHeaders,
reportQueuedCommandError,
waitForQueuedCommand,
} from "../../helpers/commands.js";
import { textInput } from "../../helpers/prompt-input.js";
import {
seedEnvironment,
seedHostSession,
seedProjectWithSource,
seedThread,
seedThreadRuntimeState,
} from "../../helpers/seed.js";
import { withTestHarness, type TestAppHarness } from "../../helpers/test-app.js";
interface FamilyFixture {
child: ReturnType<typeof seedThread>;
environment: ReturnType<typeof seedEnvironment>;
parent: ReturnType<typeof seedThread>;
sessionId: string;
}
function seedFamily(harness: TestAppHarness, suffix: string): FamilyFixture {
const { host, session } = seedHostSession(harness.deps, { id: `host-${suffix}` });
const { project } = seedProjectWithSource(harness.deps, { hostId: host.id });
const environment = seedEnvironment(harness.deps, {
hostId: host.id,
projectId: project.id,
});
const parent = seedThread(harness.deps, {
environmentId: environment.id,
projectId: project.id,
title: "Parent",
});
seedThreadRuntimeState(harness.deps, {
environmentId: environment.id,
providerThreadId: `provider-parent-${suffix}`,
threadId: parent.id,
});
const child = seedThread(harness.deps, {
environmentId: environment.id,
parentThreadId: parent.id,
projectId: project.id,
title: "Child",
});
seedThreadRuntimeState(harness.deps, {
environmentId: environment.id,
providerThreadId: `provider-child-${suffix}`,
threadId: child.id,
});
return { child, environment, parent, sessionId: session.id };
}
async function postEventBatch(args: {
events: HostDaemonEventEnvelope[];
harness: TestAppHarness;
sessionId: string;
}): Promise<Response> {
return args.harness.app.request("/internal/session/events", {
method: "POST",
headers: internalAuthHeaders(args.harness),
body: JSON.stringify({
sessionId: args.sessionId,
eventGroups: groupHostDaemonEvents(args.events),
}),
});
}
function parentSystemMessageKinds(harness: TestAppHarness, parentThreadId: string): string[] {
return harness.db
.select()
.from(events)
.where(and(eq(events.threadId, parentThreadId), eq(events.type, "client/turn/requested")))
.all()
.flatMap((row) => {
const data = turnRequestEventDataSchema.parse(JSON.parse(row.data));
return data.initiator === "system" && data.systemMessageKind
? [data.systemMessageKind]
: [];
});
}
describe("child outcome reconciliation", () => {
it("does not report a failed outcome after a later accepted turn becomes active", async () => {
await withTestHarness(async (harness) => {
const fixture = seedFamily(harness, "late-acceptance");
await sendThreadMessage(harness.deps, {
environment: fixture.environment,
payload: {
input: textInput("continue"),
mode: "start",
model: "gpt-5",
permissionMode: "full",
reasoningLevel: "medium",
serviceTier: "default",
},
thread: fixture.child,
trigger: "user",
});
const queued = await waitForQueuedCommand(
harness,
({ command }) =>
command.type === "turn.submit" && command.threadId === fixture.child.id,
);
if (queued.command.type !== "turn.submit") {
throw new Error("Expected a turn submission");
}
await reportQueuedCommandError(harness, queued, {
errorCode: "command_timeout",
errorMessage: "Timed out waiting for command result",
});
const turnId = "turn-late-acceptance";
const response = await postEventBatch({
harness,
sessionId: fixture.sessionId,
events: [
{
threadId: fixture.child.id,
event: {
type: "turn/started",
threadId: fixture.child.id,
providerThreadId: "provider-child-late-acceptance",
scope: turnScope(turnId),
},
},
{
threadId: fixture.child.id,
event: {
type: "turn/input/accepted",
threadId: fixture.child.id,
providerThreadId: "provider-child-late-acceptance",
scope: turnScope(turnId),
clientRequestId: queued.command.requestId,
},
},
],
});
expect(response.status).toBe(200);
expect(getThread(harness.db, fixture.child.id)?.status).toBe("active");
await new Promise((resolve) => setTimeout(resolve, 2_100));
expect(getThread(harness.db, fixture.child.id)?.status).toBe("active");
expect(parentSystemMessageKinds(harness, fixture.parent.id)).toEqual([]);
});
});
it("keeps one outcome when the same child is queued twice", async () => {
await withTestHarness(async (harness) => {
const fixture = seedFamily(harness, "duplicate");
for (let index = 0; index < 2; index += 1) {
await queueChildThreadTurnNotificationBestEffort(harness.deps, {
childThread: fixture.child,
parentThreadId: fixture.parent.id,
turnStatus: "failed",
});
}
await new Promise((resolve) => setTimeout(resolve, 2_100));
expect(parentSystemMessageKinds(harness, fixture.parent.id)).toEqual([
"child-failed",
]);
});
});
});
5. Root cause
The server catches every live command error and creates a failure report.
It then applies the failure report before it returns the error to the caller.
See live-command.ts lines 232–257.
const failureReport = buildLiveHostCommandFailureReport(...);
await applyLiveHostCommandReport(deps, { command, execution, report: failureReport });
The thread settlement code does not distinguish an uncertain timeout from an authoritative command rejection.
Without an accepted event in the database, it appends a rejection, marks the run failed, and queues a parent failure.
See thread-lifecycle.ts lines 729–780.
A later root turn/started event can restore active state.
However, the notification batch keeps the earlier result for two seconds and does not read the child again.
It also appends repeated child entries without an identity check.
See child-thread-notifications.ts lines 358–435.
The host router has a deeper order risk on errors.
It flushes events only after a successful dispatch, although the command contract requests a flush before each result.
See command-router.ts lines 213–220.
6. Proposed fix (first principles)
Treat command_timeout as uncertain delivery during thread command settlement.
Do not append a terminal rejection, apply run.failed, or notify a parent for that result alone.
Also replace an existing batch item when another outcome arrives for the same child.
Keep normal failure settlement for explicit non-timeout errors and native terminal events.
7. Related issues
These links show related areas. They do not prove the cause for this issue.
8. Verification
The first test ran in /tmp/bb-2677-repro-cLi3U1 at the recorded commit.
The second test ran in /tmp/bb-2677-verify-qUiVOl at the same commit.
Both runs used the same Turbo command and produced the same two failures.
The second run required no report correction.
9. Appendix
Commands
git fetch origin main git worktree add --detach <clean-dir> fc94f46c13b89f54e9c8ba53600352df64b81798 pnpm install --frozen-lockfile --prefer-offline pnpm exec turbo run build pnpm exec turbo run test --filter=@bb/server --force -- test/services/threads/issue-2677-repro.test.ts git log fc94f46c1..origin/main --oneline -- <affected paths>
Local test environment note
An unrelated native-module hardlink fault stopped the first Vitest launch in each checkout.
I rebuilt the isolated test binary and reran the same command.
Both final runs reached Vitest and reproduced the defect.
Trust note
The issue content was untrusted input.
I did not run issue code, follow issue links, or use issue attachments.