← reports

#4607 · Scheduled-message promotion loses delivery provenance

BugMedium priorityMedium effortthreadspartial-reproGitHub issue2026-10-01

Trusted base: 4a5ab46d3a4a730b8eafefd8313b2d089e9be5ce

PARTIALLY REPRODUCED · Root-cause confidence: high for the promotion-provenance gap; the historical early-delivery cause remains unverified.

1. TL;DR

Trusted main retains a future scheduled message while an active Codex-selected thread receives ordinary queue wakes and a child completion notice. The same message dispatches once the test clock reaches its deadline. An explicit promotion intentionally sends before that deadline and removes the queued row, but the durable outbound turn request does not preserve the original queue id, due time, or promotion reason. That confirmed observability gap prevents a receiver from proving whether an early arrival was an intentional override. This report does not establish that an automatic drain caused the original incident.

2. Claims vs Findings

3. Environment

• Public get-bb/bb repository, trusted origin/main base 4a5ab46d3a4a730b8eafefd8313b2d089e9be5ce.
• Linux 4.19.0-gvisor; Node v22.19.0; pnpm 9.15.0; Vitest 4.1.1.
• Two separate temporary checkouts: base/ and verification/. Both production trees remain at the recorded commit; the sole source addition is this report's test.
• The existing harness initializes migrated in-memory SQLite and an isolated temporary data directory per case, then cleans up. Requests use Hono's in-process request interface; no listening server, ports, user database, or production instance is used.
• The seeded thread selects Codex, has a provider session and an active turn. Host dispatch is observed through the repository's test harness, not a live Codex executable or authenticated provider. Provider binary version: not applicable.
• The controlled test clock starts at 2026-01-15T12:00:00Z; the schedule is exactly one minute later. Only Date is faked; other timers remain real.

4. Minimal Reproduction

1. Check out the recorded trusted commit in a fresh temporary clone and install existing dependencies with the frozen lockfile.
2. Save the linked authored test in the server's existing test directory. It uses production admission, queue draining, parent notice delivery, and the public queue-promotion endpoint; it does not mock the database.
3. Build with Turbo, then run the focused reproduction. Exit code 1 is expected because the final provenance assertion demonstrates the gap.

git clone https://github.com/get-bb/bb.git bb-4607-repro
cd bb-4607-repro
git checkout --detach 4a5ab46d3a4a730b8eafefd8313b2d089e9be5ce
pnpm install --frozen-lockfile
curl -fsS https://get-bb.github.io/reports/issues/4607/repro/scheduled-message-provenance.test.ts -o apps/server/test/threads/scheduled-message-provenance.test.ts
pnpm exec turbo run build --filter=@bb/server
pnpm exec turbo run test --filter=@bb/server -- scheduled-message-provenance

Expected: Ordinary wakes and child notices retain a future row; a due wake dispatches it. A deliberate early promotion remains identifiable through its original queue identity, original due time, and override reason.

Actual: All automatic-retention and due-time checks pass. Explicit promotion succeeds and removes the row. The original queue id and due time are absent from the durable request, so the provenance assertion fails. The early send itself is allowed behavior, not the defect proven here.

promotion evidence {"dispatchedBeforeDue":true,"queueRowsRemaining":0,"containsQueuedId":false,"containsOriginalSendAt":false,"requestDataKeys":["direction","execution","initiator","input","request","requestId","senderThreadId","source","target"]}
✓ retains a future row across automatic wakes and a child notice, then sends when due
× allows explicit promotion before the deadline but loses the queue provenance
AssertionError: delivery must retain the original queued message id

Download: scheduled-message-provenance.test.ts. Full authored test:

import { listEvents, listQueuedThreadMessages } from "@bb/db";
import { turnRequestEventDataSchema } from "@bb/domain";
import { afterEach, describe, expect, it, vi } from "vitest";
import { deliverParentSystemMessage } from "../../src/services/threads/parent-system-messages.js";
import { runQueuedMessageDispatch } from "../../src/services/threads/queued-message-dispatch.js";
import { acceptThreadSendRequest } from "../../src/services/threads/thread-send-request.js";
import { textInput } from "../helpers/prompt-input.js";
import {
  seedEnvironment,
  seedHostSession,
  seedProjectWithSource,
  seedThread,
  seedThreadRuntimeState,
  seedTurnStarted,
} from "../helpers/seed.js";
import { withTestHarness, type TestAppHarness } from "../helpers/test-app.js";

afterEach(() => vi.useRealTimers());

function seedActiveThread(harness: TestAppHarness) {
  const { host } = seedHostSession(harness.deps, { id: "host-scheduled-provenance" });
  const { project } = seedProjectWithSource(harness.deps, {
    hostId: host.id,
    path: "/tmp/scheduled-provenance-project",
  });
  const environment = seedEnvironment(harness.deps, {
    hostId: host.id,
    projectId: project.id,
    path: "/tmp/scheduled-provenance-project",
  });
  const thread = seedThread(harness.deps, {
    environmentId: environment.id,
    projectId: project.id,
    status: "active",
  });
  seedThreadRuntimeState(harness.deps, {
    environmentId: environment.id,
    providerThreadId: "provider-scheduled-provenance",
    threadId: thread.id,
  });
  seedTurnStarted(harness.deps, {
    environmentId: environment.id,
    threadId: thread.id,
    turnId: "turn-scheduled-provenance",
    providerThreadId: "provider-scheduled-provenance",
  });
  return thread;
}

function requests(harness: TestAppHarness, threadId: string) {
  return listEvents(harness.db, { threadId }).filter(
    (event) => event.type === "client/turn/requested",
  );
}

describe("scheduled active-thread dispatch", () => {
  it("retains a future row across automatic wakes and a child notice, then sends when due", async () => {
    await withTestHarness(async (harness) => {
      const thread = seedActiveThread(harness);
      vi.useFakeTimers({ toFake: ["Date"] });
      const now = Date.UTC(2026, 0, 15, 12);
      vi.setSystemTime(now);
      const sendAt = now + 60_000;
      const result = await acceptThreadSendRequest(harness.deps, {
        thread,
        payload: { input: textInput("scheduled probe"), mode: "steer-if-active", sendAt },
      });
      expect(result.delivery).toBe("queued");
      const queued = listQueuedThreadMessages(harness.db, thread.id);
      expect(queued).toHaveLength(1);
      expect(result).toMatchObject({
        delivery: "queued",
        queuedMessage: {
          sendAt,
          waitingOn: { kind: "time" },
          groupWithNext: false,
          failureReason: null,
        },
      });
      const before = requests(harness, thread.id).length;
      for (const kind of ["thread-ready", "turn-started", "workspace-ready", "interaction-settled"] as const) {
        await runQueuedMessageDispatch(harness.deps, { kind, threadId: thread.id });
      }
      await runQueuedMessageDispatch(harness.deps, { kind: "time-reached", now });
      await runQueuedMessageDispatch(harness.deps, { kind: "plugin-recheck" });
      expect(requests(harness, thread.id)).toHaveLength(before);
      expect(listQueuedThreadMessages(harness.db, thread.id)).toEqual(queued);
      expect(await deliverParentSystemMessage(harness.deps, {
        parentThread: thread,
        input: textInput("child completion probe"),
        systemMessageKind: "child-completed",
        systemMessageSubject: { kind: "thread", threadId: "thr_child_probe", threadName: "Probe child" },
      })).toBe(true);
      expect(requests(harness, thread.id)).toHaveLength(before + 1);
      expect(listQueuedThreadMessages(harness.db, thread.id)).toEqual(queued);
      vi.setSystemTime(sendAt);
      await runQueuedMessageDispatch(harness.deps, { kind: "time-reached", now: sendAt });
      expect(listQueuedThreadMessages(harness.db, thread.id)).toEqual([]);
      expect(requests(harness, thread.id)).toHaveLength(before + 2);
      expect(turnRequestEventDataSchema.parse(JSON.parse(requests(harness, thread.id).at(-1)!.data))).toMatchObject({ input: textInput("scheduled probe") });
    });
  });

  it("allows explicit promotion before the deadline but loses the queue provenance", async () => {
    await withTestHarness(async (harness) => {
      const thread = seedActiveThread(harness);
      vi.useFakeTimers({ toFake: ["Date"] });
      const now = Date.UTC(2026, 0, 15, 12);
      vi.setSystemTime(now);
      const sendAt = now + 60_000;
      await acceptThreadSendRequest(harness.deps, {
        thread,
        payload: { input: textInput("promotion probe"), mode: "steer-if-active", sendAt },
      });
      const queued = listQueuedThreadMessages(harness.db, thread.id);
      expect(queued).toHaveLength(1);
      const response = await harness.app.request(
        `/api/v1/threads/${thread.id}/queued-messages/${queued[0]!.id}/send`,
        { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ mode: "steer" }) },
      );
      expect(response.status).toBe(200);
      expect(await response.json()).toMatchObject({ ok: true, delivery: "sent" });
      expect(Date.now()).toBeLessThan(sendAt);
      expect(listQueuedThreadMessages(harness.db, thread.id)).toEqual([]);
      const event = requests(harness, thread.id).at(-1);
      const data = turnRequestEventDataSchema.parse(JSON.parse(event!.data));
      expect(data).toMatchObject({ input: textInput("promotion probe") });
      const serialized = event!.data;
      console.log("promotion evidence", JSON.stringify({
        dispatchedBeforeDue: Date.now() < sendAt,
        queueRowsRemaining: listQueuedThreadMessages(harness.db, thread.id).length,
        containsQueuedId: serialized.includes(queued[0]!.id),
        containsOriginalSendAt: serialized.includes(String(sendAt)),
        requestDataKeys: Object.keys(data).sort(),
      }));
      expect(serialized, "delivery must retain the original queued message id").toContain(queued[0]!.id);
    });
  });
});

5. Root Cause

Automatic dispatch respects the clock in the tested paths

Admission records a time wait for a future sendAt. Automatic group eligibility rejects a time-wait member until due. The scheduled sweep query selects only sendAt ≤ now. The periodic sweep invokes the same production time-reached dispatcher. This is why ordinary active-thread wakes do not consume the row in the positive control.

case "time":
  return member.sendAt !== null && member.sendAt <= args.now;

The real CLI maps its steer mode to steer-if-active and forwards sendAt to the SDK: CLI request translation. The reproduction uses that translated mode rather than copying an issue command. Parent notices create their own turn request without consuming unrelated queue rows: Active-parent notice transaction.

Promotion discards durable schedule provenance

sendQueuedMessageNow selects explicit-send, which becomes the internal sendNow bypass. That intentionally permits a send before the scheduled clock.

sendQueuedMessagePayload rebuilds content and execution fields, but does not carry the original queue id or sendAt. The internal sendNow flag is not a durable promotion marker. The dispatch preflight consumes the claimed rows, and the claimed-row deletion is transactional. The public durable turn-request schema has retry provenance but no scheduled-queue provenance. After dispatch, the only persistent request cannot explain the override and the consumed queue row is unavailable.

Scope qualification: settleQueueRowDispatched emits the queued row to plugin subscribers. A plugin observing that transient event can see the row id and due time. This report therefore does not claim that no component ever receives those values; the demonstrated gap is the receiver's durable outbound request and the missing explicit promotion reason.

6. Proposed Fix and Simple-Fix Decision

Define an explicit durable schedule-provenance contract, including original queued ids, original deadlines, and an automatic-due versus explicit-promotion reason. Propagate that contract through the persisted request and appropriate receiver-visible delivery surfaces, with a focused regression that compares automatic delivery to explicit promotion. Do not change the valid send-now override or weaken the clock guard.

No pull request: Automatic early release was not reproduced. Repairing the confirmed provenance gap requires a public event/schema contract change, excluded by this rule's simple-fix conditions. No production fix, branch, or code push was attempted.

No open pull request linked to #4607 was found in the issue timeline or the open-PR search. No linked code was checked out or executed.

7. Verification

The same agent repeated the final test in a second clean checkout at 4a5ab46d3a4a730b8eafefd8313b2d089e9be5ce. Its Git status was clean before adding only the authored reproduction. Dependencies were installed again with the frozen lockfile, the server build passed again, and Turbo's --force option ensured the second test actually ran rather than replaying a cache result. Each case creates a new temporary data directory and in-memory database; no network ports are opened.

pnpm install --frozen-lockfile
pnpm exec turbo run build --filter=@bb/server
pnpm exec turbo run test --force --filter=@bb/server -- scheduled-message-provenance

Test Files  1 failed (1)
Tests       1 failed | 1 passed (2)

The first checkout also ran the existing requested-queue-drain and queue-drain-failure suites. All 31 existing tests passed; together with the positive-control reproduction, the final run had 32 passing tests and only the intentional provenance failure. The second checkout reproduced exactly the same promotion evidence and the same failing assertion. This is a same-agent verification, not an independent review.

During test construction, raw SQLite JSON was initially compared to typed values. Those fixture assertions were corrected to use the typed response and the existing event decoder. The final runs linked here contain only the intended provenance failure; those initial harness mistakes do not support a product defect.

A later trusted main commit, d7ae1580b, changes task-panel context. The recorded base-to-main history has no change in the queue subsystem or turn-request schema. There is no evidence for an ALREADY FIXED verdict.

8. Related Issues and Trust Boundary

The search found the older scheduled-message feature issue #603, but no duplicate established from the reviewed metadata. It is context, not evidence for this incident's cause.

Issue content was treated only as untrusted claims. Its suggested actions were not executed; the test was designed from trusted main's contracts and code. No external issue-supplied links, branches, patches, scripts, credentials, or runtime data were accessed.

9. Appendix

• First final run: reproduction plus existing queue suites
• Second clean-checkout final run
• First server build
• Second server build

Investigation used read-only GitHub issue/type/field/label metadata, issue comments, timeline and open-PR searches; Git fetch, clone, log and blame; source inspection; frozen installs; and Turbo build/test commands. Classification and reproduction labeling use the automation identity. No live app or provider was started. Harness cleanup completed in both runs. The public artifact omits machine home paths and installation logs.

> AGENT GENERATED