← reports

#1706 · queued thread messages can silently vanish: reported accepted, never delivered, no longer queued

Bug High Effort: Medium threads open on GitHub 2026-08-18 base 16ceb3a540f81c1189efaffb27a39b1d9443abf5 (main)

Verdict: PARTIALLY REPRODUCED · root-cause confidence: medium · linked open PRs: none

TL;DR

Plain-language framing. When you bb thread tell … --mode queue a thread that is busy, the server stores the message in a small "queued messages" table. When the busy turn ends, a background job (the drain worker) reserves the stored row (sets claimed_at/claim_token; the code calls this a claim), does some slow preparation with the machine that runs the agent (host RPCs), and only then turns the row into a real turn on the thread. bb thread queue list hides reserved rows, and a reservation that is interrupted (for example by a server restart) is only cleaned up by a periodic job (the stale-claim sweep, every 10 s) after it is 5 minutes old. client/turn/requested is the event that appears in the thread's log once a message becomes a turn.

The reporter ran bb thread tell <id> "…" --mode queue, got Thread … updated, and later found the message neither in bb thread log nor in bb thread queue list. I could not find or provoke a code path on main that permanently discards an accepted queue-mode message: every accepting path either appends a client/turn/requested event or inserts a queued_thread_messages row in the same SQLite transaction that returns success, and every consuming path deletes the row in the same transaction that appends the turn event. Idle target, active target, three concurrent senders, and queue-then-stop all delivered on my instance.

What I can reproduce is the exact observable state the reporter describes, from two independent gaps: (1) bb thread queue list lists only unclaimed rows, so a message claimed by the drain worker (for the whole duration of an in-flight send, which awaits several host RPCs with 30 s timeouts, or for up to 5 minutes after a server restart mid-drain, which I exercised for real in exp. H) is invisible while still being stored; and (2) bb thread log is a window: the default view is the newest 20 user-message segments (1500-event budget) and --format json returns the oldest 100 events, so grep-based verification on a busy thread yields false negatives. Neither the send nor the queue API returns anything a sender could use as a delivery receipt (send returns {ok:true} without a request or queued-message id), so a sender cannot distinguish "in flight" from "lost". The reporter's third bullet (a queue-mode STOP "sat undelivered while the worker continued") is expected queue semantics: queue mode waits for the current turn to finish; steer mode is what interrupts.

Claims vs findings

ClaimStatusEvidence
tell --mode queue reports "Thread … updated" on successVerifiedThe CLI prints updated for both queue and auto modes; the server returns {ok:true} with no request id (tell action in apps/cli/src/commands/thread/actions.ts; send route in apps/server/src/routes/threads/actions.ts). See outputs below.
Message accepted but absent from target's event logCould not reproduce as lossAll accepting paths persist inside a transaction before responding (Root cause). Reproduced the observation only via a claimed row (exp. F simulated, exp. H real restart mid-drain) and via log windowing (--format json default omitted the newest event on a 110-event thread, exp. G).
Message absent from bb thread queue listVerified as possible while still storedlistQueuedThreadMessages = listUnclaimedQueuedThreadMessages; claimed rows are hidden. Exp. F, exp. H and the unit test.
Message was "neither delivered nor pending"Refuted for the state I reproducedIn exp. F (simulated claim) and exp. H (real server restart mid-drain) the row sat in SQLite the whole time and was delivered by the stale-claim sweep about 5 minutes later. No evidence of a permanent drop was found.
Target thread was idle at send timeUnverifiableIf truly idle, queue-if-active resolves to start and behaves exactly like steer mode (which the reporter calls reliable). The empty-queue observation only makes sense if the target was active (row created) at accept time.
Steer mode unaffectedConsistent with codeSteer never creates a queue row; the claim lifecycle is the only queue-specific state.
Queue-mode STOP "sat undelivered while the worker continued"Not a bugBy design: queue-if-active on an active thread waits for the turn to end; only steer/auto interrupts (resolveSendMode in thread-send.ts).
Two amendments "never arrived; worker log still showed original brief"UnverifiableConsistent with messages queued behind a long turn (or invisible while claimed) at the time the log was read.

Environment

Minimal reproduction

Precondition (all scripts read their environment from these three values; nothing else is hardcoded):

  1. Build the worktree and start your dev instance once: pnpm install --frozen-lockfile --prefer-offline && pnpm exec turbo run build && scripts/bb-dev-app current. Note the printed Data dir. Do not run scripts/bb-dev-app current again afterwards: it restarts the instance (my first re-run of exp. F did that from inside the script and the target thread ended in error: host daemon disconnected; the scripts now call scripts/bb-dev-app status, which is read-only).
  2. export BB_REPO=/abs/path/to/your/bb/worktree. The wrapper 1706-bb.sh and the experiment scripts derive BB_SERVER_URL and the SQLite path <Data dir>/bb.db from it (override with BB=/DB= if needed). sqlite3 must be on PATH.
  3. Create a project (curl -s -X POST $BB_SERVER_URL/api/v1/projects … as in the Appendix) and spawn an idle codex thread; pass its id as the first argument: bb thread spawn --project <proj id> --provider codex --permission-mode accept-edits --prompt "Reply only with ok." --json | grep '"id"'.

F. The reported observation: accepted, not in log, not in queue, while the row is stored (claimed)

Script: 1706/repro/1706-expF.sh (usage: BB_REPO=… ./1706-expF.sh <thread id>). It (1) makes the target active with a 45 s turn, (2) sends a queue-mode tell, (3) marks the row claimed via sqlite exactly as claimNextQueuedThreadMessageGroup does when the drain worker picks it up (a simulation of the claimed state; exp. H below produces the same state for real), then (4) shows what the CLI reports. Output of the re-run on the revision instance (1706-expF.out; the original run on thr_m2agvswnpb produced the same lines):

$ export BB_REPO=/home/sawyer/projects/bb/.claude/worktrees/wf_debcf606-e4a-38
$ /tmp/bb-reports/issues/1706/repro/1706-expF.sh thr_u2867dqs2y
--- using thread=thr_u2867dqs2y db=/home/sawyer/.bb-dev/projects-bb-.claude-worktrees-wf_debcf606-e4a-38-144c59319504/bb.db
Thread thr_u2867dqs2y steered
--- target status:
    "status": "active",
Thread thr_u2867dqs2y updated
--- queue list (unclaimed row visible):
        "type": "text",
        "text": "MARKER_F_CLAIMED reply only with ok",
--- sqlite row still present:
qmsg_ckva4564t4|1|[{"type":"text","text":"MARKER_F_CLAIMED reply only with ok"
--- bb thread queue list:
[]
--- bb thread log | grep MARKER_F:
0
--- target idle now (05:35:26); queue list:
[]
--- log grep MARKER_F:
0
--- sqlite:
qmsg_ckva4564t4|1787031280000|tok_simulated
--- now wait ~5 min for the stale-claim sweep, then run:  /tmp/bb-reports/issues/1706/repro/1706-bb.sh thread log thr_u2867dqs2y | tail -6

Expected (issue's "Expected" section): the message is visible either in the queue or in the log. Actual: bb thread queue list prints [], bb thread log has no MARKER_F, the target reads idle, and the CLI had said Thread thr_u2867dqs2y updated. The row is in SQLite the whole time (claimed_at = 1787031280000 = 05:34:40Z). About five minutes later (05:39:52Z, i.e. 5 min threshold + up to one 10 s sweep tick) the stale-claim sweep released it and it was delivered (1706-wait-marker.sh polls for that; output 1706-expF-after-5min.out):

$ /tmp/bb-reports/issues/1706/repro/1706-wait-marker.sh thr_u2867dqs2y MARKER_F
Tue Aug 18 05:39:52 AM UTC 2026
--- queue list:
[]
--- sqlite:
--- log tail:

── User ────────────────────────────────────────────────────
MARKER_F_CLAIMED reply only with ok

── Assistant ───────────────────────────────────────────────
ok

H. Real server restart mid-drain (no simulation): the claim survives the restart

Script: 1706/repro/1706-expH.sh (usage: BB_REPO=… SERVER_PID=<pid> ./1706-expH.sh <thread id>; find the pid with pgrep -af "tsx src/index.ts" and pick the one whose readlink /proc/<pid>/cwd is $BB_REPO/apps/server). It queues a message on an active thread, then polls SQLite every 20 ms and kill -9s the server the instant claimed_at is set. The dev supervisor restarts the server in 1 s. Output (1706-expH.out):

$ SERVER_PID=1677854 /tmp/bb-reports/issues/1706/repro/1706-expH.sh thr_nyediyxgy6
--- using thread=thr_nyediyxgy6 db=/home/sawyer/.bb-dev/projects-bb-.claude-worktrees-wf_debcf606-e4a-38-144c59319504/bb.db server_pid=1677854
Thread thr_nyediyxgy6 steered
--- target status:
    "status": "active",
Thread thr_nyediyxgy6 updated
--- queue list (unclaimed row visible):
        "type": "text",
        "text": "MARKER_H_RESTART reply only with ok",
--- polling sqlite for claimed_at (drain worker picks the row up when the turn ends) ...
--- 05:40:36.229 row claimed -> kill -9 1677854 (server)
--- sqlite right after kill:
qmsg_bqrwu2sehj|1787031636219|qclaim_s55amuww52|[{"type":"text","text":"MARKER_H_RESTART reply onl
--- waiting for the dev supervisor to restart the server ...
--- server back at 05:40:39
--- target status:
    "status": "idle",
--- bb thread queue list:
[]
--- bb thread log | grep -c MARKER_H:
0
--- sqlite:
qmsg_bqrwu2sehj|1787031636219|qclaim_s55amuww52

The claim (qclaim_s55amuww52, set at 05:40:36.219Z) was written by the real drain worker; the server was killed 10 ms later, before the consuming transaction. dev.log shows [dev-supervisor:server] Child exited unexpectedly with signal SIGKILL. Restarting in 1s. and Server listening at 05:40:38. After the restart the thread is idle, the queue list is empty, the log has no MARKER_H, and the row is still claimed. The new server process has an empty activeQueuedMessageClaimTokens set and no startup release, so nothing touches the row until it is 5 minutes old (1706-expH-after-5min.out):

$ /tmp/bb-reports/issues/1706/repro/1706-wait-marker.sh thr_nyediyxgy6 MARKER_H
Tue Aug 18 05:45:42 AM UTC 2026
--- queue list:
[]
--- sqlite:
--- log tail:

── User ────────────────────────────────────────────────────
MARKER_H_RESTART reply only with ok

── Assistant ───────────────────────────────────────────────
ok

Delivered at 05:45:42Z, 5 min 06 s after the claim; the first host RPC of the drained turn appears in dev.log at 05:45:38. During those five minutes a sender or the reporter's checker would have observed exactly the issue's state: updated from the CLI, thread idle, queue list empty, nothing in the log. No log line is emitted when the sweep releases the orphaned claim.

Unit-level repro of the queue-list gap

File: 1706/repro/issue-1706-claimed-queue-visibility.test.ts (copy it to packages/db/test/data/ in your worktree). It passes on main, i.e. it documents current behavior: a claimed row is stored (hasQueuedThreadMessages = true) but hidden from listQueuedThreadMessages, and a stale release restores it only once the claim is older than the 5-minute threshold. Run with cd packages/db && pnpm exec vitest run test/data/issue-1706-claimed-queue-visibility.test.ts.

import { describe, expect, it } from "vitest";
import type { PromptInput } from "@bb/domain";
import { createConnection } from "../../src/connection.js";
import { migrate } from "../../src/migrate.js";
import { noopNotifier } from "../../src/notifier.js";
import {
  claimNextQueuedThreadMessageGroup,
  createQueuedThreadMessage,
  hasQueuedThreadMessages,
  listQueuedThreadMessages,
  releaseStaleQueuedMessageClaims,
} from "../../src/data/queued-thread-messages.js";
import { createProject } from "../../src/data/projects.js";
import { createThread } from "../../src/data/threads.js";
import { upsertHost } from "../../src/data/hosts.js";

// Issue #1706: a queued message that is *claimed* by the drain worker (or
// orphaned by a server restart mid-drain) is still stored, but the public
// queue listing (`GET /threads/:id/queued-messages`, `bb thread queue list`)
// hides it. To an observer the message is "accepted, not delivered, not
// queued" until the claim is consumed or released (stale after 5 minutes).

function textInput(text: string): PromptInput[] {
  return [{ type: "text", text, mentions: [] }];
}

function setup() {
  const db = createConnection(":memory:");
  migrate(db);
  const host = upsertHost(db, noopNotifier, {
    name: "test-host",
    type: "persistent",
  });
  const { project } = createProject(db, noopNotifier, {
    name: "test-project",
    source: { type: "local_path", hostId: host.id, path: "/tmp/test" },
  });
  const thread = createThread(db, noopNotifier, {
    projectId: project.id,
    providerId: "codex",
  });
  return { db, thread };
}

describe("issue #1706: claimed queued messages disappear from the queue listing", () => {
  it("hides a claimed (in-flight or orphaned) message from listQueuedThreadMessages", () => {
    const { db, thread } = setup();
    createQueuedThreadMessage(db, noopNotifier, {
      threadId: thread.id,
      content: textInput("MARKER queued while target was active"),
      model: "gpt-5",
      reasoningLevel: "medium",
      permissionMode: "accept-edits",
      serviceTier: "default",
      senderThreadId: null,
    });
    expect(listQueuedThreadMessages(db, thread.id)).toHaveLength(1);

    // The auto-send worker claims the message before dispatching it.
    const claimed = claimNextQueuedThreadMessageGroup(
      db,
      noopNotifier,
      thread.id,
    );
    expect(claimed).not.toBeNull();

    // The row is still durably stored ...
    expect(hasQueuedThreadMessages(db, thread.id)).toBe(true);
    // ... but the public listing (what `bb thread queue list` shows) is empty.
    // This is the "not queued" observation from the issue.
    expect(listQueuedThreadMessages(db, thread.id)).toEqual([]);

    // A stale-claim release younger than the 5 minute threshold does not
    // bring it back either (claimedBefore = now - 5min in the sweep).
    releaseStaleQueuedMessageClaims(db, noopNotifier, {
      claimedBefore: Date.now() - 5 * 60 * 1000,
      protectedClaimTokens: [],
    });
    expect(listQueuedThreadMessages(db, thread.id)).toEqual([]);

    // Only once the claim is considered stale does the message reappear.
    releaseStaleQueuedMessageClaims(db, noopNotifier, {
      claimedBefore: Date.now() + 1,
      protectedClaimTokens: [],
    });
    expect(listQueuedThreadMessages(db, thread.id)).toHaveLength(1);
  });
});
$ cd packages/db && pnpm exec vitest run test/data/issue-1706-claimed-queue-visibility.test.ts
 Test Files  1 passed (1)
      Tests  1 passed (1)

G. bb thread log is a window, so grep is not proof of absence

$ bb thread log thr_p4k2xuzmqk --format json | jq 'length, .[-1].seq'
100
101
$ bb thread log thr_p4k2xuzmqk --format json | grep -c MARKER_E      # newest message, seq 103
0
$ bb thread log thr_p4k2xuzmqk --format json --limit 500 | jq length
109
$ curl -s "$BB_SERVER_URL/api/v1/threads/thr_p4k2xuzmqk/timeline" | jq .timelinePage
{ "kind": "latest", "segmentLimit": 20, "returnedSegmentCount": 9, "hasOlderRows": false, "olderCursor": null }

--format json lists the oldest 100 events (afterSeq defaults to 0), and the default minimal view is the newest 20 user-message segments under a 1500-event budget. On a long-running fleet thread either form can miss a delivered message; the CLI offers no way to page the minimal view.

Control experiments that did deliver

$ 1706/repro/1706-expB.sh thr_p4k2xuzmqk        # original run
Thread thr_p4k2xuzmqk steered
    "status": "active",
Thread thr_p4k2xuzmqk updated
=== queue after tell ===
[
  {
    "id": "qmsg_jimz5qvscx",
    "content": [ { "type": "text", "text": "MARKER_B_ACTIVE_QUEUE reply only with ok", "mentions": [] } ],
    "model": "gpt-5.6-sol", "reasoningLevel": "medium", "permissionMode": "accept-edits",
    "serviceTier": "default", "groupWithNext": false,
    "createdAt": 1787028058532, "updatedAt": 1787028058532
  }
]
=== log ===
…
── User ────────────────────────────────────────────────────
Run the shell command 'sleep 40' and then reply only with ok.

── Worked for (48s) ────────────────────────────────────────

── Assistant ───────────────────────────────────────────────
ok

── User ────────────────────────────────────────────────────
MARKER_B_ACTIVE_QUEUE reply only with ok

── Assistant ───────────────────────────────────────────────
ok
=== queue ===
[]

Root cause

What the accept path actually does. bb thread tell --mode queue calls threads.send with mode: "queue-if-active", not queuedMessages.create (actions.ts#L560-L590). The route queues only when the target is active; otherwise it starts a turn right away (routes/threads/actions.ts#L319-L341):

const shouldQueue =
  thread.status === "active" &&
  (payload.mode === "queue-if-active" ||
    (payload.mode !== "start" && isManualCompactionActive(deps, thread)));
if (shouldQueue) {
  ensureThreadIsNotAwaitingUserInteraction(deps, thread.id);
  await createQueuedMessageForThread(deps, { payload: queuedMessagePayloadFromSendRequest(payload), thread });
  return context.json({ ok: true });
}
…
await sendThreadMessage(deps, { environment, payload, thread, trigger: "user" });
return context.json({ ok: true });

Both branches persist before responding: createQueuedThreadMessage inserts the row; sendThreadMessage appends client/turn/requested and applies run.started inside one db.transaction(…, {behavior:"immediate"}) (thread-send.ts#L327-L385). Consumption is atomic too: deleteClaimedQueuedThreadMessageBatchInTransaction runs in the same transaction as the event append in both drain paths (queued-messages.ts#L323-L376, #L400-L437). The only other deletes are the explicit DELETE /queued-messages/:id route and the FK cascade on thread deletion. I found no path that removes a row without appending the turn event.

Gap 1: claimed rows are invisible. The public list is listUnclaimedQueuedThreadMessages (queued-thread-messages.ts#L565-L570). The drain worker claims the group first (claimNextQueuedThreadMessageGroup in sendNextQueuedMessageIfPresent, queued-messages.ts#L513-L519), then awaits requireReadyQueuedMessageEnvironment, buildExecutionOptions, ensureHostSessionReadyForWork, and prepareTurnSubmitCommandPayload, which fans out host RPCs (project skills, shared skills, workspace instructions via resolveThreadRuntimeCommandConfig) each bounded by COMMAND_TIMEOUT_MS = 30_000, before the consuming transaction. During that window, and for up to STALE_QUEUED_MESSAGE_CLAIM_MS = 5 min after a server restart orphans the claim (constant at #L125; the sweep that releases claims older than that, protecting only the tokens in the in-memory activeQueuedMessageClaimTokens set, at #L601-L607, run from a 10 s setInterval in start-server.ts#L255-L258; there is no release-on-startup, which exp. H confirms), the message is stored but reported nowhere: not in the queue list, not in the log, while the thread may already read idle. On a starved host (cf. #1334) or after a host RPC timeout the claim is released and retried on the next 10 s sweep, so the message flickers in and out of the listing.

Gap 2: no delivery receipt. POST /threads/:id/messages returns {ok:true} in both branches, and the CLI prints the same updated line whether a turn started or a row was queued. The sender receives neither a request id nor a queued-message id, so it cannot poll for delivered / still queued / claimed / rejected. Drain failures are only server-side log lines ("Queued message auto-send failed").

Gap 3: the log is windowed. GET /threads/:id/timeline defaults to the latest page of THREAD_TIMELINE_DEFAULT_SEGMENT_LIMIT = 20 segments (timeline.ts#L169) under a 1500-event budget (timelineWindowEventBudget); bb thread log --format json calls events.list with limit=100 from sequence 0 (show.ts#L455-L463, data.ts#L479-L488). An agent that greps either output for a unique string will conclude "absent" on a long thread.

Why the symptom follows. A fleet sender tells an active worker in queue mode → row created → sender is told updated. Some seconds or minutes later the worker's turn ends; the drain claims the row and starts awaiting host RPCs (slow on a busy fleet host); a checker now sees status: idle, an empty queue list, and no message in the (windowed) log. If the drain then times out the row is released and retried; if the server restarted it sits for 5 minutes. From outside this is indistinguishable from a drop, and no API can tell the sender otherwise. I could not demonstrate a permanent drop; if one exists it is not in the queue storage layer.

Proposed fix (first principles)

  1. Return a receipt from send/queue. Make POST /threads/:id/messages return {ok:true, outcome:"started"|"queued", requestId?, queuedMessageId?} and have bb thread tell print which happened (and expose it in --json). Server ↔ client only, no daemon protocol bump. Additive fields keep old SDK clients working; update packages/server-contract, packages/sdk, CLI docs.
  2. Make claimed rows visible. Have listQueuedThreadMessages return all rows with a state: "queued"|"sending" field (and bb thread queue list render it) instead of filtering claimed rows out; keep the claim filter only for the drain's own selection. The app's optimistic reorder/edit code assumes unclaimed rows (apps/app/src/lib/queued-message-reorder.ts, queued-message actions), so treat sending rows as read-only there.
  3. Give the CLI an exhaustive log path. Make bb thread log --format json default to the newest events (or add --all) and print the timeline page metadata (returnedSegmentCount, hasOlderRows) so agents know they saw a window.
  4. Optional: on the receiver side, surface a system row when a drain attempt fails or is deferred, so the target agent is not blind (same asymmetry #1650 raises).

Not confident about a permanent-loss code path; the experiment that would settle it is instrumenting the reporter's fleet: record the receipt id from fix 1, then query queued_thread_messages and events for that id/text at the moment of the "vanished" observation.

PR review

No open PRs are linked to this issue.

Related issues

Appendix

Commands run

# original run (worktree wf_debcf606-e4a-5; app :17606, server :25606, daemon :33606). `bb` = 1706/repro/1706-bb.sh
pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build
scripts/bb-dev-app current
export BB_REPO=$PWD
curl -s -X POST $BB_SERVER_URL/api/v1/projects -H 'content-type: application/json' \
  -d '{"name":"qa","source":{"type":"local_path","path":"/tmp/1706-qa","hostId":"host_3xq9fvix2q"}}'   # hostId from `bb machine list --json`
bb thread spawn --project proj_peuzy9zhn9 --provider codex --permission-mode accept-edits --title "1706 target" --prompt "Reply only with ok." --json
1706/repro/1706-expA.sh thr_p4k2xuzmqk ; 1706/repro/1706-expB.sh thr_p4k2xuzmqk ; 1706/repro/1706-expC.sh thr_p4k2xuzmqk   # control experiments
bb thread tell thr_p4k2xuzmqk "MARKER_D_BOGUS_MODEL reply only with ok" --mode queue --model totally-bogus-model-xyz
bb thread tell thr_p4k2xuzmqk "MARKER_E_QUEUED_THEN_STOP reply only with ok" --mode queue ; bb thread stop thr_p4k2xuzmqk
1706/repro/1706-expF.sh thr_m2agvswnpb
cp 1706/repro/issue-1706-claimed-queue-visibility.test.ts packages/db/test/data/ && cd packages/db && pnpm exec vitest run test/data/issue-1706-claimed-queue-visibility.test.ts
pnpm dev:stop

# revision run (worktree wf_debcf606-e4a-38 at 16ceb3a54; app :14889, server :22889, daemon :30889)
scripts/bb-dev-app current
export BB_REPO=/home/sawyer/projects/bb/.claude/worktrees/wf_debcf606-e4a-38
curl -s -X POST http://localhost:22889/api/v1/projects -H 'content-type: application/json' \
  -d '{"name":"qa","source":{"type":"local_path","path":"/tmp/1706-qa","hostId":"host_j5wc27bxsu"}}'      # -> proj_iaw7pxfexi
bb thread spawn --project proj_iaw7pxfexi --provider codex --permission-mode accept-edits --title "1706 target 2" --prompt "Reply only with ok." --json   # thr_u2867dqs2y
1706/repro/1706-expF.sh thr_u2867dqs2y > 1706/repro/1706-expF.out
1706/repro/1706-wait-marker.sh thr_u2867dqs2y MARKER_F > 1706/repro/1706-expF-after-5min.out
bb thread spawn --project proj_iaw7pxfexi --provider codex --permission-mode accept-edits --title "1706 target 3" --prompt "Reply only with ok." --json   # thr_nyediyxgy6
pgrep -af "tsx src/index.ts"; readlink /proc/1677854/cwd        # server pid of this instance
SERVER_PID=1677854 1706/repro/1706-expH.sh thr_nyediyxgy6 > 1706/repro/1706-expH.out
1706/repro/1706-wait-marker.sh thr_nyediyxgy6 MARKER_H > 1706/repro/1706-expH-after-5min.out
pnpm dev:stop

Event summary of target thread thr_p4k2xuzmqk (every accepted send has its client/turn/requested)

1 client/turn/requested thread-start Reply only with ok.
18 turn/completed
19 client/turn/requested new-turn MARKER_A_IDLE_QUEUE reply only with ok
28 turn/completed
29 client/turn/requested new-turn Run the shell command 'sleep 40' and then reply on
55 turn/completed
56 client/turn/requested new-turn MARKER_B_ACTIVE_QUEUE reply only with ok
65 turn/completed
66 client/turn/requested new-turn MARKER_C3 concurrent queue send; reply only with o
75 turn/completed
76 client/turn/requested new-turn MARKER_C2 concurrent queue send; reply only with o
86 turn/completed
87 client/turn/requested new-turn MARKER_C1 concurrent queue send; reply only with o
96 turn/completed
97 client/turn/requested new-turn MARKER_D_BOGUS_MODEL reply only with ok
101 system/thread/interrupted manual-stop
102 turn/completed
103 client/turn/requested new-turn MARKER_E_QUEUED_THEN_STOP reply only with ok
110 turn/completed

Concurrent-send experiment (C) output

--- sender 1 --- Thread thr_p4k2xuzmqk updated exit=0
--- sender 2 --- Thread thr_p4k2xuzmqk updated exit=0
--- sender 3 --- Thread thr_p4k2xuzmqk updated exit=0
=== queue right after === [ MARKER_C1 ]        # C3 started the turn; C2 and C1 were queued
=== log (45 s later) === MARKER_C3 … ok / MARKER_C2 … ok / MARKER_C1 … ok
=== queue === []

Things checked and ruled out

Verification

An independent verifier re-ran the report on their own dev instance at 16ceb3a54 (server :26397, codex thread thr_6j7semtv5a). Exp. F reproduced line-for-line (queue list [], grep -c MARKER_F = 0 while active and after idle, row still claimed in SQLite; consumed 5m09s after the simulated claim with the assistant reply ok), the unit test passed on main, exp. G's mechanism was confirmed by code (events route default limit 100 ascending from seq 0, CLI passes 100, timeline default 20 segments) and the 10 s sweep interval was confirmed in start-server.ts. Their findings and what changed in this revision:

Verdict and root-cause confidence are unchanged (PARTIALLY REPRODUCED, medium): still no permanent-loss path found; the "accepted, not in log, not in queue" observation is reproduced both by simulation and by a real restart mid-drain.