← reports

#4674 · Full-turn timeline context defeats the source-work budget

BugHigh priorityEffort: Highthreadsperfconfirmed-repro

GitHub issue · October 2, 2026 · trusted main e7d200a6549cdfb1d0a1e0103e44224eaf3c78a0

REPRODUCED · Root-cause confidence: high

1. TL;DR

A small timeline page can require work proportional to an entire long-running agent turn. The server first chooses a budgeted window, then expands every selected turn back to the context-clear boundary and forward to the snapshot head, before decoding and projecting those events. With a 100-leaf budget, a synthetic open turn containing 3,000 completed commands selects 6,003 events and 12,973,222 bytes of event JSON, yet returns only 100 rows. Appending one command causes another full selection of 6,005 events; an older page selects the same full turn. Both clean checkouts reproduced this source-work growth and the deliberately failing bounded-work assertion.

2. Claims vs findings

ClaimFindingEvidence
The page budget does not bound selected events for a large open turn.Verified100-leaf pages select 603 and 6,003 events for 300 and 3,000 commands, respectively; selecting with budget 1,000,000 yields the same event counts.
Older pages inside the same turn do the same large selection.VerifiedBoth older-page builds select the full turn, even though their leaf offsets differ from the latest page.
Item lifecycle appends invalidate selection reuse.VerifiedAn active-thread fixture appends item/started and item/completed; the next latest build performs group-context-query and selects every retained event again. Existing selection-memo tests explicitly protect this behavior.
Every selected event is JSON-parsed on every build.QualifiedThe decode stage visits every selected event, but the stored-event decode cache can reuse parsed values. Its 8,000,000-character two-generation cap cannot retain this 12.97 MB ASCII fixture in full. The profile's decodedEventCount is not a count of JSON.parse calls; exact parse misses were not instrumented.
The reported production stall frequency, worst stall, and thread ages.UnverifiedNo reporter database or production logs were accessed. Local synchronous-build timings demonstrate expensive work, not the stated one-day production distribution.
A turn-size workaround caps this source-work growth.Not directly testedSelection is turn-scoped in the verified query; replacement-thread behavior and the reporter's operational workaround were not exercised.

3. Environment

4. Minimal reproduction

  1. Start from the trusted repository and commit, install, and build: 
    git clone https://github.com/get-bb/bb.git bb-repro
cd bb-repro
git checkout e7d200a6549cdfb1d0a1e0103e44224eaf3c78a0
pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build
  2. Copy the complete authored reproduction test embedded in this report into apps/server/test/services/threads/timeline-source-budget-repro.test.ts. It creates a migrated in-memory database and an active open-turn fixture; no provider executes.
  3. Run the owning server tests through Turbo: 
    pnpm exec turbo run test --filter=@bb/server -- test/services/threads/timeline-source-budget-repro.test.ts --silent=false
  4. Two evidence tests pass; the deliberately proposed bounded-work regression fails: 
    Expected: no more than 203 source events for this 100-leaf page
Actual:   603 source events
AssertionError: expected 603 to be less than or equal to 203
Tests  1 failed | 2 passed (3)
Exit status: 1

The 203-event threshold is a generous experimental invariant for this fixture: two lifecycle events per returned command plus three turn-anchor events. It is not an existing API guarantee or a proposed pagination-contract change.

First-run buildSource eventsJSON bytesDecode visitsProjected rowsReturned rowsBuild ms
300:latest603129712260330110099.1
300:older603129712260330110051.6
300:after-item-append605130144560530210058.6
3000:latest60031297322260033001100581.2
3000:older60031297322260033001100672.4
3000:after-item-append60051297754760053002100625.8
Complete reproduction test
import { describe, expect, it } from "vitest";
import {
  createConnection,
  createProject,
  createThread,
  getLatestThreadSequence,
  insertEvents,
  migrate,
  noopNotifier,
  upsertHost,
} from "@bb/db";
import type { DbConnection } from "@bb/db";
import {
  encodeClientTurnRequestIdNumber,
  threadScope,
  turnScope,
} from "@bb/domain";
import type { Thread } from "@bb/domain";
import { buildThreadTimelineWithProfile } from "../../../src/services/threads/timeline.js";
import type { ThreadTimelinePageRequest } from "../../../src/services/threads/timeline-pagination.js";

type EventInput = Parameters<typeof insertEvents>[2][number];
const eventBudget = 100;
const turnId = "repro-open-turn";

function setup(itemCount: number, commandChars: number) {
  const db = createConnection(":memory:");
  migrate(db);
  const host = upsertHost(db, noopNotifier, { name: "repro-host" });
  const { project } = createProject(db, noopNotifier, {
    name: "repro-project",
    source: { type: "local_path", hostId: host.id, path: "/tmp/repro-scratch" },
  });
  const created = createThread(db, noopNotifier, {
    projectId: project.id,
    providerId: "claude-code",
  });
  const thread: Thread = { ...created, status: "active" };
  const clientRequestId = encodeClientTurnRequestIdNumber({ value: 1 });
  const events: EventInput[] = [];
  const append = (event: Omit<EventInput, "threadId" | "sequence">) => {
    events.push({ ...event, threadId: thread.id, sequence: events.length + 1 });
  };
  append({
    type: "client/turn/requested",
    scope: threadScope(),
    itemId: null,
    itemKind: null,
    parentToolCallId: null,
    data: JSON.stringify({
      direction: "outbound",
      source: "tell",
      initiator: "user",
      request: { method: "turn/start", params: {} },
      requestId: clientRequestId,
      senderThreadId: null,
      input: [
        { type: "text", text: "Synthetic command workload", mentions: [] },
      ],
      target: { kind: "thread-start" },
      execution: {
        model: "gpt-5",
        serviceTier: "default",
        reasoningLevel: "medium",
        permissionMode: "full",
        source: "client/turn/requested",
      },
    }),
  });
  append({
    type: "turn/started",
    scope: turnScope(turnId),
    providerThreadId: "repro-provider",
    itemId: null,
    itemKind: null,
    parentToolCallId: null,
    data: "{}",
  });
  append({
    type: "turn/input/accepted",
    scope: turnScope(turnId),
    providerThreadId: "repro-provider",
    itemId: null,
    itemKind: null,
    parentToolCallId: null,
    data: JSON.stringify({ clientRequestId }),
  });
  insertEvents(db, noopNotifier, events);
  appendItems(db, thread, 0, itemCount, commandChars);
  return { db, thread };
}

function appendItems(
  db: DbConnection,
  thread: Thread,
  start: number,
  count: number,
  commandChars: number,
) {
  const events: EventInput[] = [];
  let sequence = getLatestThreadSequence(db, { threadId: thread.id });
  for (let index = start; index < start + count; index += 1) {
    const itemId = `repro-command-${index}`;
    for (const completed of [false, true]) {
      sequence += 1;
      events.push({
        threadId: thread.id,
        sequence,
        type: completed ? "item/completed" : "item/started",
        scope: turnScope(turnId),
        providerThreadId: "repro-provider",
        itemId,
        itemKind: "commandExecution",
        parentToolCallId: null,
        data: JSON.stringify({
          item: {
            type: "commandExecution",
            id: itemId,
            command: "x".repeat(commandChars),
            cwd: "/tmp/repro-scratch",
            status: completed ? "completed" : "pending",
            approvalStatus: null,
            ...(completed ? { exitCode: 0, aggregatedOutput: "ok" } : {}),
          },
        }),
      });
    }
  }
  insertEvents(db, noopNotifier, events);
}

function build(
  db: DbConnection,
  thread: Thread,
  budget: number,
  page: ThreadTimelinePageRequest = { kind: "latest", segmentLimit: 20 },
) {
  return buildThreadTimelineWithProfile(db, thread, {
    completedTurnDisplay: "collapse",
    eventBudget: budget,
    includeDiagnosticOperations: false,
    includeNestedRows: false,
    maxInlineOutputChars: 32_000,
    maxSeq: 0,
    page,
  });
}

function logBuild(name: string, result: ReturnType<typeof build>) {
  const { profile, response } = result;
  console.log(
    JSON.stringify({
      name,
      eventBudget,
      eventRowCount: profile.eventRowCount,
      eventDataBytes: profile.eventDataBytes,
      decodedEventCount: profile.decodedEventCount,
      projectedRowCount: profile.projectedRowCount,
      responseRowCount: profile.responseRowCount,
      contentPage: response.timelinePage.contentPage,
      fullSelection: profile.stageTimings.some(
        (entry) => entry.stage === "group-context-query",
      ),
      totalDurationMs: profile.totalDurationMs,
    }),
  );
}

describe("open-turn source selection reproduction", () => {
  it.each([300, 3_000])(
    "records source work for %i command items",
    (itemCount) => {
      const { db, thread } = setup(itemCount, 2_000);
      try {
        const latest = build(db, thread, eventBudget);
        logBuild(`${itemCount}:latest`, latest);
        const unlimited = build(db, thread, 1_000_000);
        expect(latest.profile.eventRowCount).toBe(2 * itemCount + 3);
        expect(latest.profile.eventRowCount).toBe(
          unlimited.profile.eventRowCount,
        );
        const contentPage = latest.response.timelinePage.contentPage;
        expect(contentPage).toBeDefined();
        expect(contentPage!.end - contentPage!.start).toBeLessThanOrEqual(
          eventBudget,
        );
        const beforeCursor = latest.response.timelinePage.olderCursor;
        expect(beforeCursor).not.toBeNull();
        const older = build(db, thread, eventBudget, {
          kind: "older",
          beforeCursor: beforeCursor!,
          segmentLimit: 20,
        });
        logBuild(`${itemCount}:older`, older);
        expect(older.profile.eventRowCount).toBe(latest.profile.eventRowCount);
        appendItems(db, thread, itemCount, 1, 2_000);
        const appended = build(db, thread, eventBudget);
        logBuild(`${itemCount}:after-item-append`, appended);
        expect(appended.profile.eventRowCount).toBe(2 * itemCount + 5);
        expect(
          appended.profile.stageTimings.some(
            (entry) => entry.stage === "group-context-query",
          ),
        ).toBe(true);
      } finally {
        db.$client.close();
      }
    },
  );

  it("bounds source events for a 100-leaf latest page", () => {
    const { db, thread } = setup(300, 2_000);
    try {
      const latest = build(db, thread, eventBudget);
      expect(latest.profile.eventRowCount).toBeLessThanOrEqual(
        2 * eventBudget + 3,
      );
    } finally {
      db.$client.close();
    }
  });
});

5. Root cause

The thread-data route passes the configured timelineWindowEventBudget into the timeline builder. The initial selection computes an event-budget floor. However, turn-group context expansion replaces the window bounds with the whole epoch-to-snapshot span for every touching turn:

...listStoredTimelineTurnEventRows(db, {
  ...windowArgs,
  sequenceStart: epochSequenceStart,
  beforeSequence: maxSeq + 1,
  turnIds,
})

The turn-row database query filters by turn IDs and the expanded sequence span but has no event-budget limit. Consequently, one large retained turn overrides the initial budget floor. There are excluded event types and context-clear boundaries, so this is not literally every event ever stored for the thread.

The decode stage visits all selected rows, and projection precedes pagination. Only then does the response enforce maxLeaves: Math.max(1, options.eventBudget). Full-turn projection provides the leaf-index cursor and total leaf count, so a naive SQL limit would change page contents, cursor meaning, and summary behavior.

The selection memo's appendable set allows streaming deltas/progress, not item start/completion. Its append validation therefore returns a miss for the new lifecycle events in this reproduction. Even a reused selection is still decoded/projected by the builder; the memo stores selection, not an incrementally maintained full projection. The decode cache reduces actual parsing for cache-resident events but does not bound source querying or projection.

Repository history traces the broad turn-context query to commit 8d2a0280ae. The existing large-running-turn test intentionally expects budgeted and unbudgeted event counts to match while the response is paginated. This is an established design limitation, not an accidental missing limit with a safe one-line repair.

6. Proposed fix and simple-fix decision

Preserving the public cursor contract would require a bounded, incrementally maintained turn projection with careful invalidation for lifecycle events, parented content, rewritten events, pruning, context clears, snapshot reads, and active-to-idle transitions. Validate warm append costs against turn length and compare every incremental result with a cold projection across those cases. A sequence-based content window is another direction, but changes cursor/page semantics and is outside the authorized simple-fix criteria.

No pull request: a root-cause repair needs a projection/cache or pagination architecture decision. Limiting the context query breaks current semantics; merely widening the appendable memo list leaves full projection work intact. No production changes, fix branch, dependency, or schema modification were made. No linked open PR was present in the issue timeline or the repository's open-PR search for this numeric issue at the final check.

7. Verification

The same agent repeated the reproduction in a second clean detached worktree at e7d200a6549cdfb1d0a1e0103e44224eaf3c78a0, with a fresh in-memory database for every test. The second checkout's production Git diff was empty before and after the run; only the authored reproduction test was added. Its frozen install and full Turbo build succeeded. The exact command was the same Turbo test command shown above, and the result again was two evidence tests passing and the proposed source bound failing at 603 versus 203. Both checkouts selected 6,003 events / 12,973,222 bytes before append and 6,005 events / 12,977,547 bytes afterward. This is a repeated check by one agent, not independent verification.

Correction incorporated into this report: decodedEventCount measures decode-stage visits, not fresh JSON parsing. Exact production stall statistics remain unverified. No other report correction was necessary.

Relevant existing tests were also run unchanged:

pnpm exec turbo run test --filter=@bb/server -- test/services/threads/timeline-in-turn-window.test.ts test/services/threads/timeline-selection-memo.test.ts
@bb/server:test:  ✓ |@bb/server| test/services/threads/timeline-in-turn-window.test.ts (33 tests) 7681ms
@bb/server:test:  ✓ |@bb/server| test/services/threads/timeline-selection-memo.test.ts (14 tests) 6746ms
@bb/server:test:  Test Files  2 passed (2)
@bb/server:test:       Tests  47 passed (47)
@bb/server:test:    Duration  31.42s (transform 33.12s, setup 9.67s, import 26.10s, tests 14.43s, environment 0ms)
 Tasks:    9 successful, 9 total

8. Related issues

#1749 and #3951 concern adjacent timeline performance costs. They were reviewed only as repository metadata for classification context; their code or proposed changes were not run. This reproduction isolates whole-turn source expansion, separate from build sharing or per-event cost.

9. Appendix

The sanitized logs and structured measurements from both runs are embedded below. The complete reproduction test is embedded in section 4. Raw artifacts remain in the investigator's local work directory rather than the public repository. There are no screenshots because this is a server performance/data-selection issue.

First clean run

@bb/server:test: cache miss, executing 0061058208c0f63e
@bb/server:test:
@bb/server:test: > @bb/server@0.0.1 test first-clean-checkout/apps/server
@bb/server:test: > vitest run --config vitest.config.ts "test/services/threads/timeline-source-budget-repro.test.ts" "--silent=false"
@bb/server:test:
@bb/server:test:
@bb/server:test:  RUN  v4.1.1 first-clean-checkout/apps/server
@bb/server:test:
@bb/server:test: stdout | test/services/threads/timeline-source-budget-repro.test.ts > open-turn source selection reproduction > records source work for 300 command items
@bb/server:test: {"name":"300:latest","eventBudget":100,"eventRowCount":603,"eventDataBytes":1297122,"decodedEventCount":603,"projectedRowCount":301,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":201,"end":301,"total":301},"fullSelection":true,"totalDurationMs":99.1}
@bb/server:test: {"name":"300:older","eventBudget":100,"eventRowCount":603,"eventDataBytes":1297122,"decodedEventCount":603,"projectedRowCount":301,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":101,"end":201,"total":301},"fullSelection":true,"totalDurationMs":51.6}
@bb/server:test: {"name":"300:after-item-append","eventBudget":100,"eventRowCount":605,"eventDataBytes":1301445,"decodedEventCount":605,"projectedRowCount":302,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":202,"end":302,"total":302},"fullSelection":true,"totalDurationMs":58.6}
@bb/server:test:
@bb/server:test: stdout | test/services/threads/timeline-source-budget-repro.test.ts > open-turn source selection reproduction > records source work for 3000 command items
@bb/server:test: {"name":"3000:latest","eventBudget":100,"eventRowCount":6003,"eventDataBytes":12973222,"decodedEventCount":6003,"projectedRowCount":3001,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":2901,"end":3001,"total":3001},"fullSelection":true,"totalDurationMs":581.2}
@bb/server:test: {"name":"3000:older","eventBudget":100,"eventRowCount":6003,"eventDataBytes":12973222,"decodedEventCount":6003,"projectedRowCount":3001,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":2801,"end":2901,"total":3001},"fullSelection":true,"totalDurationMs":672.4}
@bb/server:test: {"name":"3000:after-item-append","eventBudget":100,"eventRowCount":6005,"eventDataBytes":12977547,"decodedEventCount":6005,"projectedRowCount":3002,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":2902,"end":3002,"total":3002},"fullSelection":true,"totalDurationMs":625.8}
@bb/server:test:
@bb/server:test:  ❯ |@bb/server| test/services/threads/timeline-source-budget-repro.test.ts (3 tests | 1 failed) 3474ms
@bb/server:test:      ✓ records source work for 300 command items  450ms
@bb/server:test:      ✓ records source work for 3000 command items  2812ms
@bb/server:test:      × bounds source events for a 100-leaf latest page 209ms
@bb/server:test:
@bb/server:test: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
@bb/server:test:
@bb/server:test:  FAIL  |@bb/server| test/services/threads/timeline-source-budget-repro.test.ts > open-turn source selection reproduction > bounds source events for a 100-leaf latest page
@bb/server:test: AssertionError: expected 603 to be less than or equal to 203
@bb/server:test:  ❯ test/services/threads/timeline-source-budget-repro.test.ts:215:44
@bb/server:test:     213|     try {
@bb/server:test:     214|       const latest = build(db, thread, eventBudget);
@bb/server:test:     215|       expect(latest.profile.eventRowCount).toBeLessThanOrEqual(
@bb/server:test:        |                                            ^
@bb/server:test:     216|         2 * eventBudget + 3,
@bb/server:test:     217|       );
@bb/server:test:
@bb/server:test: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
@bb/server:test:
@bb/server:test:
@bb/server:test:  Test Files  1 failed (1)
@bb/server:test:       Tests  1 failed | 2 passed (3)
@bb/server:test:    Start at  04:12:47
@bb/server:test:    Duration  13.02s (transform 8.51s, setup 5.05s, import 4.30s, tests 3.47s, environment 0ms)
@bb/server:test:
@bb/server:test:  ELIFECYCLE  Test failed. See above for more details.
 Tasks:    8 successful, 9 total
Cached:    0 cached, 9 total
  Time:    37.497s

Second clean run

@bb/server:test: cache miss, executing 0061058208c0f63e
@bb/server:test:
@bb/server:test: > @bb/server@0.0.1 test second-clean-checkout/apps/server
@bb/server:test: > vitest run --config vitest.config.ts "test/services/threads/timeline-source-budget-repro.test.ts" "--silent=false"
@bb/server:test:
@bb/server:test:
@bb/server:test:  RUN  v4.1.1 second-clean-checkout/apps/server
@bb/server:test:
@bb/server:test: stdout | test/services/threads/timeline-source-budget-repro.test.ts > open-turn source selection reproduction > records source work for 300 command items
@bb/server:test: {"name":"300:latest","eventBudget":100,"eventRowCount":603,"eventDataBytes":1297122,"decodedEventCount":603,"projectedRowCount":301,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":201,"end":301,"total":301},"fullSelection":true,"totalDurationMs":109.5}
@bb/server:test: {"name":"300:older","eventBudget":100,"eventRowCount":603,"eventDataBytes":1297122,"decodedEventCount":603,"projectedRowCount":301,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":101,"end":201,"total":301},"fullSelection":true,"totalDurationMs":69.7}
@bb/server:test: {"name":"300:after-item-append","eventBudget":100,"eventRowCount":605,"eventDataBytes":1301445,"decodedEventCount":605,"projectedRowCount":302,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":202,"end":302,"total":302},"fullSelection":true,"totalDurationMs":69.2}
@bb/server:test:
@bb/server:test: stdout | test/services/threads/timeline-source-budget-repro.test.ts > open-turn source selection reproduction > records source work for 3000 command items
@bb/server:test: {"name":"3000:latest","eventBudget":100,"eventRowCount":6003,"eventDataBytes":12973222,"decodedEventCount":6003,"projectedRowCount":3001,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":2901,"end":3001,"total":3001},"fullSelection":true,"totalDurationMs":587.5}
@bb/server:test: {"name":"3000:older","eventBudget":100,"eventRowCount":6003,"eventDataBytes":12973222,"decodedEventCount":6003,"projectedRowCount":3001,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":2801,"end":2901,"total":3001},"fullSelection":true,"totalDurationMs":519.6}
@bb/server:test: {"name":"3000:after-item-append","eventBudget":100,"eventRowCount":6005,"eventDataBytes":12977547,"decodedEventCount":6005,"projectedRowCount":3002,"responseRowCount":100,"contentPage":{"anchorSeq":1,"start":2902,"end":3002,"total":3002},"fullSelection":true,"totalDurationMs":571.8}
@bb/server:test:
@bb/server:test:  ❯ |@bb/server| test/services/threads/timeline-source-budget-repro.test.ts (3 tests | 1 failed) 3421ms
@bb/server:test:      ✓ records source work for 300 command items  504ms
@bb/server:test:      ✓ records source work for 3000 command items  2702ms
@bb/server:test:      × bounds source events for a 100-leaf latest page 212ms
@bb/server:test:
@bb/server:test: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
@bb/server:test:
@bb/server:test:  FAIL  |@bb/server| test/services/threads/timeline-source-budget-repro.test.ts > open-turn source selection reproduction > bounds source events for a 100-leaf latest page
@bb/server:test: AssertionError: expected 603 to be less than or equal to 203
@bb/server:test:  ❯ test/services/threads/timeline-source-budget-repro.test.ts:215:44
@bb/server:test:     213|     try {
@bb/server:test:     214|       const latest = build(db, thread, eventBudget);
@bb/server:test:     215|       expect(latest.profile.eventRowCount).toBeLessThanOrEqual(
@bb/server:test:        |                                            ^
@bb/server:test:     216|         2 * eventBudget + 3,
@bb/server:test:     217|       );
@bb/server:test:
@bb/server:test: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
@bb/server:test:
@bb/server:test:
@bb/server:test:  Test Files  1 failed (1)
@bb/server:test:       Tests  1 failed | 2 passed (3)
@bb/server:test:    Start at  04:13:04
@bb/server:test:    Duration  12.74s (transform 8.32s, setup 5.04s, import 4.10s, tests 3.42s, environment 0ms)
@bb/server:test:
@bb/server:test:  ELIFECYCLE  Test failed. See above for more details.
 Tasks:    8 successful, 9 total
Cached:    6 cached, 9 total
  Time:    36.921s

Other investigation commands: read repository visibility and issue properties through GitHub; fetch trusted main; inspect the timeline service, its database query, pagination, memo and decode cache; inspect git log/blame; query linked PR metadata; compare the trusted base to refreshed main; verify empty production diffs. No issue-supplied command, patch, linked branch, external URL, or real runtime data was executed or accessed. Issue content was treated solely as untrusted claims.

> AGENT GENERATED