← reports

#6172 · Pool recovery leaves retry decisions stale

BugPriority: MediumEffort: Highproviders · threads · provider-retryGitHub issue2026-10-23

Base: 1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5, trusted origin/main.

Verdict: REPRODUCED · Root-cause confidence: high. The same agent repeated the reproduction in a second clean trusted checkout. This is not an independent review.

1. TL;DR

Account Pooler can become usable while an automatic retry still waits for an old quota reset. The real Provider Retry entry point queries the real pool once, when the turn fails, then records either a fixed queue schedule or a decline. Enabling a previously disabled reserve account changes the pool's answer to ready, but neither pending nor declined decisions are revisited. A plugin reload also leaves that decision unchanged. Four focused integration scenarios reproduce this on current main with a real migrated core database, real pool storage, and no live provider accounts.

2. Claims vs findings

3. Environment and isolation

• Public repository get-bb/bb; base and refreshed origin/main both at 1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5.
• Linux x86_64; Node v22.19.0; pnpm 9.15.0; Vitest 4.1.1; Account Pooler and Provider Retry package versions 0.1.0.
• Two separately installed temporary checkouts, referred to here as WORK/base and WORK/verify. Both began clean at the same trusted commit; only the local reproduction test was added.
• The server test harness uses createMigratedConnection(); pool accounts and quota observations use the SDK harness's real SQLite storage and a fresh temporary pool data directory per scenario. Pool routing and both plugin entry points are real; cross-plugin SDK transport and upstream HTTP are test adapters.
• No external development server or browser was started. No listening ports, enrolled hosts, real provider binaries, credentials, or production data were used. Fresh per-test data directories are removed in finally blocks; fake timers are restored.

4. Minimal reproduction

1. Clone the trusted target and select the recorded base:

   git clone https://github.com/get-bb/bb.git bb-6172
   cd bb-6172
   git checkout --detach 1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5
   pnpm install --frozen-lockfile --prefer-offline
   pnpm exec turbo run build --filter=@get-bb/plugin-sdk
   pnpm exec turbo run build

2. Save the complete test below as apps/server/test/threads/issue-6172.test.ts. Run:

   pnpm exec turbo run test --force --filter=@bb/server -- test/threads/issue-6172.test.ts test/threads/turn-failed-retry.test.ts

3. The table-driven fixture creates an active thread, two actual pool accounts, and a real upstream-429 quota observation. The reserve starts disabled. Depending on the scenario it schedules a two-hour row, declines for no enabled accounts, or declines for a seven-hour reset. It enables the reserve, verifies the pool itself now answers ready, optionally reloads Provider Retry, then advances fake time by one minute.
4. Expected: the recovery is observed; a still-current failed request is scheduled promptly rather than waiting for the old two-hour reset, and recoverable declines gain a retry. Actual: every scenario still has exactly one Provider Retry availability query in total. The pending rows retain their old timestamps; declined cases have zero rows. The diagnostic remains the initial decision.

Exact diagnostic output from the first run, with no private paths or real account data:

{"scenario":"pending","poolAfterEnable":"ready","availabilityReads":1,"rowCount":1,"originalSendAt":1792764027596,"currentSendAt":1792764027596,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"blocked","retryAt":1792764000000},"decision":{"kind":"retry","reason":"Rate limited","sendAt":1792764027596}}}
{"scenario":"unavailable","poolAfterEnable":"ready","availabilityReads":1,"rowCount":0,"originalSendAt":null,"currentSendAt":null,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"unavailable","reason":"no-enabled-account"},"decision":{"kind":"decline","reason":"pool-unavailable"}}}
{"scenario":"beyond-maximum-wait","poolAfterEnable":"ready","availabilityReads":1,"rowCount":0,"originalSendAt":null,"currentSendAt":null,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"blocked","retryAt":1792782000000},"decision":{"kind":"decline","reason":"beyond-maximum-wait"}}}
{"scenario":"restart","poolAfterEnable":"ready","availabilityReads":1,"rowCount":1,"originalSendAt":1792764037583,"currentSendAt":1792764037583,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"blocked","retryAt":1792764000000},"decision":{"kind":"retry","reason":"Rate limited","sendAt":1792764037583}}}

Key verbatim failures and second-run summary:

AssertionError: Provider Retry must re-query after recovery: expected 1 to be greater than 1
AssertionError: The recovered error-state request must have a retry: expected [] to have a length of 1 but got +0
AssertionError: The retry must no longer wait for the stale reset: expected 1792764032732 to be less than or equal to 1792756860000
Test Files  1 failed | 1 passed (2)
     Tests  4 failed | 13 passed (17)

The four failures are the intended behavioral assertions, not build, import, authentication, or harness setup failures. The existing pooled-429 scheduling control passes. This is nonvisual; screenshots add no evidence.

import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { z } from "zod";
import { expect, it, vi } from "vitest";
import {
  getLatestThreadSequence,
  getThread,
  listEvents,
  listQueuedThreadMessages,
} from "@bb/db";
import { createFakePluginHost } from "@get-bb/plugin-sdk/testing";
import { createAccountPoolPlugin } from "../../../../plugins/account-pool/src/server.js";
import providerRetryPlugin from "../../../../plugins/provider-retry/server.js";
import {
  retryAvailabilityMethod,
  retryAvailabilitySchema,
} from "../../../../plugins/provider-retry/src/retry-contract.js";
import { retryDiagnosticSchema } from "../../../../plugins/provider-retry/src/diagnostics.js";
import { applyLoggedThreadLifecycleEvent } from "../../src/services/threads/lifecycle-outcome.js";
import { retryFailedTurn } from "../../src/services/threads/turn-retry.js";
import { buildTurnFailedEvent } from "../../src/services/threads/turn-failed.js";
import { toThreadQueuedMessage } from "../../src/services/threads/thread-queued-messages.js";
import {
  seedEnvironment,
  seedEvent,
  seedHostSession,
  seedProjectWithSource,
  seedThread,
  seedThreadRuntimeState,
} from "../helpers/seed.js";
import { withTestHarness } from "../helpers/test-app.js";

it.each(["pending", "unavailable", "beyond-maximum-wait", "restart"] as const)(
  "issue 6172: %s observes actual pool recovery within one minute",
  async (scenario) => {
    await withTestHarness(async (harness) => {
      const { host } = seedHostSession(harness.deps, {
        id: "host-pool-recovery",
      });
      const { project } = seedProjectWithSource(harness.deps, {
        hostId: host.id,
        path: "/tmp/pool-recovery-project",
      });
      const environment = seedEnvironment(harness.deps, {
        hostId: host.id,
        projectId: project.id,
        path: "/tmp/pool-recovery-project",
      });
      const thread = seedThread(harness.deps, {
        environmentId: environment.id,
        projectId: project.id,
        status: "active",
      });
      seedThreadRuntimeState(harness.deps, {
        environmentId: environment.id,
        threadId: thread.id,
        providerThreadId: "provider-recovery",
        inputText: "Reply with ok",
      });
      const request = z
        .object({
          requestId: z.string(),
          execution: z.object({ model: z.string() }),
        })
        .parse(
          JSON.parse(
            listEvents(harness.db, { threadId: thread.id }).find(
              (event) => event.type === "client/turn/requested",
            )!.data,
          ),
        );
      const dataDir = await mkdtemp(path.join(tmpdir(), "bb-6172-pool-"));
      vi.useFakeTimers();
      vi.setSystemTime(new Date("2026-10-23T12:00:00.000Z"));
      const resetAt =
        Date.now() + (scenario === "beyond-maximum-wait" ? 7 : 2) * 3_600_000;
      let availabilityReads = 0;
      const pool = createFakePluginHost({
        pluginId: "account-pool",
        dataDir,
        sdk: {
          hosts: { list: async () => [host] },
          system: { providerStates: async () => ({ providers: [] }) },
          threads: { events: { list: async () => [{ data: request }] } },
        },
      });
      let retry = createFakePluginHost({
        pluginId: "provider-retry",
        sdk: {
          plugins: {
            experimental_discoverRpc: async () => [
              {
                pluginId: "account-pool",
                displayName: "Account Pooler",
                method: retryAvailabilityMethod,
              },
            ],
            callRpc: async ({ method, input }) => {
              availabilityReads += 1;
              return retryAvailabilitySchema.parse(
                await pool.harness.behavior.callRpc(method, input),
              );
            },
          },
          threads: {
            queue: {
              list: async () =>
                listQueuedThreadMessages(harness.db, thread.id).map(
                  toThreadQueuedMessage,
                ),
            },
            get: async () => getThread(harness.db, thread.id),
            retry: async (input) => {
              const current = getThread(harness.db, input.threadId);
              if (current === null) throw new Error("Missing test thread");
              return retryFailedTurn(harness.deps, {
                thread: current,
                request: {
                  turnRequestId: input.turnRequestId ?? null,
                  sendAt: input.sendAt ?? null,
                  reason: input.reason ?? "Rate limited",
                },
              });
            },
          },
        },
      });
      try {
        await createAccountPoolPlugin({
          now: () => Date.now(),
          env: {},
          fetch: async () =>
            Response.json(
              {},
              {
                status: 429,
                headers: {
                  "anthropic-ratelimit-unified-5h-status": "rejected",
                  "anthropic-ratelimit-unified-5h-reset": String(
                    resetAt / 1000,
                  ),
                },
              },
            ),
        })(pool.bb);
        const accountSchema = z.object({ id: z.string() });
        const primary = accountSchema.parse(
          await pool.harness.behavior.callRpc("account.add", {
            provider: "claude",
            source: { kind: "api-key", apiKey: "test-primary" },
            label: null,
            priority: 1,
          }),
        );
        const reserve = accountSchema.parse(
          await pool.harness.behavior.callRpc("account.add", {
            provider: "claude",
            source: { kind: "api-key", apiKey: "test-reserve" },
            label: null,
            priority: 2,
          }),
        );
        await pool.harness.behavior.callRpc("account.disable", {
          id: reserve.id,
        });
        pool.harness.behavior.runService("hub");
        const entries = await pool.harness.behavior.resolveProviderEnv(
          "claude-code",
          { threadId: thread.id, projectId: thread.projectId, hostId: host.id },
        );
        const token = entries.find(
          (entry) => entry.name === "ANTHROPIC_AUTH_TOKEN",
        )?.value;
        if (typeof token !== "string")
          throw new Error("Missing test pool route");
        const response = await pool.harness.behavior.fetchHttp(
          "POST",
          "/v1/messages",
          {
            headers: { authorization: `Bearer ${token}` },
            body: JSON.stringify({ model: "claude-opus-4-1", messages: [] }),
          },
        );
        expect(response.status).toBe(429);
        await response.text();
        if (scenario === "unavailable")
          await pool.harness.behavior.callRpc("account.disable", {
            id: primary.id,
          });
        seedEvent(harness.deps, {
          threadId: thread.id,
          environmentId: environment.id,
          sequence:
            getLatestThreadSequence(harness.db, { threadId: thread.id }) + 1,
          type: "provider/error",
          scope: { kind: "thread" },
          data: {
            providerThreadId: "provider-recovery",
            message: "Pool unavailable",
            errorInfo: {
              category: "rate-limit",
              providerCode: null,
              httpStatusCode: 429,
            },
          },
        });
        applyLoggedThreadLifecycleEvent(harness.deps, {
          event: { type: "run.failed" },
          threadId: thread.id,
        });
        const failed = buildTurnFailedEvent(harness.db, thread.id);
        if (failed === null) throw new Error("Missing failure event");
        await providerRetryPlugin(retry.bb);
        expect(
          (await retry.harness.behavior.emitThreadEvent("turn.failed", failed))
            .errors,
        ).toEqual([]);
        const originalRows = listQueuedThreadMessages(harness.db, thread.id);
        const declined =
          scenario === "unavailable" || scenario === "beyond-maximum-wait";
        expect(originalRows).toHaveLength(declined ? 0 : 1);
        if (!declined)
          expect(originalRows[0]!.sendAt).toBeGreaterThanOrEqual(
            resetAt + 15_000,
          );
        const originalDiagnostic = retryDiagnosticSchema.parse(
          await retry.harness.behavior.callRpc("decision.get", {
            threadId: thread.id,
          }),
        );
        expect(originalDiagnostic.decision).toMatchObject(
          declined
            ? {
                kind: "decline",
                reason:
                  scenario === "unavailable"
                    ? "pool-unavailable"
                    : "beyond-maximum-wait",
              }
            : { kind: "retry" },
        );
        expect(availabilityReads).toBe(1);
        await pool.harness.behavior.callRpc("account.enable", {
          id: reserve.id,
        });
        expect(
          await pool.harness.behavior.callRpc(retryAvailabilityMethod, {
            threadId: thread.id,
            requestId: request.requestId,
          }),
        ).toEqual({ kind: "ready" });
        if (scenario === "restart")
          retry = await retry.harness.reload(providerRetryPlugin);
        await vi.advanceTimersByTimeAsync(60_000);
        const rows = listQueuedThreadMessages(harness.db, thread.id);
        const diagnostic = await retry.harness.behavior.callRpc(
          "decision.get",
          { threadId: thread.id },
        );
        console.log(
          JSON.stringify({
            scenario,
            poolAfterEnable: "ready",
            availabilityReads,
            rowCount: rows.length,
            originalSendAt: originalRows[0]?.sendAt ?? null,
            currentSendAt: rows[0]?.sendAt ?? null,
            now: Date.now(),
            diagnostic,
          }),
        );
        expect
          .soft(
            availabilityReads,
            "Provider Retry must re-query after recovery",
          )
          .toBeGreaterThan(1);
        expect
          .soft(rows, "The recovered error-state request must have a retry")
          .toHaveLength(1);
        if (rows[0] !== undefined)
          expect
            .soft(
              rows[0].sendAt,
              "The retry must no longer wait for the stale reset",
            )
            .toBeLessThanOrEqual(Date.now());
      } finally {
        vi.useRealTimers();
        await retry.harness.dispose();
        await pool.harness.dispose();
        await rm(dataDir, { recursive: true, force: true });
      }
    });
  },
);

5. Root cause

The decision has no refresh lifecycle

Provider Retry server entry, lines 52–110, performs discovery and availability reads only inside turn.failed. It either returns after recording a decline or submits a fixed sendAt through threads.retry. There is no scheduler service, startup queue scan, account-change subscription, or message.dispatch hook in this entry.

bb.events.on("turn.failed", async (event) => {
  ...
  const decision = decideRetry({ availability, failure: event, ... });
  ...
  if (decision.kind === "decline") {
    await recordDecision();
    return;
  }
  await bb.sdk.threads.retry({
    threadId: event.threadId,
    turnRequestId: event.requestId,
    sendAt: decision.sendAt,
    reason: decision.reason,
  });
});

Retry policy, lines 53–60, fixes the timestamp to the later of reset and now, plus a 15-second buffer and up to 30 seconds of jitter. Lines 126–143 immediately decline unavailable sources and resets beyond maximumWait. The server stores only one diagnostic under a per-thread key, not a recovery-candidate lifecycle.

The pool recovers correctly, but the retry plugin never learns it

PoolOperations.enable enables the account and clears its error and hold, then notifies its callbacks. The server callback wiring publishes a realtime client signal and refreshes usage. PluginRealtime is an ephemeral client-WebSocket broadcast, not a server plugin event subscription. Config changes similarly publish a client signal in config.set.

Account availability correctly returns ready when any enabled, nonerrored account is usable. Both runs directly verified this after enable, ruling out a stale pool answer as the cause.

A complete repair crosses the queue boundary

Core retry creation enforces one live row and the current failed request. Queued-message update accepts a concurrency timestamp and prompt input, not a replacement schedule. Sending now is not an equivalent reschedule: its contract bypasses all plugin waits. The queue drain forwards the stored retry reference; the separate stale-request dispatch concern was inspected but not dynamically exercised.

6. Proposed fix and simple-fix gate

A complete repair needs a cross-plugin availability-change path, Provider Retry reconciliation of still-current candidates at recovery and startup, and bounded missed-signal recovery. It must reschedule a live row without incrementing attempts or bypassing ordinary dispatch hooks. Candidate retention, original-failure maximum-wait accounting, jitter, later resets, and due-time request validity need explicit semantics and tests.

No automatic code PR is safe under this rule. The end-to-end repair spans Account Pooler, Provider Retry, and core queued retry behavior, and needs a rescheduling/notification contract decision. It fails the one-existing-subsystem and no-public-contract/product-decision gates. No production patch, fix branch, push, or code PR was attempted. GitHub issue timeline and open-PR search showed no linked open PR during this run. A narrow send-now workaround would violate plugin wait policies.

7. Verification

A second temporary clone was fetched directly from the trusted target, checked out at the exact base, and confirmed clean before applying only the reproduction test. It received its own frozen install and SDK/full builds; the full build completed with 71 successful tasks. The same agent repeated the test with fresh migrated test databases and new per-scenario pool directories. No external server ports were needed.

git rev-parse HEAD
1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5

pnpm exec turbo run test --force --filter=@bb/server -- test/threads/issue-6172.test.ts test/threads/turn-failed-retry.test.ts
Test Files  1 failed | 1 passed (2)
     Tests  4 failed | 13 passed (17)

Both test copies have SHA-256 a1d106fd823186b89b0d0157f564d869205cca6d374d24d268ea11a8e03ec43d. Every scenario again observed ready pool availability with exactly one scheduler availability read. Jitter produced different initial sendAt values, but each row remained byte-for-byte unchanged within its run.

{"scenario":"pending","poolAfterEnable":"ready","availabilityReads":1,"rowCount":1,"originalSendAt":1792764032732,"currentSendAt":1792764032732,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"blocked","retryAt":1792764000000},"decision":{"kind":"retry","reason":"Rate limited","sendAt":1792764032732}}}
{"scenario":"unavailable","poolAfterEnable":"ready","availabilityReads":1,"rowCount":0,"originalSendAt":null,"currentSendAt":null,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"unavailable","reason":"no-enabled-account"},"decision":{"kind":"decline","reason":"pool-unavailable"}}}
{"scenario":"beyond-maximum-wait","poolAfterEnable":"ready","availabilityReads":1,"rowCount":0,"originalSendAt":null,"currentSendAt":null,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"blocked","retryAt":1792782000000},"decision":{"kind":"decline","reason":"beyond-maximum-wait"}}}
{"scenario":"restart","poolAfterEnable":"ready","availabilityReads":1,"rowCount":1,"originalSendAt":1792764023756,"currentSendAt":1792764023756,"now":1792756860000,"diagnostic":{"requestId":"creq_2222222223","observedAt":1792756800000,"availability":{"kind":"blocked","retryAt":1792764000000},"decision":{"kind":"retry","reason":"Rate limited","sendAt":1792764023756}}}

No report correction was needed after the second run. Scope limits remain: plugin reload is not a whole-process restart; unavailable was exercised with disabled accounts, not a real OAuth error; full live dispatch, ten-thread staggering, parent pools, every recovery trigger, and superseded-request dispatch were not reproduced. This report establishes the central stale-decision bug rather than claiming all proposed acceptance cases were executed.

8. Related work

The trusted repository history includes the pooled-retry availability change. It supplies the initial pool-aware scheduling path used by the passing control; the present failure is the missing lifecycle after that initial decision. There was no linked open pull request to review.

9. Appendix and evidence hygiene

Initial checkout commands included a frozen install followed by the full Turbo build. The first full build stopped because the GitHub plugin could not resolve an unbuilt SDK host runtime. Building @get-bb/plugin-sdk and repeating the full build succeeded with 71 successful tasks; no source or dependency changes were needed. The second checkout built SDK first and then passed the same full build.

pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build --filter=@get-bb/plugin-sdk
pnpm exec turbo run build
pnpm exec oxfmt apps/server/test/threads/issue-6172.test.ts
pnpm exec turbo run test --filter=@bb/server -- test/threads/turn-failed-retry.test.ts -t 'pooled provider retry integration'
pnpm exec turbo run test --filter=@bb/server -- test/threads/issue-6172.test.ts
pnpm exec turbo run test --force --filter=@bb/server -- test/threads/issue-6172.test.ts test/threads/turn-failed-retry.test.ts
git fetch origin main
git log --oneline HEAD..origin/main -- plugins/provider-retry plugins/account-pool apps/server/src/services/threads
git diff --check

The first control run passed one existing integration test; the second passed all 13 tests in that existing file. Refreshing origin/main showed no intervening changes. Classification was applied through the SlopCop GitHub wrapper and verified as Bug / Medium / High with providers, threads, and provider-retry labels; no existing classification was overwritten. Raw build/test logs remain outside the public reports repository. Per that repository's privacy rules, the complete reproduction and exact relevant output are embedded here, not committed as separate tests, logs, or artifacts.