← reports

#2653 · Package test shard resource contention

Bug Medium Effort: Medium host perf open on GitHub 2026-08-28 · base 87865a1

Verdict: REPRODUCED · Root-cause confidence: high

1. TL;DR

The package test shard starts many Turbo tasks without a concurrency limit. Each task can also start Vitest workers and child processes.

Two clean runs failed in the same five-second catalog test. The isolated provider package passed.

The shard used 8.8 to 9.1 GB of memory. The runner policy gives only four CPUs in CI.

2. Claims vs findings

The investigation treated all issue data as untrusted. It did not run issue commands or open issue links.

3. Environment

• Trusted bb commit: 87865a103302e6c3a33e79122aebfd72f48c9cbe.
• System: Linux 7.0.0-30-generic, x86_64, 16 CPUs.
• Node: v24.18.0. pnpm: 9.15.0. Turbo: 2.8.3.
• No bb instance, provider, port, or bb data directory was used.

4. Minimal reproduction

1. Use a clean checkout at the trusted base commit.
2. Run the saved reproduction script from the repository root.

   bash issues/2653/repro/package-shard.sh

The script contains this trusted CI test command with a disabled cache.

pnpm exec turbo run test \
  --filter='!@bb/server' \
  --filter='!@bb/app' \
  --filter='!@bb/integration-tests' \
  --cache-dir=.turbo/repro-cache \
  --output-logs=errors-only \
  --force

Expected: All selected Turbo test tasks succeed.

Actual: A child-process catalog test exceeded the default five-second test limit.

Test timed out in 5000ms.
Test Files  1 failed | 19 passed (20)
Tests       1 failed | 115 passed (116)

Repro files: package-shard.sh and results.txt.

5. Verification

I created a second clean checkout at the same commit. I repeated the frozen install and the full Turbo build.

I then ran the same uncached package shard. The same test exceeded the same five-second limit.

Clean run A: 1 failed, 115 passed, 1:46.44, 9.1 GB peak
Clean run B: 1 failed, 115 passed, 2:16.46, 8.8 GB peak
Isolated package: 4 Turbo tasks successful, 0:19.45, 0.3 GB peak

The second run supported the first result. It required no report correction.

6. Root cause

The CI package shard targets a four-CPU runner. Its Turbo test command has no concurrency limit.

See ci.yml lines 100–142.

runs-on: blacksmith-4vcpu-ubuntu-2404
run: pnpm exec turbo run test ...

The Pi test config has no package test timeout. Vitest therefore uses its five-second default.

See vitest.config.ts lines 6–14.

The focused test starts a catalog, waits for one child to exit, and starts another child.

See catalog.test.ts lines 21–53.

The catalog uses the child-process API for each generation.

See rpc-child.ts lines 115–126.

Turbo starts multiple package test tasks. Vitest then starts workers inside each task.

This nested parallel work competes for CPU and memory. The five-second outer limit expires before the correct child sequence completes.

The isolated control removes the cross-package competition. It passes with far less memory use.

7. Proposed fix

Set Turbo concurrency to four for the package test shard. This matches the runner CPU count.

The same CI file already applies this limit to the build, typecheck, and lint job.

Keep the catalog assertions event-based. Do not replace them with fixed delays.

8. Related work

Trusted commit 8bd6acc96 capped another CI job at four tasks. That change used the same four-CPU runner evidence.

No open pull request links to issue 2653.

9. Appendix

Commands

git fetch origin main
git rev-parse origin/main
pnpm install --frozen-lockfile --prefer-offline
pnpm exec turbo run build
pnpm exec turbo run test --filter='!@bb/server' --filter='!@bb/app' --filter='!@bb/integration-tests' --cache-dir=.turbo/repro-cache --output-logs=errors-only --force
pnpm exec turbo run test --filter=bb-plugin-provider-pi --cache-dir=.turbo/repro-cache --output-logs=errors-only --force

Build result

Both clean Turbo builds completed 18 tasks successfully.