🚨 **SLOP COP** 🚨 · `new-issue-autopilot`
#4589 · Required skill transport failure
Verdict: PARTIALLY REPRODUCED · Root-cause confidence: low for the origin of the production HTTP 500; high for propagation from an injected HTTP 500 to required-skill staging rejection.
1. TL;DR
A required skill must be downloaded before a host can prepare an agent runtime. Two fresh trusted checkouts show that an HTTP 500 from the skill-tree GET rejects staging with the reported error message, and a second attempt against the still-failing endpoint rejects again. This is a controlled transport fault, not a reproduction of what generated the production 500. When the endpoint recovers, the next staging attempt succeeds; a subsequent warm-cache attempt succeeds without another GET even when the endpoint is failing again. The production server/Connect cause, actual turn command, Mac environment, and outer error response were not reproduced, so no root-cause fix or pull request is justified.
2. Claims vs findings
| Claim | Finding | Evidence |
|---|---|---|
| A required tree fetch can reject with HTTP 500. | Verified under controlled fault injection | Real loopback HTTP, native fetch, production createServerClient and stageInjectedSkillSources; two cold-cache rejections in both runs. |
| Repeated submissions encounter the same failure. | Partially verified | Repeated staging fails while the endpoint remains 500. No actual turn.submit was executed. |
| The incident involved three persisted turn rejections on a remote Mac host. | Unverified in this investigation | Issue evidence is untrusted input. No production logs, databases, real host state, or private runtime data were accessed. |
| Missing or changed skill roots explain the 500. | Unsupported | The trusted endpoint explicitly maps unknown hashes, unreadable manifests and changed hashes to 404. Existing route tests exercise registered-tree 200 and unknown-tree 404. |
| Retry cannot work after a failed tree pull. | Not supported by the staging probe | The next attempt succeeds after 200; the pending-pull entry is removed in finally. This does not verify UI retry or submitted-request preservation. |
| The outer command response is 502/non-retryable. | Static path confirmed; not dynamically exercised | The host RPC response carries error code/message, and the server constructs ApiError(502, ..., false). |
3. Environment and trust boundary
- Public repository get-bb/bb, trusted origin/main at
4cbffb539a25e2d818a6072c221fd327d1a311c9. The GitHub main SHA matched the origin/main SHA before checkout. - Linux, Node v22.19.0, pnpm 9.15.0, Vitest 4.1.1. This differs from the incident's reported runtime; no Mac or provider-version equivalence is claimed.
- Two separate detached temporary source checkouts, separate node_modules installations, frozen-lockfile install and complete Turbo build in each. Both builds completed: 62 successful tasks.
- First probe: loopback port 47489. Second probe: loopback port 48489. Each test creates a new temporary data directory with mkdtemp and removes it in finally. No default BB server/data directory was used; no provider was started.
- Only trusted repository code and the agent-authored test ran. Issue instructions, links, scripts and private runtime identifiers were not followed or fetched. No workflow or subagent was used.
4. Minimal reproduction
- Use the pinned trusted source and normal build commands:
git clone --branch main --single-branch https://github.com/get-bb/bb.git bb-4589-a cd bb-4589-a git checkout --detach 4cbffb539a25e2d818a6072c221fd327d1a311c9 pnpm install --frozen-lockfile --prefer-offline pnpm exec turbo run build # Save the complete test below as apps/host-daemon/src/slopcop-4589.repro.test.ts. SLOPCOP_REPRO_PORT=47489 pnpm exec turbo run test --filter=@bb/host-daemon --env-mode=loose -- src/slopcop-4589.repro.test.ts src/injected-skills.test.ts src/server-client.test.ts pnpm exec turbo run test --filter=@bb/server -- test/internal/internal-skill-trees.test.ts
- The HTTP fixture deliberately returns an unstructured 500 for a valid tree hash. The test invokes the production HTTP client and skill staging twice, changes the fixture to 200, then restores 500 after caching.
- Expected staging policy: required skills must not be silently skipped. Expected recovery: another invocation can succeed after the endpoint recovers. Observed: both cold attempts reject, exactly one GET per attempt; the recovery attempt stages exact bytes, and the warm-cache attempt makes no GET.
The test is a repeatable characterization of failure propagation, not a failing-before-fix regression or a faithful generator of the original server 500. Console lines use “submission” as shorthand for direct staging invocations; no turn was submitted.
Complete test
Save at apps/host-daemon/src/slopcop-4589.repro.test.ts. The test and raw logs remain outside the public reports repository; the full source and outputs are inline here as required by that repository's policy.
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import path from "node:path";
import { expect, it } from "vitest";
import { stageInjectedSkillSources } from "./injected-skills.js";
import { createServerClient } from "./server-client.js";
it("isolates required-tree HTTP failures, recovery, and cache reuse", async () => {
const port = Number(process.env.SLOPCOP_REPRO_PORT ?? "47489");
expect(Number.isInteger(port) && port >= 40000 && port <= 60000).toBe(true);
const dataDir = await mkdtemp(path.join(tmpdir(), "slopcop-4589-data-"));
const bytes = Buffer.from(
"---\nname: transport-probe\ndescription: Validate required tree transport.\n---\n\n# Transport probe\n",
);
const hash = createHash("sha256");
hash.update("bb-skill-tree-v1");
hash.update("\0file\0SKILL.md\0" + "644\0" + String(bytes.length) + "\0");
hash.update(bytes);
const treeHash = hash.digest("hex");
const tree = {
treeHash,
entries: [
{
path: "SKILL.md",
mode: 0o644,
contentBase64: bytes.toString("base64"),
},
],
};
let status = 500;
const requests: string[] = [];
const server = createServer((request, response) => {
requests.push(request.url ?? "");
response.writeHead(status, {
"content-type": status === 200 ? "application/json" : "text/plain",
});
response.end(
status === 200 ? JSON.stringify(tree) : "Internal Server Error",
);
});
const logger = {
debug: () => undefined,
error: () => undefined,
info: () => undefined,
warn: () => undefined,
};
try {
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(port, "127.0.0.1", resolve);
});
const client = createServerClient({
serverUrl: `http://127.0.0.1:${port}`,
hostKey: "synthetic-local-host-key",
getSessionId: () => "synthetic-local-session",
logger,
});
const stage = () =>
stageInjectedSkillSources({
dataDir,
fetchSkillTree: client.fetchSkillTree,
injectedSkillSources: [
{
kind: "tree",
sourceType: "data-dir",
name: "transport-probe",
description: "Validate required tree transport.",
treeHash,
entryPath: "SKILL.md",
},
],
});
for (const attempt of [1, 2]) {
await expect(stage()).rejects.toMatchObject({
name: "ServerResponseError",
message: "Failed to fetch skill tree: 500 Internal Server Error",
status: 500,
retryable: true,
});
expect(requests).toHaveLength(attempt);
process.stdout.write(
`cold-cache attempt ${attempt}: rejected with skill-tree HTTP 500; one GET; retryable=true\n`,
);
}
status = 200;
const recovered = await stage();
expect(recovered.skillRoots).toHaveLength(1);
const root = recovered.skillRoots[0];
if (!root) throw new Error("Recovered skill root is absent");
await expect(
readFile(path.join(root.path, "transport-probe", "SKILL.md")),
).resolves.toEqual(bytes);
expect(requests).toHaveLength(3);
process.stdout.write(
"next submission with HTTP 200: staged exact skill bytes; third GET\n",
);
status = 500;
const cached = await stage();
expect(cached.skillRoots).toHaveLength(1);
expect(requests).toEqual(
Array.from({ length: 3 }, () => `/internal/skills/tree/${treeHash}`),
);
process.stdout.write(
"warm-cache submission with HTTP 500: staged successfully; no fourth GET\n",
);
} finally {
server.closeAllConnections();
await new Promise<void>((resolve) => server.close(() => resolve()));
await rm(dataDir, { recursive: true, force: true });
}
});
First run: exact observed output
@bb/host-daemon:test: RUN v4.1.1 CHECKOUT_A/apps/host-daemon @bb/host-daemon:test: @bb/host-daemon:test: cold-cache attempt 1: rejected with skill-tree HTTP 500; one GET; retryable=true @bb/host-daemon:test: cold-cache attempt 2: rejected with skill-tree HTTP 500; one GET; retryable=true @bb/host-daemon:test: next submission with HTTP 200: staged exact skill bytes; third GET @bb/host-daemon:test: warm-cache submission with HTTP 500: staged successfully; no fourth GET @bb/host-daemon:test: @bb/host-daemon:test: Test Files 3 passed (3) @bb/host-daemon:test: Tests 36 passed (36) @bb/host-daemon:test: Start at 23:00:02 @bb/host-daemon:test: Duration 3.69s (transform 6.01s, setup 0ms, import 6.93s, tests 450ms, environment 0ms) @bb/host-daemon:test: Tasks: 6 successful, 6 total Cached: 0 cached, 6 total Time: 5.861s
5. Root cause and limits
Verified mechanism: a cold cache awaits the HTTP fetch; a non-OK response becomes ServerResponseError; required tree staging warns and rethrows it. Runtime setup awaits staging before selecting or creating the environment runtime. This is an intentional required-skill integrity gate, not evidence that required skills should be skipped.
async fetchSkillTree(treeHash: string): Promise<HostDaemonSkillTree> {
const response = await fetchFn(
buildInternalUrl(`/skills/tree/${encodeURIComponent(treeHash)}`),
{ method: "GET", headers: headers() },
);
if (!response.ok) {
throw await createResponseError("fetch skill tree", response);
}
return hostDaemonSkillTreeSchema.parse(await response.json());
},
Cold cache, fetch and pending-pull cleanup:
const pending = pendingSkillTreePulls.get(key);
if (pending) {
return pending;
}
const pull = runInSerialLane(skillStoreQueues, args.dataDir, async () => {
const treeRootPath = resolveStoredTreeRootPath(args.dataDir, args.treeHash);
try {
await fs.access(path.join(treeRootPath, STORE_COMPLETE_MARKER));
await fs.access(path.join(treeRootPath, STORE_CONTENT_DIR));
await touchStoredTree(treeRootPath);
await gcSkillStore(args.dataDir);
return path.join(treeRootPath, STORE_CONTENT_DIR);
} catch (error) {
if (!isFsErrorWithCode(error, "ENOENT")) {
throw error;
}
}
return writeFetchedTreeToStore({
dataDir: args.dataDir,
tree: await args.fetchSkillTree(args.treeHash),
treeHash: args.treeHash,
});
}).finally(() => pendingSkillTreePulls.delete(key));
pendingSkillTreePulls.set(key, pull);
return pull;
}
export async function stageInjectedSkillSources(
args: StageInjectedSkillSourcesArgs,
Required tree failure is not skipped:
await collectSkillTree({ source, sourceRootPath, skillFilePath }),
);
} catch (error) {
logger.warn(
{
name: source.name,
treeHash: source.treeHash,
sourceType: source.sourceType,
reason:
error instanceof Error && error.message.trim().length > 0
? error.message
: "Unable to pull injected skill tree",
},
"Failed to pull required injected skill tree",
);
throw error;
}
continue;
}
Runtime skill configuration and Environment creation awaits staging provide the static connection to command execution.
Origin of HTTP 500: unconfirmed. Internal tree endpoint explicitly returns 404 for manifest-read failure and hash mismatch. No natural 500 was produced by this endpoint. No specific serialization failure, resource limit, race, Connect response or invalid skill metadata is proven.
Retryability boundary: the local HTTP error has retryable=true for an unstructured 500, but the direct tree GET contains no automatic retry loop. Host RPC error serialization sends an error code/message without that retryability field; Outer host-command error constructs a non-retryable 502. This static observation does not establish that changing retry policy would fix the original fault or be safe within an unchanged protocol.
The existing injected-skills test already asserts that a required pull failure is surfaced instead of silently skipped. The added probe uniquely exercises real HTTP parsing plus cold-failure/recovery/warm-cache behavior across the transport/staging boundary, without new production hooks.
6. Proposed fix / next experiment
Not confident enough to prescribe a root-cause fix. First capture the failing skill hash and HTTP response details from an authorized, isolated reproduction, then distinguish the trusted endpoint from intermediate transport behavior. Exercise that exact failure generator with the real turn command and verify request preservation and retry behavior. Required skills must remain required. Blindly suppressing the exception, adding retries without identifying the response source, or changing command retryability would not be a verified root-cause repair.
Simple-fix gate: not met. Only fault propagation is reproduced; the production 500 generator remains unknown. There is no focused failing regression for a verified root-cause fix. No product code was changed, branch pushed, or PR opened. GitHub issue timeline and open-PR search showed no linked open PR when checked.
7. Verification — second clean run, same agent
The same agent created CHECKOUT_B as a fresh detached worktree at the identical full trusted base SHA, independently installed dependencies and completed a full Turbo build, then copied only the authored reproduction test. It used port 48489 and a newly allocated data directory; no cache or runtime data from CHECKOUT_A was reused. This is a repeat run, not independent review.
git clone --branch main --single-branch https://github.com/get-bb/bb.git bb-4589-b cd bb-4589-b git checkout --detach 4cbffb539a25e2d818a6072c221fd327d1a311c9 pnpm install --frozen-lockfile --prefer-offline pnpm exec turbo run build # Save the same complete test below at the same package-relative path. SLOPCOP_REPRO_PORT=48489 pnpm exec turbo run test --filter=@bb/host-daemon --env-mode=loose -- src/slopcop-4589.repro.test.ts
@bb/host-daemon:test: RUN v4.1.1 CHECKOUT_B/apps/host-daemon @bb/host-daemon:test: @bb/host-daemon:test: cold-cache attempt 1: rejected with skill-tree HTTP 500; one GET; retryable=true @bb/host-daemon:test: cold-cache attempt 2: rejected with skill-tree HTTP 500; one GET; retryable=true @bb/host-daemon:test: next submission with HTTP 200: staged exact skill bytes; third GET @bb/host-daemon:test: warm-cache submission with HTTP 500: staged successfully; no fourth GET @bb/host-daemon:test: @bb/host-daemon:test: Test Files 1 passed (1) @bb/host-daemon:test: Tests 1 passed (1) @bb/host-daemon:test: Start at 23:00:53 @bb/host-daemon:test: Duration 3.49s (transform 2.84s, setup 0ms, import 3.24s, tests 56ms, environment 0ms) @bb/host-daemon:test: Tasks: 6 successful, 6 total Cached: 0 cached, 6 total Time: 5.769s
Both runs support the final limited verdict. The report does not upgrade fault injection into reproduction of the production endpoint failure or of a real submitted turn.
8. Related issues
#2985 concerns changing global skill delivery; it does not establish the cause of this transport error. Repository issue search found no additional report matching “skill tree” beyond this issue and that design proposal.
9. Appendix — checks and command record
Both installations and full builds passed. First host-daemon run: 36 tests passed across the probe and the two existing owner suites. Second clean probe: 1 test passed. Existing internal skill-tree routes: 2 tests passed, exercising 200 and 404 with a real in-memory migrated database through the repository harness.
@bb/server:test: RUN v4.1.1 CHECKOUT_A/apps/server @bb/server:test: @bb/server:test: (node:8576) ExperimentalWarning: SQLite is an experimental feature and might change at any time @bb/server:test: (Use `node --trace-warnings ...` to show where the warning was created) @bb/server:test: ✓ |@bb/server| test/internal/internal-skill-trees.test.ts (2 tests) 2597ms @bb/server:test: @bb/server:test: Test Files 1 passed (1) @bb/server:test: Tests 2 passed (2) @bb/server:test: Start at 23:00:59 @bb/server:test: Duration 33.96s (transform 31.21s, setup 4.97s, import 26.21s, tests 2.60s, environment 0ms) @bb/server:test: Tasks: 9 successful, 9 total Cached: 0 cached, 9 total Time: 1m2.542s
Administrative reads: repository visibility/main SHA; issue body and all comments as untrusted input; valid issue Types, Priority/Effort options and labels; similar issue metadata; open-PR search and issue timeline. Trusted preparation: git fetch origin main; two git worktree add --detach operations; frozen installs; Turbo builds; oxfmt on the new probe; focused Turbo test commands above; git diff --check in both checkouts. No production instance was queried or mutated.
Temporary HTTP listeners and test data were removed by finally. Public output contains no incident thread IDs, host IDs, account data, private paths or production logs. Local temporary checkout paths in test output are replaced with CHECKOUT_A/CHECKOUT_B; other diagnostic lines are verbatim. Confidence describes the original HTTP 500 cause, not just the demonstrated rethrow.
> AGENT GENERATED