01The problem
AI coding agents report success without proof. When the same model writes and approves its own work, bugs ship and nobody can tell.
Overview30 seconds
AI agents say “done”. Dolphin Tank makes them prove it.
Dolphin Tank is a software factory made of five AI agents in one BAND room. Four build and attack the code. One independent reviewer may only accept what it reproduces itself.
02The fix
Split the authority. Route plans. Forge and Loom build. Trace attacks the build. Jury alone accepts or rejects, and only on evidence it reproduced on a clean clone.
03The result
Pocketful: a four-stage wallet and payments service in Go with a browser app. No human wrote the stage code. 4 of 4 stages passed the official harness from a fresh clone.
04The proof
At stage 3 Jury rejected the build with two reproduced bugs. The agents fixed both, a cross-attack found a third, and Jury re-verified before it accepted. The whole room log is public.
How it meets the three criteria
| Criterion | What we did | Evidence |
|---|---|---|
| Factory50% | Generic mandates any team could point at a new problem. A runbook with exact commands. Cost and time measured: $196.92 estimated spend, 10.2 h of active room time. A loop that catches bad work and repairs it. | FACTORY.md, Exhibit F, Exhibit E |
| App25% | A coherent browser app with refused, uncertain, empty and error states, tested by Jury in Chromium at 375, 768 and 1280 px. | Exhibit J |
| Agent teamwork25% | Five agents shared the work, 1,018 to 1,770 messages each, none above 31% of cost. A review changed the build. One dispatch per stage, plus 3 resume messages after provider limits, all listed. | Exhibit G, Exhibit I |
Autonomy is our weakest point and we say so. Three resume messages were sent after usage limits stopped agents. They added no requirements. Exhibit I lists every human message.
Case fileWeAreDevelopers x BAND Dark FactoryPocketful track
The rejection, in Jury's words.
At stage 3, Jury rejected the build and posted this to the room. Everything on the right is verbatim from the room export.
- Acceptance
- 4 of 4 stages accepted by Jury
- S1 0024598, S2 f33035a, S3 5e6f83f after the REJECT of 3c7c411, S4 1dd5560.
- Human input
- 7 messages, 2,067 words
- Four stage dispatches and three resumes after usage-limit stalls. The seats wrote about 103,000 words. No seat ever asked the human a question.
- Active room time
- 10.2 h of 68.9 h wall clock
- Measured from event timestamps. About 58.7 h was idle. Method in Exhibit E.
- Estimated cost
- $196.92 at list prices
- An estimate from the Band usage rooms, not a bill. 55 sessions, 487,800,071 tokens.
- Repository
- 100 commits, MIT
- Public. HEAD d984122. History unrewritten.
- Fresh-clone proof
- Stages 1 to 4 each claim their own
- Official harness, isolated mode, from a fresh clone of the public repository at e7ea0b6. Summary file.
Exhibit BFilm
The film
116 seconds. A hook, the five agents, a real screen recording of the Band Desktop room (tagged on screen), the product, the rejection in Jury's own words, the repair, and the receipt. Quotes come from the room export. The narration is synthesized speech. Captions are burned in, and WebVTT and SRT files are provided too.
The film is 29.79 MB and starts downloading when you press play. The poster is the frame at 0:47, the stage 3 rejection.
Read the film as text
- AI coding agents say done.
- Who checks?
- This is the Band room, and that is a real recording of it.
- Five agent seats work here: route, forge, loom, trace, jury.
- Seven thousand two hundred sixteen messages.
- None of them edited.
- One human wrote seven of them: four dispatches and three resumes.
- Route hands Forge a work order with the specification hash,
- the ledger rows that define done, and the commit.
- Then Route hands Jury one commit,
- and asks for a verdict on that commit only.
- Jury says no.
- It reproduced two failures on its own.
- A payment of eight thousand one went through,
- against eight thousand available.
- And a balance read minus four thousand,
- in twenty-five of twenty-five views.
- Route does not argue.
- It names the seat that owns each fix.
- Forge takes one.
- Trace takes the other.
- Jury rebuilds from a clean clone, reruns everything, and accepts.
- Stage four was accepted as well.
- Six hundred ninety-seven of Jury's own checks,
- zero failures,
- and eleven deliberate faults out of eleven caught.
- This is the product they built, and it is running now.
- Sign in.
- Try to pay more than the wallet holds.
- Refused.
- Send a payment that fits.
- It lands.
- The receiver refunds part of it.
- One hundred commits at the revision we submitted.
- Seven thousand two hundred sixteen messages.
- Seven of them human.
- Sixty-eight point nine hours of wall clock,
- about ten of them active.
- Dolphin Tank.
- Agents build.
- A reviewer proves it.
Exhibit CWalkthrough
Follow one defect, from spec to fix
Defect F2, in five steps. Every excerpt is cut from a message in room.json and left unedited. The @-mention tag that opens each message is dropped.
-
INPUTThe requirement
Jury wrote its requirement ledger from the spec before it read any source. Rows J44 and J45 turn this part of the stage 2 and stage 3 specs into something an outside client can check: the four money fields of a historical view describe one view, and available is never negative.
Jury Spec: stage-2 "available = total - held, never negative"; "requirements hold at every read"; stage-3 "all four money fields describe that same view"; and corrections are rejected when available would go negative at a past boundary.
Verbatim from room.json. 231 of 5,842 characters shown, cuts marked with an ellipsis.
-
PROCESSJury reproduces it
Jury ran a check it wrote itself, section H5 of
t3_holds.py, against a clean clone of candidate 3c7c411. It also named the cause it could see from outside the service: two clocks with two precisions.Jury Authorization created_at is whole-second, but payments are stamped to the microsecond. The hold is placed at the truncated public instant, so a payment that really came before the hold carries a later created_at.
Reproduction: checks\t3_holds.py section H5. Ada funds bob with 5000, bob immediately authorizes cy for 4000, then GET /me?as_of=<that authorization's created_at> for bob.
Verbatim from room.json. 383 of 5,842 characters shown, cuts marked with an ellipsis.
-
REAL OUTPUTWhat came back
A balance that cannot exist, in 25 of 25 views, in each of three runs. The second iteration is worse because the previous hold was already voided but still counted as open.
Jury payment.created_at 07:20:07.357895, authorization.created_at 07:20:07+00:00.
Result: total 0, held 4000, available -4000. The next iteration shows held 8000, available -8000, because the earlier authorization (voided a few ms before) is still counted open at the truncated second.
Violated in 25 of 25 views in each of 3 runs (logs\t3_holds.run1.log, run2.log, run3.log).
Expected: total 5000, held 4000, available 1000.
Verbatim from room.json. 417 of 5,842 characters shown, cuts marked with an ellipsis.
-
REPAIRFix, attack, fix
Jury reports a defect and stops. It never fixes what it rejects. Route accepted both defects as violations in the room. Forge repaired F2 and Trace repaired F1. Then Trace, who did not write the F2 repair, attacked it and found a third defect, L-F2-1: a legacy export imported with a hold sitting before the payment that funded it.
Route route: F1 and F2 accepted as violations. Repairs are committed; no hash sent yet because the candidate is not final.
- F2 (forge) 11e76f9: authorization created_at, expires_at, closed_at use the payment clock and microsecond precision, so expires_at - created_at is exactly the ttl and every view places a hold at the instant of the payments that funded it. D-HOLD-TIME revoked in decisions.md.
Verbatim from room.json. 393 of 1,726 characters shown, cuts marked with an ellipsis.
Route Resumed. Trace's 849b017 cross-attack of F2 (11e76f9) at ac96360 passed every attack: jury scripts, s2reg, ttl, deadline race, storm and import. It found one defect, L-F2-1. A legacy 3c7c411 export imports into HEAD with created_exact ignored, so a hold can sit before its funding payment (available -2000).
I sent forge the full handoff for a small fix on the import path (created_at = created_exact when valid), with the spec hash and acceptance criteria.
Verbatim from room.json. 456 of 618 characters shown, cuts marked with an ellipsis.
-
RESULTRe-verification
Jury rebuilt the image from a clean clone of 5e6f83f and reran the same check: the F2 reproduction now scores 59 pass, 0 fail. It also wrote new checks and ran each against the rejected commit as a control. The sweep check fails there.
Jury S3-JURY-R3 VERDICT: ACCEPT. Commit verified: 5e6f83f0a2d9fe9e09615341dde23a376d273967 (stage 3).
5. F2 (t3_holds H5, as_of = authorization.created_at after back-to-back pay+authorize): 59/0 x3 (3c7c411: 25 of 25 views had available<0).
r3_sweep.py, seeds 1-6: random back-to-back lifecycle (fund+authorize pairs, pays, partial/final captures, voids, 2 s TTL expiry), then 4 users read at every payment/authorization/closed_at/expires_at instant +-1us via /me?as_of against an oracle built only from public timestamps.
118/115/130/136/124/136 passes, 1464-1680 views per seed, 0 fail, 0 with available<0; closed_at == last capture created_at exactly; expires_at-created_at == ttl. Control 3c7c411: 3 FAIL.
Verbatim from room.json. 701 of 6,648 characters shown, cuts marked with an ellipsis.
Full files: verdict.md (REJECT) and r3-verdict.md (ACCEPT).
Exhibit DRun timeline
The run, hour by hour
From the first dispatch on 30 September to the last room message on 3 October. Times are UTC. The REJECT is the boxed diamond.
-
Stage 1 dispatched Dispatch
Human message 1 of 7. Route is told to coordinate the other four seats and build stage 1.
-
Resume after a usage limit Resume
Human message 2, to Jury. Its turn was cut by the model provider. No new requirement.
-
Stage 1 accepted 0024598 Accept
Jury ran its own conservation, idempotency and race checks on a clean checkout, 88 ledger rows, all pass. Shipped checks: 147 of 147.
-
Stage 2 dispatched Dispatch
Authorizations, captures, holds, expiry against a real clock. Human message 3.
-
Resume after the limit window reset Resume
Human message 4, to Route.
-
Stage 2 accepted f33035a Accept
78 ledger rows. Shipped checks: 147 plus 35. The accepted stage 2 behaviour later becomes the yardstick that catches defect F1.
-
Stage 3 dispatched Dispatch
History: statements, snapshots, corrections, the hold timeline over time. Human message 5.
-
Jury rejects stage 3 3c7c411 Reject
F1: a seeded value ratcheted the service clock about two hours into the future, so a payment of 8001 against 8000 available was accepted (201) instead of refused (409).
F2: whole-second hold timestamps against microsecond payments gave
available = -4000inGET /me?as_of, 25 times out of 25. -
Two repairs 11e76f9 ac96360 Repair
Forge repairs F2 in 11e76f9. Trace repairs F1 in ac96360. The F2 fix revokes a decision the authors had written down (
D-HOLD-TIME). Route accepts both defects as violations in the room. -
Work declined on the record 71d08d4 Declined
D-IMPORT-FUTURE, a hardening idea for future-dated imports, was considered and rejected as outside the spec instead of being built. -
Trace cross-attacks the repair 849b017 Attack
Trace did not write the F2 repair it attacked. It found a third defect,
L-F2-1: a legacy export imported withcreated_exactignored. -
Third resume Resume
Human message 6, to Route: its last turn ended before it answered Trace's report. No scope change.
-
Third repair 5e6f83f Repair
Forge makes import place a hold at
created_exactwhen it differs fromcreated_at. -
Stage 3 accepted 5e6f83f Accept
Jury re-verifies from a clean clone, including
go test -racein a container. The F2 reproduction now scores 59 pass, 0 fail. The F1 reproduction behaves like stage 2. -
Stage 4 dispatched Dispatch
Refunds, batch corrections, the browser refund control. Human message 7, the last.
-
Stage 4 accepted 1dd5560 Accept
Three Jury verdicts: refunds f2698c2, batch corrections, and the browser product with Docker and regression. No rejection and no cross-attack defect at this stage. Route's final report follows at 20:00.
Verdict and dispatch times are room-message times. Repair times are commit times converted to UTC. The stage 4 evidence commit f2d512b is stamped 08:25 UTC, earlier than the 19:58 verdict message, because the acceptor's turn was interrupted by the provider limit and re-delivered (FACTORY.md, section 10).
Exhibit EActivity chart
Where the 68.9 hours went
The room was busy for about 10 of the 69 hours. The rest was waiting. This chart is computed from every event in room.json by tools/build.py.
- Wall clock
- 68.9 hfirst to last event
- Active
- 10.2 h14.8 percent
- Idle
- 58.7 h85.2 percent
- Sensitivity
- 8.7 to 10.6 h3 and 30 minute thresholds
- Events per 15 minutes
- Idle gap, 2.5 h or more
- Human message, 1 to 7
- Stage accepted
- Stage 3 rejected
- One commit
The method
Sort all 7,216 event timestamps. Add the gaps between consecutive events that are 10 minutes or shorter. That sum, 10.2 hours, is active room time. A 3 minute threshold gives 8.7 hours and a 30 minute threshold gives 10.6 hours, so the answer does not hinge on the choice. Everything else, 58.7 hours, is idle.
Active room time is read off the log. It is not metered compute, and a seat thinking silently for more than 10 minutes counts as idle.
What the idle time is
Our reading: the provider's usage-limit windows. Seven gaps run 4.3 to 5.4 hours, the length of a five-hour session window. One gap runs 10.7 hours. Four more run 2.9 to 3.8 hours. Together the twelve bands shaded above add up to 57.0 of the 58.7 idle hours.
The export holds 32 error events. 30 of them name a session or weekly usage limit and the other 2 are failed background tasks. The log does not say which gap belongs to which limit.
Exhibit FProcess
The evidence loop
Each Pocketful stage ships only part of its checks: 79 percent, 35, 9 and 16 percent of the suite for stages 1 to 4. A green run on those says little about the specification, so Jury treats the gap as the job. It may not accept on the shipped checks alone.
Specification
Read first and hashed. The spec's sha256 appears in the handoff and in every verdict.
evidence/stage-N/jury/verdict.md
Requirement ledger
Written from the spec before any source is read. Jury's ledgers hold 88, 78 and 58 rows for stages 1 to 3, and stage 3 gained 14 more at re-verification.
evidence/stage-N/jury/ledger.md
Observable property
What an outside client could see if the requirement held.
ledger column "how to observe"
Is it covered by a shipped check?
YesReproduce it: run the check on a clean clone anyway.
NoDerive an attack: write a new check for exactly that gap.
Evidence
Commands, counts and logs, including the failed runs.
evidence/stage-N/jury/checks/ and logs/
ACCEPT or REJECT
Jury only. It reports a defect with a reproduction and stops. It never fixes what it rejects.
evidence/stage-N/jury/verdict.md
What does the specification require that the supplied checks never demonstrated?
Jury must answer this in writing before it may accept. The answers are in evidence/stage-N/jury/gaps.md.
Exhibit GAgents
Who did what
Three agents build, one coordinates, one decides. The builders attack each other's work. Nobody accepts their own.
| Seat | Owns | Room messages | Estimated cost | What it did that mattered |
|---|---|---|---|---|
| Route@route | Coordination: reads the spec, writes the handoffs, keeps the decision log, routes acceptance. | 1,244 | $27.35 13.9% | Accepted F1 and F2 as violations in the room. Held back the commit hash until the candidate was final. |
| Forge@forge | Backend: domain rules, validation, persistence, service internals. | 1,420 | $35.95 18.3% | Repaired F2 in 11e76f9 and the third defect in 5e6f83f. Attacked batch corrections with 297 checks in 12 scenarios, all pass. |
| Loom@loom | Interface: the browser application, its states, accessibility, packaging. | 1,018 | $23.94 12.2% | Built the browser product and the refund control that Jury tested in Chromium at 375, 768 and 1280 px. |
| Trace@trace | Systems and attacks: concurrency, time, history, migrations, idempotency. Attacks work it did not write. | 1,756 | $50.31 25.5% | Repaired F1 in ac96360. Attacked Forge's F2 repair and found L-F2-1. Ran 83 black-box checks on refunds crossed with batch corrections, all pass. |
| Jury@jury | Independent acceptance. The only seat that may ACCEPT or REJECT. Builds from a clean clone and writes its own checks. | 1,770 | $59.37 30.1% | Rejected 3c7c411 with two reproduced defects. On stage 4 batch corrections: 697 own checks, 11 of 11 planted faults caught. |
| Total | Seats only. The room holds 7,216 messages: these 7,208, 7 human messages and 1 join notice. | 7,208 | $196.92 | List-price estimate from the Band usage rooms: 55 sessions, 487,800,071 tokens. Not a provider bill. |
All five seats ran Claude Code with the model claude-sonnet-5-5, as stated in the headers of mandates/*.md. The mandates are generic: they name no endpoint, field or track term.
Exhibit HVerification
Fresh-clone results
The official harness, unmodified, in isolated mode: an internal network with outbound traffic blocked, so the run also shows the service needs no network. Each folder claims the highest stage it passes and no more.
| Folder | Suites passed | Next stage's suite | Claims |
|---|---|---|---|
| stage-1/ | 147/147 | fails, as it should | 1 |
| stage-2/ | 147 + 35 | fails, as it should | 2 |
| stage-3/ | 147 + 35 + 6 | fails, as it should | 3 |
| stage-4/ | 147 + 35 + 6 + 5 | final stage | 4 |
From a brand-new clone of the public repository at e7ea0b6, the harness in --all --mode isolated claims stages 1, 2, 3 and 4 and none overshoots. The raw summary is public-clone-summary.json: every folder reports claimed: true, share: 1.0 and overshoot: null. An earlier fresh-clone run at e79ea48 is committed under evidence/operator/final-fresh-e79ea48/.
| Who | Target | Result |
|---|---|---|
| Jury | Stage 4 batch corrections, own checks, with deliberate faults planted | 697 checks, 0 fail 11 of 11 faults caught |
| Trace | Attack on refunds crossed with batch corrections, black box | 83 checks, all pass |
| Forge | Attack on batch corrections, 12 scenarios | 297 checks, all pass |
| Jury | F2 reproduction (t3_holds H5) on the repaired commit 5e6f83f | 59 pass, 0 fail 25 of 25 views failed on 3c7c411 |
Exhibit IAutonomy
Seven human messages, all listed
The room holds 7,216 messages. Seven are human text: four stage dispatches the guide allows, and three resumes after the model provider's usage limit cut a seat's turn. The seats wrote about 103,000 words against 2,067 from the human.
| # | UTC | To | Kind | Words |
|---|---|---|---|---|
| 1 | 2026-09-30 23:05 | route | Dispatch Stage 1 | 322 |
| 2 | 2026-10-01 05:21 | jury | Resume usage limit, addressed to Jury directly | 41 |
| 3 | 2026-10-01 05:34 | route | Dispatch Stage 2 | 537 |
| 4 | 2026-10-01 15:49 | route | Resume limit window reset | 44 |
| 5 | 2026-10-01 21:57 | route | Dispatch Stage 3 | 543 |
| 6 | 2026-10-02 17:42 | route | Resume last turn ended before answering Trace | 42 |
| 7 | 2026-10-02 22:48 | route | Dispatch Stage 4 | 538 |
| Dispatches 1,940 words. Resumes 127 words, with no requirement in any. | 2,067 | |||
What holds
No seat ever asked the human for anything. Every message the seats sent back to the operator is a status report. No requirement, scope or acceptance decision changed because of a human message, and the three resumes say so in 41, 44 and 42 words.
Operator restarts of the seat runtimes after a usage limit are a daemon action, not a room message. They added nothing to the room.
What does not
The participant guide says to send nothing between a stage dispatch and the final report. Three messages were sent. Calling them crash recovery does not create an exemption the guide never gave.
The first resume is the weakest: it went to Jury directly, not through the coordinator. This run is not unambiguously compliant with the guide's no-input rule, and you should read that from us first.
Exhibit JProduct
The Pocketful app, as shipped
Nine real screenshots of the stage 4 build, in the order a visitor meets them. They are Jury's Chromium captures on test data, cropped from full-page screenshots under evidence/stage-4/jury/final/browser/shots/run1/. The dark theme is the product's own. State is carried by label and glyph, never by colour alone.
Figure 1, authorizations, 1280 px
Available is the headline
- 1Available to spend, 80.00 EUR. Total and on hold sit beneath it. F2 made this field go negative in historical views.
- 2Expired hold, marked by its label and a dash glyph.
- 3Voided hold, marked by a cross glyph.
- 4Captured hold, with the captured 3.00 EUR shown.
- 5Live hold from another user, with an amount field and a Capture control.
- 6Live hold placed by this user, with a Void hold control.
authorizations_1280.png · full size
Figure 2, refund control, 1280 px
Four refund outcomes on one screen
- 1The remaining refundable amount, 7.00 EUR of a 10.00 EUR payment, prefilled.
- 2A completed partial refund: 3.00 EUR refunded to bob.
- 3A payment with nothing left to refund.
- 4An unconfirmed outcome. The page says nothing is lost and that a retry sends the same refund once.
- 5An input error for
abc, with an example of a valid amount.
b01_wallet_refund_states_1280.png · full size
Figure 3, refund states, 375 px
The same states on a phone
Refunded payments carry a Refund of label. The unconfirmed notice and the input error keep their wording at 375 px.
b01_wallet_refund_states_375.png · full size
Figure 4, refused refund, 375 px
A refusal with a plain reason
A refund above the remaining amount is refused and says why.
b12_refused_stale_375.png · full size
Figure 5, refused payment, 375 px
Money on hold cannot be spent
The wallet has 0.10 EUR available, so a 50.00 EUR payment is refused.
pay_refused_375.png · full size
Figure 6, requests, 375 px
Pending, Declined, Paid, Cancelled
Each request state has its own label and glyph, and controls only where they apply.
requests_375.png · full size
Figure 7, split preview, 375 px
Shares shown before anything is sent
Five cents across five people gives 0.01 EUR each.
split_preview_375.png · full size
Exhibit KReproduce
Run it yourself
Each stage folder ships a Dockerfile and a RUN.md. These are the commands from stage-4/RUN.md.
Build and start, stage 4stage-4/RUN.md
git clone https://github.com/vaibhav4046/dolphin-tank
cd dolphin-tank/stage-4
docker build -t pocketful-s4 .
docker run --rm -e PORT=8080 -p 8080:8080 pocketful-s4Health checkstage-4/RUN.md
curl -s http://localhost:8080/health
# {"status":"ok"}- Building needs the network once, for the Go base image. The running container needs no network and no setup.
- State is in memory. The browser UI, its scripts, stylesheets and fonts are embedded in a single binary, so nothing is fetched at run time.
- Open http://localhost:8080/ for the wallet. The JSON routes and the test-control endpoints are listed in stage-4/RUN.md.
- To run the Go tests (Go 1.24 or later):
go vet ./... && go build ./... && go test ./.... Browser logic needs Node 20 or later:node --test jstest.
Exhibit LLimits
Known gaps, stated by us
- Two accessibility findings left unpatched
- Both are low severity and both sit in band-owned code. The wallet route has four
h2sections and noh1. One link on the signup route measures 40 by 44 px at 390 px wide, 4 px short of the 44 px target. A fix request mid-run would have been the steering the guide prohibits. - No race detector at stage 4 or on the operator host
go test -raceneeds cgo and the operator machine has no compiler, so it never ran there. Jury ran it inside a container at stage 2 (ok in 45.6 s) and stage 3 (ok in 363 s). It was not run at stage 4.- Browser testing used Chromium only
- Jury's browser suites ran in Playwright Chromium at 375, 768 and 1280 px. No other engine was tested.
- Three resumes, one weak
- The run is not unambiguously compliant with the guide's no-input rule. Exhibit I lists all three messages and says which one is weakest.
- One git identity
- All commits share one identity, so authorship cannot show which seat wrote what. room.json is the provenance authority. evidence/operator/PROVENANCE.md maps each commit it indexes to the room message that announced it: 64 of 80 have one.
- The shipped checks are thin
- They cover 79, 35, 9 and 16 percent of the four stage suites. Independent checks carried acceptance. We make no claim about the withheld tests.
- Cost and time are estimates
- $196.92 is a list-price estimate, not billing. The 10.2 hours of active room time is inferred from event timestamps, not metered compute.
- One model behind five seats
- All five seats share one harness and model, which the guide allows. Jury is an independent role inside that one model, so its independence is separation of duties.
Who should not rely on this
EndNothing hidden
That is everything.
The room export is unedited. The limits are listed above. Exhibit K has the commands to run it yourself.