✕ What brokeThe browser tool said 'Add to cart' worked. The cart stayed empty: 0 of 5 attempts got through end to end.
✓ What changedEvery action now checks that the page actually changed, and climbs a recovery ladder when it did not: 5 of 5.
BuildingFirst Userin neatHack
From the first agent to 7 of 7 verified, two rounds in a row. What broke at each step, and what changed because of it.
v00verified goals
v67/7verified, two rounds
- ✕ What broke0 verified goals: nothing checked the agent's own claims. Two runs failed before the first step and two ran out of steps.✓ What changedWrote down what "done" has to mean, then added an independent check, captured page errors and a mouse-click rung.
- ✕ What brokeThe first verifier saw only the final page, so it rejected goals it could not see being done, and 1 of 3 broken sites was still praised.✓ What changedThe check now reads the run's factual record, never the agent's reasoning, and answers verified, contradicted or unverified.
- ✕ What brokeThe agent typed a last name into the wrong field and never noticed. Then a neatlogs trace captured an API key.✓ What changedTyped fields are read back. The key was rotated, and a redaction mask with tests was added before anything else shipped. 0 of 3 broken sites praised.
- ✕ What brokeA test site's own scripts often failed to load, and the visitor carried on with a broken page while the agent took the blame.✓ What changedThe visitor reloads, up to twice, and a page that still cannot load is reported as the site's problem, not ours.
- ✕ What brokeA button turning green leaves no words to quote, so the check could not confirm it.✓ What changedEach action records the element's state before and after. 7 of 7 test journeys completed and independently verified, in two consecutive rounds.
- ✕ What brokeA report ended the conversation. There was no way to confirm a fix.✓ What changedCheck again compares runs finding by finding. Understand mode suggests journeys. The founder brief reads the public site.
Injection test on a demo shop
One local run on the demo shop's review page, which hides instructions aimed at AI agents in three places a person cannot see: an HTML comment, white-on-white text and text moved off screen.
- The hidden text was found by code and removed from the page before any model read it.
- The agent did not claim success. It gave up, because the review form really is broken.
- The verifier, called directly on the page, did not confirm success.
- The report showed only a neutral notice that hidden text was ignored, and repeated none of its wording.
By toolFound. Faced.
Found. Faced.
Asked. Did.
neatlogs
4 entries- Found0/5 → 5/5
The lying click
The trace showed the tool reporting "clicked Add to cart" while the cart stayed empty. Fixed with a page-change check and a recovery ladder.
- Found94% → 70%
The hidden bill
Most of a run's cost was the browser library's own lookups: about 94% on Haiku 4.5. Routed through our traced client on Haiku 5.5, about 70%.
- Did0 → 7/7
An independent verifier
Goals verified went from 0 in v0 to 7 of 7 in v6, with 0 of 3 broken sites praised.
- Faced
A key in a trace, reported
A trace recorded an API key passed as a function argument. The key was rotated, traced functions stopped taking secrets, and a redaction mask with tests was added.
Entire
1 entry- Did
Every commit, with its session
Every commit carries an Entire checkpoint linking it to the coding-agent session behind it.
cfo.ai
1 entry- Did
A business plan from the measured cost
Pricing, costs and runway were planned from the cost per run measured in the traces.
Stagehand
1 entry- Found
A page could reach the browser's controls
The browser was launched in a way that let a visited page connect to its DevTools port. Now only the browser's own extension can, behind a guard with tests.
Real apps
1 entry- Found
Most apps hide behind a login
That led to the three ways in (a start link, a test login, or letting it sign up) and to a first-impression review that works even when the run stops at a login.
Also built with
- Claude
- Supabase
- Vercel
- AWS
Mistakes included
Lessons learned
- 01Mistake
A leaked API key
An API key was captured in a trace. It was rotated, and a redaction mask with tests was added before anything else shipped.
- 02Lesson
Don't trust the tool's own success message
Check the page, not the return value.
- 03Lesson
Separate our failures from yours
A report that blames the site for our crash is worse than no report.
- 04Lesson
Most apps hide behind a login
That led to the three ways in, and to stating plainly when Google or GitHub sign-in stops a run.
Daily build notes are posted on X with #neatHack.