Four LibreOffice Calc tasks, run twelve times end to end by a computer-use agent on a real Linux desktop. It ended ten of those runs itself, reporting the work done.
Six of those ten files were wrong.
Nothing on this page is scored from what the agent wrote.
Every verdict comes from reopening the saved .xlsx and comparing it,
cell by cell, against fixed values.
Twelve runs, two verdicts each.
loading runs…
Each image below is the exact screenshot the model received before it chose where to click. The cross is where the click landed. Follow it left to the row numbers and you are reading the same evidence the verifier read. Step through the run with the bar under the image — every frame is one mark on it.
A row is twenty pixels tall.
−0.11rows of mean error, over 26 cell clicks
There is no systematic offset. The model is not aiming high or low, and a constant calibration fix would achieve nothing. What it has is spread — 0.53 rows of it, against a row twenty pixels tall. Eight of the 26 landed on the wrong row.
scripts/click_accuracy.py measures Calc's row geometry from a run's own
screenshot — it finds the grid lines rather than assuming a pitch — and compares every click that names a
cell against the band that cell actually occupies.
Row 3, the one sitting just under the header, was hit every single time. Every miss is on the three rows further down: the header is the only fixed visual anchor on the sheet, and accuracy decays with distance from it.
The lever this suggests is not better calibration but removing the need to aim — address cells by name, not by pixel. Calc offers exactly that in the Name Box, and section 04 is what happened when the agent was pointed at it.
Each failing scenario was re-run with guidance added to the task, aimed at the mechanism rather than at the mistake. Both re-runs passed. Stopping there would have been the error this whole project is about, so each was repeated twice more.
highlight was told to substitute one action
for another: use this dialog rather than that toolbar. reconciliation was told to
adopt a discipline: check an indicator before every keystroke. The substitution was
followed every time and worked. The discipline was ignored by two runs of three, and the run that did
follow it — reaching for the Name Box three times — did worst of all, failing after 40 steps.
These six runs are in the list on the right and in none of the figures anywhere else on this page.
Almost none of it is the desktop.
Capturing a 1440×900 screen costs about a tenth of a second and a click about one second. Anyone optimising the screenshot pipeline of a desktop agent is optimising the wrong thing; the lever is the model call.
Three lines of post-condition.
Every false success on this page would have been caught by a check that runs after the agent stops and reports its verdict separately from the agent's narrative. Nothing in the tooling encourages writing one, and scoring a run by reading its report uses the single source that is systematically wrong in precisely the cases that matter.
holo run --check ./check.py "fill the missing amounts" # agent says: completed successfully # check says: FAILED (3/4) - row 8 was overwritten # exit code: 1
The disagreement between those two lines is the most valuable number a customer can have, and today nothing surfaces it. It also turns any demo into a regression test.
The scoring in this repository is not clever. It is one function, applied to the file the agent left behind:
got = as_number(ws.cell(row=row, column=AMOUNT_COL).value)
verdict.add(f"amount_row_{row}", close_to(got, gold), ...)
Two of the four verifiers exist to catch an answer that looks
right in a screenshot: a SUM cell holding a number the model added up in its head rather
than a formula, and a sort that moved one column and detached every amount from its row.