Simulated machines only
Compiled rules completed 96% of 360 simulated application tasks with no model call; the model agents completed 96% and 87%
Can rules compiled from the applications' own code operate them as well as a model agent that reads the screen? On 360 runs of four tasks in Mail, Messages and Docs across five simulated operating systems, under six perturbations, the rules completed 96%, Computatrum 1.1 with DeepSeek 96%, Gemini 3.6 Flash in one call 87% of the 257 runs a model answered. The rules made no model call and decided a step in 480 µs; the model agents took 2.6 s and 2.8 s. The study's model calls, reruns included, cost $9.24 at the meter's rates.
The question
The compiled harness decides each step of a task from rules learned on the simulator: what every control does, lifted across themes and data, matched to the screen by an energy, and checked after every step against what the rule predicted. It asks a model only when the rules cannot tell. Model agents read the screen and ask a model at every step. This study puts both on the same machines, with the same goals and the same success check, and counts success, model calls, dollars and time.
Method
Tasks. Four procedures (compose and send a mail, open and archive one, reply in Messages, type into a document and save it), each on the five simulated systems where it runs: 20 procedure and system pairs, seeds 1 to 3. Each run starts from the same world with the application launched. The check is the procedure's own: a query over the mail, messages or docs service's records, run on the world the arm left, whatever the arm says about itself.
Perturbations. relabel, reorder and duplicate change what an arm reads, never the world: the matcher's UX graph (perturb::view), and the model's outline (labels, places and an extra Send/Save), whose clicks reach the control the outline showed there; the screenshot a model reads is not perturbed. insert (one more record in the service) and move focus (after the first action, a click into another of the application's text fields) change the world for every arm. the compiled harness's artifacts (cloud/compiled/artifacts.json, rules lifted from every theme) were lifted from walks of the unperturbed explorer world and a world with one more row per list (more_rows); none of the perturbations, insert's records included, was seen while lifting. Every perturbation is held out. The goals are the benchmark's; the rules were not lifted from the benchmark's procedures.
Arms. Every arm gets the same goal: the rules as the procedure's concept and bindings, the models as its instruction in words.
- Compiled rules. the compiled harness, rules only: the procedure's concept and bind as the goal, the matcher's decision per screen, each step verified against its prediction; an ask, a differing effect, a refused action or a chosen duplicate ends the run.
- Rules, model on ask. the same, with the choose route's fallback (cloud/src/compiled.ts): an ask with two or more candidates is put to the decide chain (POST /v1/agent/decide, the route's own question) and the matcher grounds the pick; every other stop of the rules hands the rest of the task to computatrum-1.1-normal with the procedure's instruction.
- Computatrum 1.1 with DeepSeek. the catalog's default model harness (DeepSeek's flash chat model plans, GLM 5.3 Flash describes thin screens, Jev picks routine clicks), in Computatrum's own loop over the machine as a hosted agent sees it, the procedure's instruction as its task, nobody to ask, confirmations allowed.
- Gemini 3.6 Flash in one call. the catalog's one-call model harness (Gemini 3.6 Flash reads the screenshot and the outline and decides in one call), in the same loop.
Measures. Success with a Wilson 95% interval; paired differences between arms over the same runs, with a 95% bootstrap interval over the 20 cases and an exact McNemar test. Dollars are the meter's rates over each call's reported tokens. The preregistration (preregistration.json) was committed before any model arm ran.
Results
The rules completed 96% of the runs; Computatrum 1.1 with DeepSeek 96%, Gemini 3.6 Flash in one call 87%
The preregistered contrasts, paired run by run (same case, seed and perturbation): the difference in success, its 95% interval over the 20 cases, the runs only one arm completed, and the exact McNemar test.
| Contrast | Difference | 95% interval | Only the first | Only the second | p |
|---|---|---|---|---|---|
| Compiled rules minus Computatrum 1.1 with DeepSeek | +0 pp | −3 pp to +4 pp | 16 | 15 | 1.000 |
| Compiled rules minus Gemini 3.6 Flash in one call | +8 pp | ±0 pp to +19 pp | 27 | 6 | < 0.001 |
| Rules, model on ask minus Compiled rules | ±0 pp | ±0 pp to ±0 pp | 0 | 0 | 1.000 |
| Rules, model on ask minus Computatrum 1.1 with DeepSeek | +0 pp | −3 pp to +4 pp | 16 | 15 | 1.000 |
| Rules, model on ask minus Gemini 3.6 Flash in one call | +8 pp | ±0 pp to +19 pp | 27 | 6 | < 0.001 |
Computatrum 1.1 with DeepSeek on the harness of the study's first run: 268 of 360 (74%). Most of its failures then were the harness's, not the model's: a screen read that no model answered ended the run. The figures above are a rerun of the whole arm on the harness that asks such a call again and goes on until the task is done or fails. The rerun also ran on a simulator in which a second Save of an unchanged document writes nothing, as Pages and Word do (the first run's made a revision each time, and the check asks for one). Sent again to the first run's simulator, 59 of its 344 successes fail, all document runs: 285 of 360 on that simulator. The other model arm and the fallback arm ran on the first run's simulator. The other way round, 50 of the first run's recorded failures (30 Computatrum 1.1 with DeepSeek, 15 Gemini 3.6 Flash in one call, 5 Rules, model on ask) end as successes when their traces are sent to the rerun's simulator, so the first run's traces replay to their recorded verdicts only on the simulator they ran on. Its dollars now count its screen reads; the first run's did not, as the platform did not name the model that read them.
Gemini 3.6 Flash in one call as run: 143 of 360 (40%). Partway through the study the accounts this harness's model is reached through stopped serving it, and 208 of its runs ended before any model saw the screen. Reruns reached a model in 105 of them, the last one run of each task, system and perturbation that had lost runs. Its figures on this page leave out the 103 runs no model answered, so they still lean to the perturbations that ran first (the preregistration's amendments say what happened and when).
The rules lost most under relabel (45 of 60)
| Arm | None | Relabel | Reorder | Insert | Duplicate | Move focus |
|---|---|---|---|---|---|---|
| Compiled rules | 60/60 | 45/60 | 60/60 | 60/60 | 60/60 | 60/60 |
| Rules, Ubuntu and Android held out | 57/60 | 45/60 | 57/60 | 57/60 | 57/60 | 57/60 |
| Rules, model on ask | 60/60 | 45/60 | 60/60 | 60/60 | 60/60 | 60/60 |
| Computatrum 1.1 with DeepSeek | 60/60 | 60/60 | 57/60 | 60/60 | 52/60 | 55/60 |
| Gemini 3.6 Flash in one call | 51/60 | 51/57 | 40/47 | 22/26 | 23/28 | 37/39 |
share of runs whose checks passeddeeper = larger
Move focus applied where an application draws two text fields after the first step: in 15 of the rules' 60 runs.
The rules did the same on every system (69 of 72 each); learned without Ubuntu and Android, Ubuntu fell to 54
| Arm | macOS | Windows | Ubuntu | iOS | Android |
|---|---|---|---|---|---|
| Compiled rules | 69/72 | 69/72 | 69/72 | 69/72 | 69/72 |
| Rules, Ubuntu and Android held out | 69/72 | 69/72 | 54/72 | 69/72 | 69/72 |
| Rules, model on ask | 69/72 | 69/72 | 69/72 | 69/72 | 69/72 |
| Computatrum 1.1 with DeepSeek | 70/72 | 72/72 | 69/72 | 68/72 | 65/72 |
| Gemini 3.6 Flash in one call | 48/51 | 44/49 | 50/53 | 42/53 | 40/51 |
share of runs whose checks passeddeeper = larger
| Arm | Docs: type and save | Docs: type and save (Start menu) | Mail: compose and send | Mail: open and archive | Messages: reply | Messages: reply (Start menu) |
|---|---|---|---|---|---|---|
| Compiled rules | 60/72 | 15/18 | 90/90 | 90/90 | 54/54 | 36/36 |
| Rules, Ubuntu and Android held out | 45/72 | 15/18 | 90/90 | 90/90 | 54/54 | 36/36 |
| Rules, model on ask | 60/72 | 15/18 | 90/90 | 90/90 | 54/54 | 36/36 |
| Computatrum 1.1 with DeepSeek | 63/72 | 18/18 | 83/90 | 90/90 | 54/54 | 36/36 |
| Gemini 3.6 Flash in one call | 17/45 | 5/10 | 68/68 | 70/70 | 39/39 | 25/25 |
The rules decided in 480 µs with no model call; the cheapest model agent took 2.6 s a decision and $0.0065 a run
| Arm | Model calls a run | Tokens a run | Dollars a run | Dollars a success | Median decision | Steps a success | Median run |
|---|---|---|---|---|---|---|---|
| Compiled rules | 0.00 | 0 | $0 | $0 | 480 µs | 4.2 | 0.8 s |
| Rules, Ubuntu and Android held out | 0.00 | 0 | $0 | $0 | 468 µs | 4.1 | 0.1 s |
| Rules, model on ask | 0.20 | 978 | $0.0001 | $0.0001 | 549 µs (model 2.8 s) | 4.2 | 1.9 s |
| Computatrum 1.1 with DeepSeek | 14.36 | 63,623 | $0.0065 | $0.0068 | 2.6 s | 7.2 | 23.1 s |
| Gemini 3.6 Flash in one call | 4.83 | 22,076 | $0.019 | $0.021 | 2.8 s | 4.6 | 15.1 s |
Spend at the meter's rates. Computatrum 1.1 with DeepSeek: $2.33 over 360 runs (20,835,514 input tokens, 7,237,075 of them cached, 2,068,742 output; models deepseek-flash, jev-1.14.0); Gemini 3.6 Flash in one call: $4.79 over 257 runs (5,462,602 input tokens, 140,148 of them cached, 210,915 output; models gemini-3.6-flash, google/gemini-3.6-flash); Rules, model on ask: $0.051 over 360 runs (318,599 input tokens, 99,968 of them cached, 33,513 output; models deepseek-flash, jev-1.14.0).
What the runs look like
Every failure of the rules was one kind: a step's effect differed from its rule (15 runs)
| Arm | Failed | A step's effect differed from its rule | Chose the duplicate control | The model returned nothing twice | Said it was done; the checks failed |
|---|---|---|---|---|---|
| Compiled rules | 15 | 15 | 0 | 0 | 0 |
| Rules, Ubuntu and Android held out | 30 | 30 | 0 | 0 | 0 |
| Rules, model on ask | 15 | 0 | 0 | 14 | 1 |
| Computatrum 1.1 with DeepSeek | 16 | 0 | 8 | 0 | 8 |
| Gemini 3.6 Flash in one call | 33 | 0 | 3 | 0 | 30 |
The document task's check asks for the line at the end and exactly one save. Sent again to a fresh world, the failed document runs left this: Compiled rules: 15 failed, 15 never saved, 0 saved more than once; Rules, Ubuntu and Android held out: 30 failed, 30 never saved, 0 saved more than once; Rules, model on ask: 15 failed, 10 never saved, 5 saved more than once; Computatrum 1.1 with DeepSeek: 9 failed, 8 never saved, 1 saved more than once; Gemini 3.6 Flash in one call: 33 failed, 6 never saved, 27 saved more than once. Every rules failure stopped before Save; the model agents' document failures either never saved or saved again and again, each press a new revision.
The shipped rules never asked: every one of their stops was a step whose effect differed from its rule, which the choose route does not put to a model. So the fallback arm put 0 choices to the decide chain and handed 15 runs, all of them Docs under relabel, to the model harness, which completed 0 of them.
Limitations
- Simulator only. Every machine is ComputerWorld's simulation of an operating system and its applications. No real operating system ran. The rules were learned on the same simulator, so this measures them where they were made.
- Three applications, four tasks. The rules exist for Mail, Messages and Docs only; the model agents can operate anything, and these tasks do not show that.
- Screen perturbations reach the model agents through the outline only. A relabel, a reorder and a duplicate change the controls' names, places and number in what the agent reads, and its clicks reach the control it saw there; the screenshot is the real one.
- Representation gaps the rules still have. A control's region is compared by its exact path of landmarks, so a wrapper added above it costs the most; icons are known only by their accessible names; label similarity is spelling, and a state such as unsaved changes is read from a label (Pages' "Save •"); the goals arrive structured, with no words to understand; the groundings are keyed by the theme's name. None of these was varied here beyond the six perturbations.
- One sample of each model. Each model arm ran once per case, seed and perturbation, at the catalog's settings; other prompts or models may do better or worse.
Reproduce it
Run, analysis and page are in research/studies/harness-study/. The rules arms replay offline; the model arms' traces (every action sent and every decision) are kept, and run.mjs --replay sends them to a fresh world and checks the verdict and the final state hash.
Commands
cargo build --release -p cw-explorer --example study
cd research/studies/harness-study
node run.mjs --arms rules,rules+model,computatrum-1.1-normal,gemini-flash-1.1-normal --out results/2026-11-21
node run.mjs --arms rules --artifacts artifacts-lifted.json --name rules=rules-heldout --out results/2026-11-21-heldout
node run.mjs --arms gemini-flash-1.1-normal --ids results/2026-11-21-rerun.ids --out results/2026-11-21-rerun
node run.mjs --arms gemini-flash-1.1-normal --ids results/2026-11-21-rerun3.ids --out results/2026-11-21-rerun3 --concurrency 1
node run.mjs --out results/2026-11-21 --replay
python3 analyze.py results/2026-11-21 results/2026-11-21-heldout results/2026-11-21-rerun results/2026-11-21-rerun2 \
results/2026-11-21-rerun3 --exclude-refused --out results/main/summary.json
python3 build_page.py2026-11-21T08:19:26.279Z: rules, rules+model, computatrum-1.1-normal, gemini-flash-1.1-normal, engine 0.3.0
2026-11-21T08:24:37.342Z: rules, engine 0.3.0
2026-11-21T08:48:56.910Z: gemini-flash-1.1-normal, engine 0.3.0
2026-11-21T08:59:46.453Z: gemini-flash-1.1-normal, engine 0.3.0
2026-11-21T12:35:37.284Z: gemini-flash-1.1-normal, engine 0.3.0
2026-11-21T12:56:01.028Z: computatrum-1.1-normal, engine 0.3.0