Computaco

Simulated machines only

Compiled rules completed 96% of 360 simulated application tasks with no model call; the model agents completed 96% and 87%

Can rules compiled from the applications' own code operate them as well as a model agent that reads the screen? On 360 runs of four tasks in Mail, Messages and Docs across five simulated operating systems, under six perturbations, the rules completed 96%, Computatrum 1.1 with DeepSeek 96%, Gemini 3.6 Flash in one call 87% of the 257 runs a model answered. The rules made no model call and decided a step in 480 µs; the model agents took 2.6 s and 2.8 s. The study's model calls, reruns included, cost $9.24 at the meter's rates.

96%tasks the rules completed (345 of 360; 95% 93%–97%)
96%Computatrum 1.1 with DeepSeek (344 of 360; 95% 93%–97%)
87%Gemini 3.6 Flash in one call (224 of 257 answered; 95% 83%–91%)
480 µsthe rules' median decision, on a CPU (1518 decisions)

The question

The compiled harness decides each step of a task from rules learned on the simulator: what every control does, lifted across themes and data, matched to the screen by an energy, and checked after every step against what the rule predicted. It asks a model only when the rules cannot tell. Model agents read the screen and ask a model at every step. This study puts both on the same machines, with the same goals and the same success check, and counts success, model calls, dollars and time.

Method

Tasks. Four procedures (compose and send a mail, open and archive one, reply in Messages, type into a document and save it), each on the five simulated systems where it runs: 20 procedure and system pairs, seeds 1 to 3. Each run starts from the same world with the application launched. The check is the procedure's own: a query over the mail, messages or docs service's records, run on the world the arm left, whatever the arm says about itself.

Perturbations. relabel, reorder and duplicate change what an arm reads, never the world: the matcher's UX graph (perturb::view), and the model's outline (labels, places and an extra Send/Save), whose clicks reach the control the outline showed there; the screenshot a model reads is not perturbed. insert (one more record in the service) and move focus (after the first action, a click into another of the application's text fields) change the world for every arm. the compiled harness's artifacts (cloud/compiled/artifacts.json, rules lifted from every theme) were lifted from walks of the unperturbed explorer world and a world with one more row per list (more_rows); none of the perturbations, insert's records included, was seen while lifting. Every perturbation is held out. The goals are the benchmark's; the rules were not lifted from the benchmark's procedures.

Arms. Every arm gets the same goal: the rules as the procedure's concept and bindings, the models as its instruction in words.

Measures. Success with a Wilson 95% interval; paired differences between arms over the same runs, with a 95% bootstrap interval over the 20 cases and an exact McNemar test. Dollars are the meter's rates over each call's reported tokens. The preregistration (preregistration.json) was committed before any model arm ran.

Results

The rules completed 96% of the runs; Computatrum 1.1 with DeepSeek 96%, Gemini 3.6 Flash in one call 87%

Figure 1. Share of runs whose checks passed, by arm, with 95% intervals.

The preregistered contrasts, paired run by run (same case, seed and perturbation): the difference in success, its 95% interval over the 20 cases, the runs only one arm completed, and the exact McNemar test.

ContrastDifference95% intervalOnly the firstOnly the secondp
Compiled rules minus Computatrum 1.1 with DeepSeek+0 pp−3 pp to +4 pp16151.000
Compiled rules minus Gemini 3.6 Flash in one call+8 pp±0 pp to +19 pp276< 0.001
Rules, model on ask minus Compiled rules±0 pp±0 pp to ±0 pp001.000
Rules, model on ask minus Computatrum 1.1 with DeepSeek+0 pp−3 pp to +4 pp16151.000
Rules, model on ask minus Gemini 3.6 Flash in one call+8 pp±0 pp to +19 pp276< 0.001

Computatrum 1.1 with DeepSeek on the harness of the study's first run: 268 of 360 (74%). Most of its failures then were the harness's, not the model's: a screen read that no model answered ended the run. The figures above are a rerun of the whole arm on the harness that asks such a call again and goes on until the task is done or fails. The rerun also ran on a simulator in which a second Save of an unchanged document writes nothing, as Pages and Word do (the first run's made a revision each time, and the check asks for one). Sent again to the first run's simulator, 59 of its 344 successes fail, all document runs: 285 of 360 on that simulator. The other model arm and the fallback arm ran on the first run's simulator. The other way round, 50 of the first run's recorded failures (30 Computatrum 1.1 with DeepSeek, 15 Gemini 3.6 Flash in one call, 5 Rules, model on ask) end as successes when their traces are sent to the rerun's simulator, so the first run's traces replay to their recorded verdicts only on the simulator they ran on. Its dollars now count its screen reads; the first run's did not, as the platform did not name the model that read them.

Gemini 3.6 Flash in one call as run: 143 of 360 (40%). Partway through the study the accounts this harness's model is reached through stopped serving it, and 208 of its runs ended before any model saw the screen. Reruns reached a model in 105 of them, the last one run of each task, system and perturbation that had lost runs. Its figures on this page leave out the 103 runs no model answered, so they still lean to the perturbations that ran first (the preregistration's amendments say what happened and when).

The rules lost most under relabel (45 of 60)

ArmNoneRelabelReorderInsertDuplicateMove focus
Compiled rules60/6045/6060/6060/6060/6060/60
Rules, Ubuntu and Android held out57/6045/6057/6057/6057/6057/60
Rules, model on ask60/6045/6060/6060/6060/6060/60
Computatrum 1.1 with DeepSeek60/6060/6057/6060/6052/6055/60
Gemini 3.6 Flash in one call51/6051/5740/4722/2623/2837/39

share of runs whose checks passeddeeper = larger

Move focus applied where an application draws two text fields after the first step: in 15 of the rules' 60 runs.

The rules did the same on every system (69 of 72 each); learned without Ubuntu and Android, Ubuntu fell to 54

ArmmacOSWindowsUbuntuiOSAndroid
Compiled rules69/7269/7269/7269/7269/72
Rules, Ubuntu and Android held out69/7269/7254/7269/7269/72
Rules, model on ask69/7269/7269/7269/7269/72
Computatrum 1.1 with DeepSeek70/7272/7269/7268/7265/72
Gemini 3.6 Flash in one call48/5144/4950/5342/5340/51

share of runs whose checks passeddeeper = larger

ArmDocs: type and saveDocs: type and save (Start menu)Mail: compose and sendMail: open and archiveMessages: replyMessages: reply (Start menu)
Compiled rules60/7215/1890/9090/9054/5436/36
Rules, Ubuntu and Android held out45/7215/1890/9090/9054/5436/36
Rules, model on ask60/7215/1890/9090/9054/5436/36
Computatrum 1.1 with DeepSeek63/7218/1883/9090/9054/5436/36
Gemini 3.6 Flash in one call17/455/1068/6870/7039/3925/25

The rules decided in 480 µs with no model call; the cheapest model agent took 2.6 s a decision and $0.0065 a run

ArmModel calls a runTokens a runDollars a runDollars a successMedian decisionSteps a successMedian run
Compiled rules0.000$0$0480 µs4.20.8 s
Rules, Ubuntu and Android held out0.000$0$0468 µs4.10.1 s
Rules, model on ask0.20978$0.0001$0.0001549 µs (model 2.8 s)4.21.9 s
Computatrum 1.1 with DeepSeek14.3663,623$0.0065$0.00682.6 s7.223.1 s
Gemini 3.6 Flash in one call4.8322,076$0.019$0.0212.8 s4.615.1 s

Spend at the meter's rates. Computatrum 1.1 with DeepSeek: $2.33 over 360 runs (20,835,514 input tokens, 7,237,075 of them cached, 2,068,742 output; models deepseek-flash, jev-1.14.0); Gemini 3.6 Flash in one call: $4.79 over 257 runs (5,462,602 input tokens, 140,148 of them cached, 210,915 output; models gemini-3.6-flash, google/gemini-3.6-flash); Rules, model on ask: $0.051 over 360 runs (318,599 input tokens, 99,968 of them cached, 33,513 output; models deepseek-flash, jev-1.14.0).

What the runs look like

Figure 2. The compiled rules send a mail on the simulated Mac: six decisions, about half a millisecond each, no model call. The ring marks each click.
Figure 3. The same goal for Computatrum 1.1 with DeepSeek, from the same world: it sends the mail, then opens Sent to check it.
Figure 4. Where the rules stop: they read Pages' Save under another name (the relabel is in what they read; the screen is the real one), so after the line is typed they no longer see the unsaved changes their rule predicted, and the run ends before Save.
Figure 5. Computatrum 1.1 on the document task: it types the line, presses Save again and again, and ends when its planner returns nothing twice.

Every failure of the rules was one kind: a step's effect differed from its rule (15 runs)

ArmFailedA step's effect differed from its ruleChose the duplicate controlThe model returned nothing twiceSaid it was done; the checks failed
Compiled rules1515000
Rules, Ubuntu and Android held out3030000
Rules, model on ask1500141
Computatrum 1.1 with DeepSeek160808
Gemini 3.6 Flash in one call3303030

The document task's check asks for the line at the end and exactly one save. Sent again to a fresh world, the failed document runs left this: Compiled rules: 15 failed, 15 never saved, 0 saved more than once; Rules, Ubuntu and Android held out: 30 failed, 30 never saved, 0 saved more than once; Rules, model on ask: 15 failed, 10 never saved, 5 saved more than once; Computatrum 1.1 with DeepSeek: 9 failed, 8 never saved, 1 saved more than once; Gemini 3.6 Flash in one call: 33 failed, 6 never saved, 27 saved more than once. Every rules failure stopped before Save; the model agents' document failures either never saved or saved again and again, each press a new revision.

The shipped rules never asked: every one of their stops was a step whose effect differed from its rule, which the choose route does not put to a model. So the fallback arm put 0 choices to the decide chain and handed 15 runs, all of them Docs under relabel, to the model harness, which completed 0 of them.

Limitations

Reproduce it

Run, analysis and page are in research/studies/harness-study/. The rules arms replay offline; the model arms' traces (every action sent and every decision) are kept, and run.mjs --replay sends them to a fresh world and checks the verdict and the final state hash.

Commands
cargo build --release -p cw-explorer --example study
cd research/studies/harness-study
node run.mjs --arms rules,rules+model,computatrum-1.1-normal,gemini-flash-1.1-normal --out results/2026-11-21
node run.mjs --arms rules --artifacts artifacts-lifted.json --name rules=rules-heldout --out results/2026-11-21-heldout
node run.mjs --arms gemini-flash-1.1-normal --ids results/2026-11-21-rerun.ids --out results/2026-11-21-rerun
node run.mjs --arms gemini-flash-1.1-normal --ids results/2026-11-21-rerun3.ids --out results/2026-11-21-rerun3 --concurrency 1
node run.mjs --out results/2026-11-21 --replay
python3 analyze.py results/2026-11-21 results/2026-11-21-heldout results/2026-11-21-rerun results/2026-11-21-rerun2 \
    results/2026-11-21-rerun3 --exclude-refused --out results/main/summary.json
python3 build_page.py

2026-11-21T08:19:26.279Z: rules, rules+model, computatrum-1.1-normal, gemini-flash-1.1-normal, engine 0.3.0

2026-11-21T08:24:37.342Z: rules, engine 0.3.0

2026-11-21T08:48:56.910Z: gemini-flash-1.1-normal, engine 0.3.0

2026-11-21T08:59:46.453Z: gemini-flash-1.1-normal, engine 0.3.0

2026-11-21T12:35:37.284Z: gemini-flash-1.1-normal, engine 0.3.0

2026-11-21T12:56:01.028Z: computatrum-1.1-normal, engine 0.3.0