Expert procedures on the screen#

A lawyer opening a matter, an accountant closing a month: each is a precise sequence of operations that a firm already writes down. Through tool calls the sequence is a handful of writes. Done by a general agent the way the person does it, it is dozens of clicks and keystrokes, each one chosen from a screenshot. That is far less efficient, and it is what an agent has to do on software that has no API.

ComputerWorld can take such a procedure, written once against what the screen shows, and have the engine perform it through pointer and keyboard. Every episode it produces is replayable to the byte and checked by the procedure's own verifier. Seeds turn one procedure into as many distinct episodes as you want, and corrupted copies give failures the same verifier rejects.

The proposition#

Games are attractive training material because their outcomes can be checked automatically. Office work has the same property once it runs in a world whose state the evaluator can read. Moving work from tool calls onto the screen multiplies the tokens a single task produces, and every one of those tokens is still checked by the same verifier. Measured on the two procedures here, over 100 seeds each, at 1280×800 (run examples/python/expert_dataset.py --episodes 100 --negatives 1):

The action counts are measured. The token columns are a convention, not a measurement: one observation per GUI action, costing ⌈w·h/750⌉ = 1,366 tokens at 1280×800 (the approximation Anthropic documents for images), plus the actions' JSON at four bytes a token. The tool side counts only the calls' JSON, so it understates the tool cost a little and the ratio is an upper bound. The 400 episodes (200 verified demonstrations and 200 rejected corruptions) took 5.6 s and 5.7 s to generate in two runs, without frames.

A procedure#

Procedures are data. They live beside the world they run in: worlds/agent-desktop/procedures/.

{
  "id": "legal-engagement",
  "domain": "legal",
  "instruction": "Open matter {{matter}} in ~/Matters. Read its intake memo, complete the engagement letter …",
  "machine": "workstation",
  "params": [
    {"name": "client", "kind": "choice", "choices": ["Okafor Logistics Ltd", "…"]},
    {"name": "sequence", "kind": "integer", "min": 1000, "max": 9999},
    {"name": "matter", "kind": "format", "template": "2026-{{sequence}}"}
  ],
  "setup": [{"path": "/home/ada/Matters/{{matter}}/intake.txt", "content": "…"}],
  "phases": [
    {"label": "Open the matter folder", "gui": [
      {"do": "click", "target": {"interaction": "shell:launch:files"}},
      {"do": "double_click", "target": {"label": "Matters/", "app": "files"}}
    ]},
    {"label": "Name the client and the matter", "gui": [
      {"do": "click", "target": {"text": "Client:", "app": "editor"}},
      {"do": "key", "key": "End"},
      {"do": "type", "text": " {{client}}"}
    ]},
    {"label": "Save the letter",
     "gui": [{"do": "key", "key": "Ctrl+s"}],
     "tool": [{"family": "filesystem.v1", "op": "write", "payload": {"path": "…", "content": "…"}}]}
  ],
  "verify": [{"file_equals": {"path": "…", "content": "…"}}]
}

A gesture is one of:

A target is matched on whatever it gives: accessible role, exact label, painted text containing a string, an interaction id with * wildcards, and the app of the owning window. The topmost matching node that a click actually reaches wins. The click point comes from Scene::reachable_point, so a control half hidden behind another window is clicked where it shows. A text target need not be interactive itself: it only has to lie over something in its own window that is, which is how "click on the line that reads Retainer:" works.

Write targets the way a person would find the thing: by its label, not its window id, which changes from run to run.

The calls#

The same four calls in Rust, Python and JavaScript. All of them belong to the owner: the actor's Environment has none of them.

options is {width, height, mode, typing}. The defaults are 1280, 800, "gui" and "text". mode: "tool" runs the phases' tool calls instead of the gestures. typing: "keys" sends one action per character, the way a person types, instead of one per run of text, the way computer-use agents send it.

project returns a demonstration, and leaves the world in the state the demonstration ends in, so fork first if you need the start again. A demonstration contains:

• steps: each step has its phase label, its role and session, the action sent, the target it resolved to (with the app of the window it landed in) and the point it clicked, the digest of the scene the action was chosen from, and whether the action succeeded;
• completed, and a failure that names the phase and the target nothing on screen matched;
• verification, from the procedure's checks;
• captures, the values the phases bound by name (absent when there are none). A check that names one fails under verify without them; verify_with takes them;
• state_hash;
• counts.

The steps are ordinary actions. Sending them to a freshly staged world reaches the same state_hash on every platform, so the frames do not need storing: re-render them by replaying.

A session needs application.v1 as well as pointer.v1 and keyboard.v1. Without it, the dock's launchers are not controls at all. Web work needs browser.v1 too. The tests, the dataset generator and debug.py grant those plus filesystem.v1. To keep an agent on the screen, accept only pointer and keyboard actions from it, as expert_env.py does.

Writing one#

Look at the machine rather than guessing. scripts/procedures/debug.py runs one procedure for one seed and prints every step with the control it hit. When a gesture finds nothing, or a check fails, it prints the acting role's windows and controls (and with --text the painted text), and saves the screen with --png. --upto k stops after phase k, which is how a procedure is written one phase at a time. Adding a JSON file to any worlds/*/procedures/ folder puts it under crates/computerworld/tests/procedures.rs.

worlds/pro-workstation is one Ubuntu workstation with the professional applications installed: FreeCAD, KiCad, DB Browser for SQLite, LibreOffice Writer and Calc, GIMP, Pinta, Kdenlive, Rhythmbox, Maps and Mail. Team procedures run in company-2026, whose three desktops share Slack, GitHub, mail, calendar and documents.

Shared worlds#

A procedure that names records by the key a freshly booted world would give them (Gmail's mail-4, pull request 21, the shop's order 5001, the second ticket in a project) works alone and breaks the moment another procedure, or a world with history, has taken that key first. Two procedures in one world both open "OPS-2"; the second clicks and checks the first one's ticket. So a procedure names what it makes by capturing it, and checks records by what they hold.

Captures. A phase may end with capture: once its actions are done, in either mode, each entry binds a name to one value read from the world. Later gestures, tool calls, captures and checks use it as {{@name}}, or {{#@name}} for a JSON number. Parameters are substituted when the instance is drawn; captured placeholders wait for the run.

{"label": "Open the pull request on GitHub", "role": "engineer", "gui": ["…"], "tool": ["…"],
 "capture": [{"name": "pr", "last": true, "from": {"state": {
   "pointer": "/state/services/github/state/repositories/atlas/pull_requests",
   "where": {"/head": "refs/heads/{{branch}}"}}}}]}

and then "text": "github.com/northstar/atlas/pull/{{@pr}}" in the reviewer's click, "url": "…/pull/{{@pr}}/files" in the tool call, and "pointer": "…/pull_requests/{{@pr}}/state" in the check.

A pattern picks the value out of a candidate's text: {{?}} is the value (an identifier: letters, digits, -, _ and a . between them) and {{*}} any run of characters, matched anywhere in the text: "Order {{?}} placed". A capture whose source holds nothing that matches stops the procedure at that phase, with the reason.

A where maps a pointer inside each element to what it must hold. A string matches as a pattern with {{*}} for any run ("/body": "Hi Lin, PO HR-PB{{*}}"), a number or an object by value, and an array field matches when any of its items does ("/to": "[email protected]"). Capture by identity (who made it, where, the newest) and leave content to the checks, so the checks still test something.

Find. {"find": {"pointer": p, "where": {…}, "count": n}} passes when some element of the collection at p holds where, or with count when exactly n do: the Slack message with this author and text, once; no second order for this buyer and total. It replaces a check on "the next key is absent", which only holds in a fresh world.

What else composes. A check on something only this procedure moves (its own record's fields) stays exact; one on a shared total (an account's balance, a hold) is replaced by the transaction that moved it. A software change checks the files it writes, not the whole tree, and its engineer switches to main and pulls before branching, because their clone from the last change is still on its old branch. A message that names a record is sent after the record exists (the incident commander opens the ticket before taking command). The instruction never quotes a captured value, which nobody knows in advance: it says "the pull request's link" or "the invoice number".

Measured: Halyard's rover-launch society runs its nine work items in one world and all nine verify (735 GUI actions, 110 tool calls), where seven did when two of them named mail-4 and PR #8. The test procedures_compose_in_one_shared_world runs ops-purchase-requisition, ops-incident-response, ops-meeting-request, ops-vendor-invoice and two software changes back to back in one company-2026 (436 GUI actions): each verifies, and all six still verify at the end. The six finance procedures likewise run back to back in one robotics-co world, and the three media procedures in one lantern-media world.

What does not compose yet is a procedure whose first step creates something a person has only once: Carol opening a city account for a second permit, Alice activating Stripe for a second venture. Those stop at their first phase in a world where it already exists, because a procedure has no "if it is not there yet".

Datasets and environments#

examples/python/expert_dataset.py writes one JSON line per episode to episodes.jsonl. Each line holds the instruction, every step with its phase label and target, the verdict, the end state hash, and the GUI and tool counts. With --frames it also saves the PNG each action was chosen from, re-rendered by replay and checked against the hash. --negatives n adds n corrupted episodes per seed: a phase that writes state is skipped, or a value the expert types is replaced by another draw's while the setup and the checks stay the original's. Every corrupted episode must be rejected, or the script stops. summary.json has the totals above.

scripts/procedures/swarm.py turns a dataset into one page showing the whole corpus at once. Each verified episode is a dot measured along nine dimensions: GUI actions, actions per tool call, keyboard share, characters typed, characters per keystroke action, distinct controls, applications, phases and people. Pick any two. Dots are coloured by family of work (engineering, analysis and office, media), each domain gets its own panel against the swarm, and there is a table of medians. Across many domains this shows what professional work is like on a screen, beyond any one procedure.

examples/python/expert_env.py is the same procedure as a reinforcement-learning task. reset(seed) gives the instruction and the screen, step(action) takes pointer and keyboard actions, and {"done": true} ends the episode with the verifier's reward. Run as a script, it checks that the expert's recorded actions earn 1.0 and that sixty random clicks and keys earn 0.0.

examples/javascript/expert-procedures.mjs projects the same procedures from Node. expert_dataset.py --compare checks that both runtimes end every demonstration on the same state hash with the same number of actions, and scripts/smoke-bindings.sh runs that pair.

Procedures are also one way a simulated person acts. In a society, a person whose policy is procedure performs a fresh instance of one each time they are active, beside people played by a language model through the same grounded gestures (World.perform).

The procedures here#

Calc rewrites a CSV it saves: dates come back as 4/3/2026 and lines end in \r\n. That is why the reconciliation's checks name each transaction as *,vendor,category,amount with file_has_line rather than as text in the file.

crates/computerworld/tests/procedures.rs holds every procedure in the folder to four properties:

• it verifies on screen and through tools for three seeds;
• it does not verify before anything is done;
• leaving out any phase that writes state fails verification;
• the recorded actions alone replay to the same hash.

A new procedure is covered by dropping it into the folder.