Lazy builds: modules, fonts and data on first use#

A one-platform Wasm build (CW_WASM_PLATFORM) is about 13 MB gzipped, and most of it is not the platform: it is the applications, the engines they run on, the interpreters, the services, the internet and the fonts. A lazy build downloads the engine, one platform's shell and the light applications first, and fetches the rest the first time a world needs it:

• the heavy applications, each a Wasm module of its own, loaded the first time one of its applications is launched;
• Python, a module loaded the first time a program runs python3;
• the standard service kinds (mail, chat, wiki, shop, search …), each a part of the engine's own code split into a module of its own and linked the first time a world that declares the kind boots;
• the built-in internet's world, a data file loaded the first time a world joins it;
• the thirteen web font families, the other platforms' UI families and the CJK and emoji pack, loaded the first time a frame draws one of their glyphs;
• the script faces (Hebrew, Arabic, Thai, Devanagari, Bengali, Georgian, Armenian) and the pack's layout stubs, loaded the first time text in one of their scripts is shaped;
• the web families' metric tables, loaded the first time web text is measured;
• what only a frame draws with (DejaVu, the platform's own UI family, its icons and wallpaper, the symbol masks) and the renderer's own code, loaded by the first frame: a world driven through its scene alone never fetches them;
• reading a snapshot back and writing one out, the engine's side of the application modules, the image editors, the debugger and the shell's larger tools, each split into a code module.

CW_WASM_LAZY=1 CW_WASM_PLATFORM=ubuntu bash scripts/build-wasm.sh

pkg/node and pkg/web then hold the engine, modules/<name>.wasm, fonts/<file> (fonts/metrics/<table>.bin), data/internet.json, render.bin (what a frame draws with, one file) and the module host (modules.cjs for Node, modules.js for a page or a worker). Without CW_WASM_LAZY the build carries everything, as before.

What is in a module#

An application module is crates/bindings/app-module built with that one feature: the applications' own code and nothing else of the desktop. Every module has three exports and one import:

• cw_alloc(len) -> ptr: a buffer for a request;
• cw_call(ptr, len) -> slot: answers the request (and frees it); slot holds the reply's address and length as two little-endian u32;
• cw_free(ptr, len): frees the reply;
• the import cw.core(ptr, len) -> ptr: a question to the core while a call runs. The host passes the request to core(request), which setModuleHost gives it, and writes the answer into a buffer from cw_alloc: its length as a little-endian u32, then the bytes. The module frees it.

The first byte of a question names its channel: 0 is text (below), 1 a program's host calls (Python).

The interface#

The engine serves a module's application as NativeApp::Module, and every NativeApp method is one Call (crates/applications/src/apps/module.rs): the method's name and arguments as JSON. The module answers with the module's copy of the same method. Two rules keep a world the same world:

• The state stays in the engine. A module application's state is held as the JSON its NativeApp variant writes, and written back field by field in that order. A snapshot, a fork and stateHash see the bytes a build carrying the application sees. The module keeps a parsed copy of each state it has seen, under a number the engine gives that state; when it has not seen one (a fork, a restored snapshot, a module instantiated again) it answers miss and the engine sends the state.
• Replay loads the same code. The engine is built after its modules and pins the SHA-256 of each (CW_APP_MODULE_<NAME>). The host must answer a module by name and SHA-256 with exactly those bytes. Each step records the modules it ran in its journal entry (RecordedStep::modules), and World::modulesUsed() lists them; a replay, or a restored snapshot, refuses a build that does not serve the same modules.

A render's pixel buffers (a 3D view's picture) travel after the reply's JSON as raw bytes. A call is synchronous: the engine waits for the host, so a host has the bytes at hand when it is asked (Node reads a file; a page preloads; a worker may fetch synchronously).

Fonts on first use#

The thirteen web families (Arimo, Tinos, Cousine, Gelasio, Carlito, Caladea, Lato, Source Sans, Source Serif, Poppins, Montserrat, Playfair, JetBrains Mono; 52 files, 4.6 MB) are not in a lazy engine (cw-render's lazy-fonts). Layout never needs them: it measures with the advance and kerning tables in cw_scene, so a page lays out the same with them or without them. Drawing needs them, and a frame is never drawn with another face in place of one: the renderer asks the host for the file the moment a glyph needs it (set_font_loader) and waits; a file pinned by SHA-256 in font_pack::WEB_FACES is installed, anything else is refused. When the host cannot give it, the frame is refused with render_files_missing and the files it needed, and the next frame repaints in full.

The four platforms' UI families (Inter for macOS and iOS, Open Sans for Windows, Ubuntu, Roboto for Android; four files each) are fetched the same way in a lazy engine that does not draw their platform (font_pack::UI_FACES, pinned likewise): its own platform's family stays, the others' twelve files (about 0.2 MB gzipped) come the first time a page names them.

With a host, the CJK and emoji pack is fetched the same way instead of drawing .notdef boxes until a page installs it.

Render files on the first frame#

Nothing a world does reads what a frame draws with: layout measures with the tables in cw_scene (the advances, kern pairs and line metrics of every face) and a scene names an icon or a wallpaper by id. So a lazy engine (cw-render's lazy-render) carries none of it. The render files are DejaVu Sans, Bold, Oblique, Bold Oblique and Mono, the platform's own UI family, its icons and wallpaper, and the symbol masks; build.rs lists them with their SHA-256 and size (cw_render::RENDER_FILES; a full build embeds the bytes from the same table). The first frame that draws with one asks the host for it, a face through font(file, sha256) (fonts/<file>), an image through asset(file, sha256) (assets/<file>), and waits. Bytes that do not hash to the pin are refused, and a frame is never drawn without a file it needs: it is refused with render_files_missing.

The renderer's code is a code module too (render, below), so the first frame links it as well. An agent that drives a world through scene(), observe() and step() downloads the engine alone; one that takes a screenshot pays the render set once. lazyManifest() marks every render file and the render code module set: "render"; a page's host fetches them before its first frame (host.preload(manifest, ['render'])), as one file (below). A world's state, scene and reply hashes are the full build's whether or not it was ever drawn, and its frames are the full build's pixels.

The render bundle#

A lazy build packs the render set into one file, render.bin, and does not ship the set's files alone (scripts/pack-render.cjs, run by build-wasm.sh), so a client fetches one file before its first frame, not 210 to 216:

u32 little-endian n | n bytes of JSON { "files": [{ "path", "sha256", "offset", "bytes" }] } | the files' bytes

Each file's bytes start at its offset from the end of the JSON. path is the path the engine asks the host for (modules/render.wasm, fonts/<file>, assets/<file>) and sha256 the engine's pin. The bundle itself is not pinned: a host hashes every file it serves from it against the pin the engine asks with, as it does a single file. modules.cjs exports pack and unpack, and modules.js exports unpack.

• modules.js, preload(manifest, ['render']): one fetch of render.bin, each file checked against the manifest. A sync host fetches render.bin the first time a file it asks for is not served alone.
• modules.cjs: a file read does not find is looked for in render.bin beside the engine (or the bundle given, bytes or a function that returns them), read once.

Web metrics on first use#

The thirteen web families' advance and kern tables and DejaVu's kern pairs (only web content measures with them) are 1,150,884 bytes (226,091 gzipped). A lazy engine (cw-scene's lazy-metrics) fetches them the first time web text is measured: examples/export_tables.rs writes them, at build time, from the tables a full build embeds (each file is the core's answer to a module asking for the same table), into fonts/metrics/; build.rs pins each by SHA-256 (CW_METRICS_DIR names the folder). Text is never measured with other tables: an engine whose host cannot give one stops, as with the script faces. A desktop that opens no page fetches none.

Script faces on first use#

A lazy engine carries none of the faces text in other scripts is shaped with: the fourteen script faces (Noto Hebrew, Arabic, Thai, Devanagari, Bengali, Georgian, Armenian, regular and bold) and the layout stubs of the pack's faces (Han, Hangul, the Japanese, Korean and Traditional Chinese Han, Tamil, Gurmukhi, Lao, Khmer, Gujarati, Ethiopic, Myanmar, Sinhala): 27 files, 1,199,396 bytes (498,525 gzipped), pinned by SHA-256 in cw_scene::lazy_faces::LAZY_FACES (cw-scene's lazy-faces). The emoji stub stays: every run of text asks it whether a character starts an emoji. Layout needs these faces, so the engine asks the host for the file (fonts/<file>, the stubs under fonts/stubs/) the first time it shapes with one, or a module asks the core for it, and waits. Bytes that do not hash to the pin are refused, and text is never shaped with another face in place of one: an engine whose host cannot give a face stops, as it would without its code. A desktop of Latin text fetches none.

The host#

// Node
const engine = require('./computerworld.js');
const { createModuleHost } = require('./modules.cjs');
engine.setModuleHost(createModuleHost()); // reads modules/, fonts/ and data/ beside the engine

// A page
import init, * as engine from './computerworld.js';
import { createModuleHost } from './modules.js';
await init();
const host = createModuleHost({ base: new URL('./', import.meta.url) });
await host.preload(engine.lazyManifest(), ['kicad', 'lato-regular.ttf']);
engine.setModuleHost(host);

lazyManifest() lists every module, font and data file the engine may ask for, with its path and SHA-256. createModuleHost({ read }) takes another reader (a cache filled from a server); loads() lists what it has loaded and what each took. A host has call(name, sha256, request), font(file, sha256), data(file, sha256) and asset(file, sha256); setModuleHost gives it core(request), which its modules' cw.core import is answered through. A host keeps one compiled module per name and reuses an idle instance; a call made while another call of the same module runs (a python3 a program started) gets an instance of its own, and a request whose first byte is 0 goes to the instance whose call is running.

Text through the core#

A module draws its windows, so it measures text, and measuring needs the advance and kern tables and the script faces text is shaped with (Noto Arabic, Hebrew, Devanagari … and the pack's stubs): 1.25 MB of faces and 1.5 MB of tables, which every module used to carry again. A module carries neither (cw-scene's core-text). The first time it measures with a table or shapes with a face it asks the core on channel 0 for that one (cw_scene::tables), keeps it for the life of the instance, and measures with it locally. The core answers from the tables it measures with itself, so a module lays out every line exactly as the engine does; a test reads every table back through the wire and compares it with the engine's (tables::tests).

Python on first use#

python3 is a module too (modules/python.wasm, crates/bindings/py-module): the interpreter and its standard library, 1,898,156 bytes (577,546 gzipped). A lazy engine is built with python-module instead of python and loads it the first time a program runs. The program runs in the module; everything it does to the machine goes back to the core through cw.core on channel 1 (cw_modules::script): its files, the world's network (HTTP, DNS, TCP), child processes run by the machine's shell, the clock, entropy and the thread scheduler's seed. The engine answers each call from the host of the run that is waiting, so the run's journal records the same calls as in a build carrying the interpreter, and an interactive session resumes by replay as before.

A child process that runs python3 again is a call of its own while the parent waits: the host gives it another instance of the module. Under the debugger the module asks the core at each stop, and the engine's debugger inspects the stopped program (threads, frames, scopes, variables, evaluation, assignment) through calls back into the module's running call: a request whose first byte is 0 goes to the instance whose call is running (cw_modules::REENTER). A test runs one program, and one debugger session, both ways and compares output, files, requests and everything the debugger saw (crates/core/modules/tests/python_bridge.rs).

The first program costs the module's read and compile: 30 ms against 23 ms in the full build for the first python3 -c; later ones 0.4–0.7 ms more than the full build's.

Code modules#

The modules above are programs of their own that the engine calls. A code module is a part of the engine's own code, split out after the build and linked back into the engine's instance the first time one of its functions runs:

web also holds the web applications (cw_applications::web_app). A module is named by its crates, by a path in a crate (paths) or by a function (functions); what only its functions call follows it. The JSON code the engine's own types need stays, once.

A lazy engine is built with all of them (services), then split (scripts/split-modules.mjs, with binaryen's wasm-split --multi-split, binaryen 121 or later):

• Which functions. A function goes to a module when it belongs to that module's crates (its path, or the type it is a method of, is theirs; a service kind before the HTML engine its pages are written with), and so does every function only that module's functions call. A generic of the standard library or the JSON code instantiated for a module's type is placed by its callers alone: the compiler merges identical instantiations, so one named for a page's type can be the one the engine calls for its own. A function that belongs to two modules, an indirect target no module's crates own, and a function nothing calls directly (an export) stay in the engine; so do the few a world runs whatever services it declares (each kind's register, kind and version, asked when a registry is made), so a world fetches only the kinds it declares.
• What every world runs stays. A desktop holds a browser's state from its first boot, and hashes, snapshots and forks it. Before splitting, the build runs the unsplit engine, instrumented (wasm-split --instrument), on scripts/split-workload.cjs: the agent-desktop world drawn as the build's platform, its own applications launched (not the browser or a web application), input, the semantic channel, a snapshot, a fork and a restore. The functions of web and javascript that run keep their place in the engine (12 do), so a world that never opens a page, a web application or node never links either.
• Which data. A module's constants (its pages' stylesheets and templates, its strings and tables) move with it. The linker's map gives every data object's address and size, and its relocations (--emit-relocs) every pointer in the data and every address the code uses. An object moves to a module when that module's code is the only code that points into it, directly or through other objects. The engine's segment loses the bytes and the module gets a segment that writes them back at the same address when it is linked, before any of its functions run.
• Linking. The split engine imports a placeholder for each function it moved (cw-code.<module>). The glue (crates/bindings/wasm/js/link.js, patched into wasm-bindgen's JavaScript) answers the first call to one by asking the module host to link(name, sha256, engine): the host instantiates modules/<name>.wasm against the engine's exports (its memory, table and functions), which puts the module's functions into the table in place of the placeholders, and the glue makes the call.
• Pins. The split writes each code module's SHA-256 into the engine, in space the code-modules feature of cw-wasm reserves; the engine gives them to the host as host.code and lists them in lazyManifest() with kind: "code", and the host refuses other bytes. A code module only fits the engine it was split from.

The split moves code and data and never changes them, so a split engine runs the code a full build runs: its state, frame and reply hashes are the full build's, and a step records no module for a service. Its kinds keep their state in the kernel as any build's do.

What it costs. The 22 code modules are 1,223,654 bytes gzipped together on Ubuntu (a kind's module 21,271–122,221), against 5,206,825 when each kind was a program of its own that carried the JSON, URL and protocol code again; that code is the engine's, once. A world that joins the internet declares 19 of the kinds: its first boot fetches them and the internet's world, 1,421,081 bytes gzipped (from 4,866,795), and takes 112 ms against 86 ms in the full build (from 311 ms). A world of the company's own services fetches only the kinds it declares (the agent-desktop world: service-wiki).

The internet's world on first use#

The internet's world (worlds/internet/world.json, 1,268,250 bytes, 292,636 gzipped) is not in a lazy engine (cw-internet's on-demand). The first time a world joins the internet the engine asks the host for data/internet.json by name and SHA-256 (computed by crates/internet/build.rs from the file it was built with, so it cannot drift) and parses it once. A world that sets internet: false never asks; a world that joins it without a host is refused with internet_unavailable.

Measured#

Release Wasm with the size flags of scripts/build-wasm.sh, GNU gzip -9, on gb10-direct, 2026-10-28, at the commit that changed this table. The engine is what a client downloads first, and all a world driven through its scene ever downloads unless it uses an application, a service, a page, a program or a snapshot that needs more. render.bin is what a client that draws fetches before its first frame (every file a frame draws with and the render code module), and "first render in all" is it with the engine. Times are the first frame's of the agent-desktop world after a scene-driven session, in one process each, on the lazy and on the full build.

Before the render bundle (07296489), a first frame fetched only the render files it drew with, one by one: 15 to 41 files, 0.61 MB (iOS, Ubuntu) to 1.18 MB (macOS) gzipped. The bundle is every render file in one request, 1.31 to 1.88 MB gzipped. Before cw-encode the engines were 1,782,156 (Ubuntu) to 1,798,566 (macOS) bytes gzipped.

So a device driven through its scene downloads 1.78–1.80 MB first; its first screenshot adds 0.61–1.18 MB (2.39–2.98 MB in all, macOS the most: its wallpaper is 0.77 MB). How the lazy Ubuntu engine came down (gzip): 8,771,353 with the applications and web fonts lazy; 8,774,781 with text measured through the core (the modules shrink, the engine does not); 8,307,694 with Python a module; 7,036,852 with the services a program each and the internet's world a data file; 7,058,221 with the services code modules (the JSON, URL and protocol code they share is the engine's again, once); 4,887,082 with the HTML engine, the browser and JavaScript code modules too; 4,411,126 with the script faces and stubs fetched on first use; 4,215,094 with sqlite3's engine a code module; 4,043,370 with the other platforms' UI families fetched on first use; 2,885,882 with the render files fetched by the first frame; with render, snapshot, app-calls and debugger code modules 2,439,718, then with imaging, the web applications in web and the shell's tools 2,057,287 (these two by the split's own zlib -9, about 1.5 % above GNU gzip); 1,782,156 with the web metrics fetched on first use.

A world that joins the internet (the harness's unrelated lab with internet: true) fetches the internet's world and 19 code modules at its first boot, and web with the web metrics when it first lays out a page. On Ubuntu the first page a desktop opens costs about 15 ms more than in the full build (linking web), the first node -e 12 ms more (linking javascript), the first sqlite3 3 ms more (linking sqlite).

The application modules, Python and the data file are the same bytes for every platform:

The code modules fit only the engine they were split from, so each platform has its own; they are the same size within 0.1 %:

Fetched fonts, the same files for every platform: the script faces and stubs, 27 files, 1,199,396 bytes (498,525 gzipped); the four UI families, 16 files, 414,256 bytes (266,372 gzipped), of which an engine fetches the twelve (eight on macOS and iOS) of the families it does not draw.

A first launch is the launch step in a fresh process on Ubuntu, the median of five: the module's read from disk, compile and instantiation (3–6 ms of it), the tables and faces it asks the core for, and the launch itself. Before text went through the core the application modules were 1.24–1.87 MB gzipped and their first launches 25–35 ms. A first frame costs 2–12 ms more than the full build's. With eight module windows open, a step takes 9–18 ms against 4–7 ms: each window's render is a call. The web fonts are 52 files, 4,583,364 bytes (2.76 MB gzipped); one face is 25–159 KB.

The SHA-256s the engines are built with (lazyManifest() lists them). Every platform's engine pins the same application modules, Python and data file: