The brain encoder#

A research program, granted by agreement: predicted cortical responses to a stimulus, from Meta AI's TRIBE v2 (facebook/tribev2, CC BY-NC 4.0; the paper), beside the environments the stimulus came from. The model is a tri-modal encoder: it takes video, audio and text and predicts the fMRI response of an average person on the fsaverage5 cortical mesh, 20,484 vertices, once a second. It is not a measurement of anyone, and it does not read minds: it is a function from a stimulus to a predicted BOLD map, trained on about a thousand hours of other people's scans.

Access is not sold. The model's licence allows non-commercial use only, so the operator grants the program to an organization by agreement, after a conversation about who holds it and what they will study, and records both. A granted organization sees Brain in the console's sidebar and may call /v1/brain; everyone else gets 403 research_access_required. Write to [email protected].

What a prediction is#

A job takes one stimulus:

Kind	What the worker does
`text`	Speaks the text with a synthetic voice, transcribes it back to timed words (whisperx), and encodes the audio and the words.
`screen`	Holds a PNG still for `seconds` as a silent video: one frame of an environment, as a buyer or reader saw it.
`media`	Takes a video (MP4, WebM, MOV) or audio file (WAV, MP3, FLAC) as it is: frames, audio and transcript.

The result is one map a second, shifted 5 s into the past for the haemodynamic lag, in z-units of BOLD, for the "average subject" of the training set. The worker writes:

File	Holds
`preds.npz`	`preds` (seconds × 20,484, float16), `times` (seconds)
`rois.json`	every Destrieux parcel (148, per hemisphere): its vertex count, time-mean and peak
`features.npz`	the backbone features the model was given, time-mean per modality (Llama 3.2 3B for words, w2v-BERT 2.0 for audio, V-JEPA 2 for frames): the same stimulus without the brain projection, for matched-control analyses
`map.png`	the time-mean on the inflated surface, lateral and medial of each hemisphere
`summary.json`	the model and checkpoint digest, the stimulus's kind and digest, seconds predicted, modalities, hemisphere means, the ten strongest parcels, the runtime

A screen job also keeps stimulus.mp4, the still as the model saw it.

In the console#

Brain lists the organization's predictions with their state (queued, running, done, failed) and when they were queued. New prediction takes a text, the current screen of one of the organization's environments (rendered at 1280×800 and held for the seconds given), or a file. A finished prediction shows the map, the strongest parcels as bars, the hemisphere means, and Difference from another prediction: this one's parcel means minus another finished job's, split into where this one is stronger and where the other is. The files are downloadable from the ⋯ menu.

Over the API#

curl -s $API/brain -H "authorization: Bearer $CW_KEY"                 # the program: granted?, holder, purpose, today's jobs
curl -s $API/brain/jobs -H "authorization: Bearer $CW_KEY" \
  -d '{"kind": "text", "name": "Company page", "text": "Kuvra Inc. Applied research in industrial sensing…"}'
curl -s $API/brain/jobs -H "authorization: Bearer $CW_KEY" \
  -d '{"kind": "screen", "name": "Storefront", "seconds": 20}'        # answers `upload` and `accepts`
curl -s -X PUT $API/brain/jobs/$J/stimulus -H "authorization: Bearer $CW_KEY" \
  -H "content-type: image/png" --data-binary @frame.png
curl -s $API/brain/jobs/$J -H "authorization: Bearer $CW_KEY"         # state, summary, the files' URLs when done
curl -s $API/brain/jobs/$J/files/rois.json -H "authorization: Bearer $CW_KEY"

A key needs the research scope. A text job is queued at once; a screen or media job is awaiting_stimulus until its upload (at most 64 MB), then queued. An organization may start 24 jobs a day. DELETE /brain/jobs/{id} removes the job and its files.

Running it#

The model runs on the research machine, not in a Worker: scripts/brain/worker.py polls the API's queue with the operator's token, claims the oldest job, fetches its stimulus, runs scripts/brain/encode.py and puts the files back. encode.py is the same program a study runs offline on many stimuli (research/studies/brain-encoder/), so a published result and a console job are computed the same way. The environment is TRIBE v2's own (pip install -e tribev2, nilearn for the atlas and the map, whisperx through uvx); the weights and backbones come from Hugging Face.

What it is good for, and what it is not#

A predicted map is a model's guess about an average brain; two stimuli's maps differ where the model's backbones see them differ. The audit of these features as predictors (Rodrigues, 2026) found that the brain projection beats its own backbone on one memorability dataset and loses on another, so a predicted map is not a domain-general prior: any claim that it predicts a behaviour needs a matched control (the same backbone without the projection, which features.npz holds) and a held-out test on that behaviour. The brain-encoder study does exactly that against the market-perception studies' simulated buyers.