Synthetic study · simulated buyers · fictional companies and laboratory
Comparative positioning: when the truth points the other way
A factual campaign made query latency the deciding criterion for 180 simulated buyers, and still lost them for the company that sent it, mostly to deferral; where the competitor was in fact faster it nearly emptied that company's pipeline.
The result
Table view
| World | Exposure | Halvard | Corvel | Defer | n |
|---|---|---|---|---|---|
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Before any exposure | 0.661 | 0.333 | 0.006 | 180 |
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Nothing new | 0.606 | 0.278 | 0.117 | 180 |
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Independent benchmark | 0.683 | 0.278 | 0.039 | 180 |
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Halvard's latency campaign | 0.500 | 0.094 | 0.406 | 180 |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Before any exposure | 0.328 | 0.672 | 0.000 | 180 |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Nothing new | 0.322 | 0.567 | 0.111 | 180 |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Independent benchmark | 0.322 | 0.661 | 0.017 | 180 |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Halvard's latency campaign | 0.050 | 0.631 | 0.318 | 179 |
Reading the chart: deferral, trials, and the reversed world
In the baseline world, where Halvard really is faster, its campaign moved buyers away from both products: Corvel's share fell from 0.28 to 0.09 and Halvard's from 0.61 to 0.50, while deferrals rose from 0.12 to 0.41. The message ended “ask every vendor for the same number before you decide”, and buyers did: 123 of 180 asked to trial both products, against 48 with nothing new. Among those who did decide, Halvard's share rose from 0.69 to 0.84 (0.77–0.91), so the campaign may be winning a later decision while losing this one; this design cannot see the later one.
In the reversed world the identical strategy, truthfully stating Halvard's slower 1.9 s, cut choice of Halvard from 0.32 to 0.05. Budget buyers, who had chosen Halvard for its price, abandoned it: −0.80 (−0.90, −0.70).
The experiment
Two fictional data historians, two worlds that swap only their latency figures, 180 buyers each seeing nothing new, a benchmark or the campaign; preregistered.
Products. Two fictional plant data historians whose vendors' pages list the same fields: Halvard Historian (Halvard Data; 46 native connectors; $19,000 per site per year) and Corvel Stream (Corvel Systems; 140 connectors; $26,000). Both pages state a vendor-measured p95 query latency. In the baseline world Halvard's is 0.4 s and Corvel's 1.9 s; in the reversed world those two numbers swap and nothing else changes. Pages are shown in a per-buyer order that is the same in both worlds.
Exposures (the intervention), a week after a first choice.
- Nothing new: “A week later you come back to the decision. You have seen nothing new.” This is the control and the measure of drift.
- Independent benchmark: a fictional testing laboratory, paid by neither vendor, reports both products' latency, connectors and price measured on the same hardware, and makes no recommendation.
- Campaign: Halvard's own message — latency is what operators feel every shift; Halvard's p95 is 0.4 s or 1.9 s, whichever is true in that world; ask every vendor for the same number before you decide. True in both worlds; it names no competitor and makes no claim about one.
Buyers. 180 personas, 60 each with a latency-sensitive workload, a thirty-vendor equipment estate, or a tight budget (needs, not demographics, are the strata), with role, industry, number of sites and current setup drawn from named streams; seeds 21, 22, 23. Every buyer is run in both worlds and all three exposures.
Two tracks. In the choice track the buyer reads both pages and chooses (Halvard, Corvel, or defer), then, in a branch per exposure, chooses again and says which product, if any, they would trial. In a separate measurement track the same buyer allocates 100 points across five criteria and states a preference from −3 to +3, before and after the exposure, and rates how independent the exposure's source is. Ratings never precede a choice in the same conversation.
Model and run. deepseek-flash, temperature 0.7, reasoning effort low, prompt format
measure-v1; engine 0.3.0; spec digest a820e54fb6d94889; 2,880
model calls, 2,392,305 prompt and 733,570 completion tokens, 507 s.
51 of 19,440 answers did not parse and are dropped from their contrasts. The primary contrast
and the reversed-world test were committed before the final run.
Effects relative to nothing new
Table view
| World | Exposure (minus nothing new) | Pairs | Difference | Low | High |
|---|---|---|---|---|---|
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Independent benchmark | 180 | +0.078 | +0.039 | +0.117 |
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Halvard's latency campaign | 180 | -0.106 | -0.167 | -0.050 |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Independent benchmark | 180 | +0.000 | -0.028 | +0.028 |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Halvard's latency campaign | 179 | -0.274 | -0.341 | -0.207 |
| Interaction | Independent benchmark | 180 | +0.078 | +0.033 | +0.128 |
| Interaction | Halvard's latency campaign | 179 | +0.168 | +0.078 | +0.257 |
Did the criteria move?
Table view
| World | Exposure | n | Latency weight before | after | Change | Change vs nothing new |
|---|---|---|---|---|---|---|
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Nothing new | 179 | 23.9 | 23.4 | -0.37 | |
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Independent benchmark | 179 | 23.9 | 25.1 | +1.28 | +1.65 (+1.33, +1.99) |
| Baseline world: Halvard is faster (0.4 s against 1.9 s) | Halvard's latency campaign | 178 | 23.9 | 28.0 | +4.12 | +4.49 (+4.21, +4.76) |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Nothing new | 180 | 24.2 | 24.0 | -0.19 | |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Independent benchmark | 180 | 24.2 | 25.2 | +0.94 | +1.13 (+0.83, +1.44) |
| Reversed world: Corvel is faster (0.4 s against 1.9 s) | Halvard's latency campaign | 180 | 24.2 | 27.6 | +3.37 | +3.56 (+3.22, +3.87) |
Yes, modestly; but the campaign also told buyers to verify, and they deferred to do so.
Where the weight came from, and stated preference
Yes, modestly. The campaign raised the weight on latency by 4.1 points in the baseline world and 3.4 in the reversed one, taken mostly from price (−1.8 (−2.1, −1.5)) and connectors (−1.4 (−1.7, −1.1)). The benchmark, which names no criterion, moved it by a third as much. Salience changed; the choice did not follow the salience as a positioning plan would hope, because the campaign also told buyers to verify, and they deferred to do so. Stated preference moved the way the facts pointed: toward Halvard in the baseline world (+0.08 (+0.02, +0.14) on −3 to +3) and away from it in the reversed one (−0.19 (−0.28, −0.12)). Buyers rated the benchmark's source 6.5 of 7 for independence and the campaign's 1.7.
Who moved
Table view
| Need | Exposure and world | Pairs | Difference | Low | High |
|---|---|---|---|---|---|
| Latency-sensitive buyers | Campaign, baseline world | 60 | -0.217 | -0.367 | -0.067 |
| Latency-sensitive buyers | Campaign, reversed world | 59 | -0.017 | -0.051 | +0.000 |
| Latency-sensitive buyers | Benchmark, baseline world | 60 | +0.167 | +0.083 | +0.267 |
| Latency-sensitive buyers | Benchmark, reversed world | 60 | -0.017 | -0.050 | +0.000 |
| Connector-heavy buyers | Campaign, baseline world | 60 | +0.000 | +0.000 | +0.000 |
| Connector-heavy buyers | Campaign, reversed world | 60 | +0.000 | +0.000 | +0.000 |
| Connector-heavy buyers | Benchmark, baseline world | 60 | +0.050 | +0.000 | +0.100 |
| Connector-heavy buyers | Benchmark, reversed world | 60 | +0.017 | +0.000 | +0.050 |
| Budget-constrained buyers | Campaign, baseline world | 60 | -0.100 | -0.183 | -0.017 |
| Budget-constrained buyers | Campaign, reversed world | 60 | -0.800 | -0.900 | -0.700 |
| Budget-constrained buyers | Benchmark, baseline world | 60 | +0.017 | +0.000 | +0.050 |
| Budget-constrained buyers | Benchmark, reversed world | 60 | +0.000 | -0.067 | +0.067 |
Connector, latency and budget buyers
Before any exposure no connector-heavy buyer chose Halvard, and after every exposure at most 3 of 60 did: nothing about latency touched a decision made on coverage. Latency-sensitive buyers, the campaign's natural audience, deferred under it in the baseline world (−0.22 (−0.37, −0.07)) and responded best to the benchmark (+0.17 (+0.08, +0.27)). The damage in the reversed world fell almost entirely on budget buyers, Halvard's natural customers, whom the campaign taught to care about the one number on which Halvard lost.
What the buyers wrote
Reversed world, buyers who had chosen Halvard before the campaign (simulated rationales from the checked-in journal):
buyers-21-11 · budget need
Before: Halvard Historian. “Cheaper by $7k/year with adequate latency and connectors, so finance can justify it.”
After the campaign: Defer the decision; trial: Both. “Halvard's own message makes latency the decisive issue; I need to measure 1.9 s vs 0.4 s on our tags before finance accepts a $7k premium.”
buyers-21-14 · budget need
Before: Halvard Historian. “It meets our historian needs at $7,000 less per site, and finance will not approve Corvel's higher price without a clear operational reason.”
After the campaign: Defer the decision; trial: Both. “I need to test both on our own tags and hardware before finance can justify Corvel's $7,000 premium or accept Halvard's slower latency.”
Baseline world, where Halvard is faster, buyers who deferred after the campaign:
buyers-21-0 · latency need
Before: Halvard Historian. “Trend speed is our main operator complaint, and Halvard's 0.4 s p95 latency beats Corvel's 1.9 s while costing less per site.”
After the campaign: Defer the decision; trial: Both. “Latency is the operator complaint, so I want both measured on our own tags and hardware, not reference specs.”
buyers-21-1 · connectors need
Before: Corvel Stream. “Its 140 native connectors far better fit our thirty-vendor plant and avoid costly custom integration work.”
After the campaign: Defer the decision; trial: Both. “Halvard's latency claim needs checking on our hardware, and I need to count how many of our thirty vendors fall outside its 46 connectors before trading integration work for speed.”
Which factors predict the choice: a null
What this benchmark is
The factor-registry benchmark was run here too: predict whether a buyer chooses Halvard after the exposure from the measurement track, fitted on five of the six (world, exposure) scenarios and scored on the sixth. The baseline is the pre-exposure choice and the stated preference.
Table view
| Factor | Reduction | Low | High |
|---|---|---|---|
| independence | +0.0320 | +0.0194 | +0.0440 |
| w_connectors | +0.0010 | -0.0019 | +0.0032 |
| w_latency | -0.0184 | -0.0246 | -0.0126 |
| need_latency | -0.0325 | -0.0390 | -0.0258 |
| w_price | -0.0342 | -0.0416 | -0.0260 |
| need_budget | -0.0432 | -0.0512 | -0.0342 |
| true_advantage_ours | -0.0602 | -0.0721 | -0.0485 |
| latency_x_advantage | -0.0804 | -0.0941 | -0.0655 |
Robustness
Measured against the nothing-new branch of the same conversation; all three seeds negative.
- Negative control. With nothing new, 22 of 180 buyers in the baseline world and 21 of 180 in the reversed one changed their choice, nearly all to deferring. That drift is why every effect above is measured against the nothing-new branch of the same conversation, not against the first choice.
- Seeds. The primary contrast by seed: seed 21: −0.10 (−0.22, +0.02); seed 22: −0.13 (−0.23, −0.05); seed 23: −0.08 (−0.18, +0.02). All three are negative.
- The benchmark repeats the vendors' numbers. The lab's figures equal those on the vendors' pages, so the benchmark adds independence, not information. Its effect is +0.078 in the baseline world and +0.000 in the reversed one, and it cut deferral in both.
- Empty replies. deepseek-flash reasons before it answers, and a reply that spends its whole 2,000-token budget reasoning comes back empty. Prompts answered with nothing usable, per arm: baseline/benchmark 1 of 720, baseline/campaign 2 of 720, baseline/none 1 of 720, reversed/benchmark 0 of 720, reversed/campaign 1 of 720, reversed/none 0 of 720; with some items unusable: 23 in all. The rate is at most 2 of 720 in any arm and does not line up with the contrasts, so it is not what moved them; the affected answers are dropped, not re-asked.
- Replay. The journal replays the run offline with 0 model calls to byte-identical records.
Limitations
One language model playing every buyer, text stimuli, one decision; a later decision the campaign may win is outside the design.
- One language model plays every buyer. The deferral response to “ask every vendor” may be a trait of this model's caution rather than of procurement managers; it is the first thing a human study should check.
- The decision is two forced choices a simulated week apart. Real evaluations run for months, trials happen, and the deferred decisions here would resolve one way or the other; the conditional share above hints at how, but is not an outcome.
- Full information: both vendors' pages state latency. Where buyers do not know a competitor's number, a benchmark and a campaign carry information as well as salience, and could behave differently.
- Within a need stratum the simulated buyers are close to deterministic. Before any exposure 59 of 60 budget buyers chose Halvard in both worlds, 0 of 60 connector-heavy buyers did, and latency-sensitive buyers chose whichever product was faster (60 of 60 and 0 of 60). The one-sentence need in the persona does most of the work; real buyers are noisier and their needs are mixed.
- The campaign is one message. Frequency, channel, sender credibility and competitor response are outside the design.
- The factor ablation has six scenarios; scenario-level factors are not identified by leave-one-scenario-out.
What would establish external validity
Real buyers shown the same stimuli, and field tests of comparative claims.
- A randomised vignette experiment with real OT and plant-IT buyers using these pages and exposures, measuring choice, deferral and trial requests.
- Field data on deferral: for a real vendor, the rate at which opportunities stall after comparative campaigns versus analyst or lab reports, from CRM stage histories.
- Trial-to-purchase conversion when the buyer's own test agrees or disagrees with the vendor's claim, which decides whether deferral here is lost or delayed revenue.
Reproduce it
Replay the checked-in journal offline: no model, no key.
# replay from the checked-in journal: no model, no key needed
python research/studies/market-perception/comparative-positioning/run.py --phase final --offline \
--out target/comparative --journal research/studies/market-perception/comparative-positioning/evidence/journal.jsonl
python research/studies/market-perception/comparative-positioning/analyze.py --evidence target/comparative
# a fresh sample asks DeepSeek (DEEPSEEK_API_KEY in .env); without it the run stops at the first call
python research/studies/market-perception/comparative-positioning/run.py --phase final --out target/comparative-newData: results.json (every number on this page), buyers.csv (one row per buyer, world and exposure), spec.json (every stimulus), preregistration.json. Source: run.py, analyze.py, measure.py, factors.py, the model journal.