How Accurate Are Synthetic Users? 14 Human Comparisons

Synthetic-user accuracy depends on the task, audience and measure. In the 14 Sapien comparisons below, the leading menu message matched human research, but one EV scenario reversed the preferred alternative. Compare results in their original units and check whether an error would change the decision.

Our company built Sapien. These comparisons put its estimates beside published human research and identify each audience, question and scale. Twelve cases report normalized accuracy scores. Two others report dollar differences and matches to interview findings. Read the paired results to see both close agreement and misses hidden by a high overall score.

Creative and message testing

1. Ad testing

USA TODAY’s Ad Meter ranks Super Bowl commercials using scores from registered viewers, who rate each ad from one to five. Sapien scored eight ads from the 2025 game to see how closely it could recreate those human ratings.

Sapien reached a reported 96.5% normalized accuracy across the eight tested ads. Its ratings differed from the human scores by about 0.14 points on average. All eight were within half a point; seven were within 0.2.

Scores are out of five. Parentheses show ranks within the eight tested spots, rather than all 57 Ad Meter ads. The human scores come from USA TODAY’s full ranking and winner announcement. The Homes.com sequel ranked 40th overall.

Uber Eats was the only break in the relative order. Remove it from both lists and the remaining seven ads have exactly the same order. Its lower Sapien position shifts Novartis, Mountain Dew and Disney+ in the displayed ranks. Those shifts are not three separate ordering errors.

Both groups chose Budweiser. This comparison supports using the model to shortlist ads for this task. The 0.46-point Uber Eats gap needs investigation. Liking an ad does not tell you which one will generate more sales.

2. Message testing

Can a sentence above a menu change what people order? Trial 2 of the World Resources Institute’s menu study tested five environmental messages and a no-message control with 2,902 people. Each person chose lunch and dinner.

Sapien recreated the conditions and counted vegetarian selections across both meals. Every condition was within 0.7 percentage points of the human result, and the complete message ranking matched.

These are shares of menu selections across lunch and dinner, not shares of participants. The reported normalized accuracy score was 97.8%.

“Small changes, big impact” led in both comparisons. It increased vegetarian selections over the control by 12.8 points in Sapien and 13.0 points in the human trial. A messaging team would have chosen the same line to advance.

3. Claim and label testing

An environmental label can improve a product’s perceived sustainability without producing an equally large change in willingness to buy it.

The Taillie and colleagues experiment randomized 1,013 US parents to see eight products with an eco-score label or a barcode control. They rated sustainability and future purchase intention on 1–5 scales. Sapien recreated those conditions in the US-parent context.

Eco-score minus barcode control, on 1–5 rating scales. Selected effects from the eight-product comparison.

The veggie-pizza result shows the difference. The sustainability-rating gain was more than twice the purchase-intention gain in both sets. A packaging team needs to know which result a label changes. Neither rating is a purchase.

Product decisions and pricing

4. Purchase intent testing

A customer who will try an unfamiliar food once may still decline to buy it regularly. Bryant and colleagues’ 2019 study asked about regular purchase of plant-based meat and “clean meat,” its term for cultivated meat. Sapien recreated the US sample context.

Stated regular-purchase intentions under the study’s 2019 product descriptions.

The high-intent cultivated-meat estimate was within 0.6 percentage points of the human result. Sapien underestimated outright rejection of plant-based meat by 6.6 points. That matters when planning for regular use rather than a first try. These results concern the study’s 2019 descriptions; they do not forecast current sales.

5. Feature prioritization

Which EV benefits deserve the most attention? The 2019 Dutch proEME survey recorded weighted responses from 1,553 participants, who could select several purchase priorities. Sapien recreated the same multi-select factor list.

These are selected factors from a 15-answer question. People could choose several, so shares do not add to 100%.

Both sets have the same order, and each displayed share is within 0.7 percentage points. Either would put price and range first in the brief. These were priorities for that audience in 2019. A launch today needs a current audience and offer.

6. Willingness to pay

A pricing test becomes more informative when buyers can choose nothing. Our recreation of the Van Loo, Caputo and Lusk burger experiment offered four burger types at different prices without brand names, alongside a no-purchase option.

These are choice-model estimates in dollars per pound, relative to buying nothing. They are compared with the human study’s unbranded MNL control. They are not prices buyers named or paid at checkout.

Across the four products, the average absolute difference was $0.24 per pound. The largest gap was $0.39 for farm-raised beef. Both sets ranked the products in the same order. This gives a pricing team a close estimate for this task. Test an actual offer before choosing a shelf price.

7. Product trade-off testing

Averages can conceal a reversal that changes the decision. Using Zou, Khaloei and MacKenzie’s EV scenarios, we varied the walk to slow charging and access to fast charging within a 15-minute drive. The selected new- and used-car scenarios specified a 40-year-old man who did not own an EV.

These are selected scenarios from a 36-scenario comparison. The human probabilities are approximate calculations from the published choice model. They are not raw observed choices or EV sales.

Without fast charging, Sapien estimated higher EV choice among used-car buyers than new-car buyers. The human-study model had the reverse order. The used-car gap was about 18.4 percentage points. A charging-access decision needs this detail even if the overall comparison has high agreement.

View an example study to see the audience, offer and findings before planning your own.

Customer behavior and segmentation

8. Shopping behavior

How do shoppers respond to rising food prices? We compared four reported behaviors from the 2024 IFIC Food and Health Survey. The audience was people who noticed food and beverage price increases; each share combines “always” and “often.”

Human eligible base: 2,728 people who noticed price increases. Human shares are from the IFIC report.

Sapien was within 1.9 points for switching to cheaper products or brands. It underestimated choosing less-premium products by 7.8 points. A consumer-goods team needs to distinguish those responses to price pressure. Both sets record what people said they did, rather than basket transactions.

9. Audience segmentation

Do people with EV experience value the same benefits as those who have never driven one? Sapien recreated the six-benefit question in the 2019 Dutch proEME survey. Each result below subtracts the never-driven share from the experienced share. It is not either group’s own percentage.

Weighted human responses from the published survey data: 212 EV-experienced and 1,323 never-driven participants, excluding 18 unsure responses. pp means percentage points.

Sapien matched the price gap and followed the human direction for range, charging stations and running costs, each within two points. The climate-effect gap had the opposite sign. These differences can inform which benefits to emphasize for each audience, but the groups contain different respondents: the comparison cannot establish that driving an EV caused the differences. See the original survey and questionnaire.

10. Brand perception

Knowing a brand and liking it are separate judgments. The BRAND dataset measures both. We tested eight food, beverage and restaurant names from its 2024 US brand-name study, using a 1–7 scale for each judgment.

Brand-name liking, rather than logo liking. Each mean in the human 2024 BRAND data is based on 39–41 ratings.

Sapien put Cheerios slightly above Chick-fil-A; people put Chick-fil-A slightly above Cheerios. Budweiser had a larger gap of 0.66 points. Distinguish a near tie from a larger miss rather than treating every rank change alike.

11. Customer satisfaction

We tested customers’ ratings of their own providers across eight product groups in FCA Financial Lives 2024. Sapien’s averages stayed between 7.57 and 7.91 out of 10. The FCA’s published results vary more sharply.

Ratings from 0 to 10. Each category covers holders of that product; FCA averages exclude “don’t know” answers.

Sapien ranked pension satisfaction highest in its tested set. The FCA found it comparatively low. Sapien’s pension estimate was 1.21 points above the human average. A team deciding which customer experience to improve should resolve that gap before spending resources.

Offers and decision clarity

12. Offer testing

Could a simpler reply turn an attractive offer into action? Trial 3 of FCA Occasional Paper 19 sent longstanding savings customers a letter offering a higher-rate account. One version added a prefilled form and prepaid envelope.

The human study recorded action within four weeks: converting or closing the account, or withdrawing at least 95% of its starting balance. Sapien simulated answers corresponding to those outcomes.

The displayed rates imply 84 additional actions per 1,000 recipients in Sapien, close to the human study’s 87. Both would point the team toward the simpler response package with a similar estimated benefit. The human result records action; the Sapien result simulates it.

13. Customer interviews

Can synthetic interviews surface the concerns found in human qualitative research? We recreated the early interviews from FSA/Ipsos research into online food ordering, before discussion of hygiene-rating display requirements.

Human participants focused on cost, delivery time, reviews and familiar outlets. Many assumed food was safe without actively checking hygiene. Unfamiliar outlets or bad experiences made hygiene more important to them. The simulated sample had 40 participants: 20 in England and 20 in Northern Ireland.

Reviewers compared simulated interviews with published human findings. These percentages count matches to findings. They do not measure how common a concern is or supply a normalized accuracy score.

Both reviewers found much of the human pattern, but they chose different sets of five matches. The comparison supports using simulated interviews to develop questions about familiar concerns. It cannot show how common those concerns are across the population.

14. Comprehension testing

Explaining an offer more clearly may help with one benefit while leaving another unchanged. The BIT/Citizens Advice experiment compared a standard home-battery offer with one explained through a plain FAQ.

We focused on understanding cheaper off-peak electricity, solar energy use and “flexibility services,” which let an energy supplier use the battery.

Each change is the share answering correctly after the FAQ minus the share after the standard offer. The human report did not test whether these differences could be due to chance.

Sapien’s correct answers for flexibility services rose from 21.8% to 37.0%, closely matching the human improvement. Its solar result fell from 66.0% to 62.5%, also in the human direction. Either would identify the same largest gain in understanding. That does not tell you whether clearer wording increases purchases.

Check the decision as well as the score

Ask whether the model works for your question. Close ad-liking scores do not establish accuracy for a different audience’s willingness to pay. Choose human comparison data that fits the decision and compare results in the units you will use to make it.

Start with the original study. Record the population, recruitment, exact questions, alternatives and measures. Keep the conditions comparable: selecting several features is different from choosing one favorite, and understanding is different from buying. Label any result calculated from a published model. The EV trade-off figures above are approximate model probabilities, not observed shares.

For a new validation, define success before reading the human results. For a message choice, check the winner and size of its lead. For a target-group decision, check whether the model identifies the same group. Keep the human answers out of simulation inputs and save what the model received. With published studies, this does not prove the answers were absent from model training data. Record what was controlled and what remains unknown.

Show the miss that could change the recommendation

Show results side by side with their differences and any disagreement that could change the choice:

• In the used-car scenario without fast charging, Sapien estimates EV choice at 53.1% while the reconstructed human estimate is 34.7%. That puts EV choice above versus below the halfway point and reverses the preferred alternative, even though other EV comparisons are closer.
• In the financial-product comparison, a narrow range of synthetic satisfaction scores conceals the pension gap. Sapien's 7.91 and the human 6.7 differ by 1.21 points on the ten-point scale, affecting which product looks most in need of attention.
• In the savings-offer comparison, the useful quantities include the 8.4- and 8.7-percentage-point increases. Readers can see the intervention's estimated size without having to infer it from a normalized score.

Check buyer-group differences too. The climate answer in the EV comparison has opposite directions in the simulated and human results. A close overall pattern does not justify using that particular group recommendation.

Keep different measures and studies separate

Ask how an accuracy score was calculated, which results it includes and how studies were combined. A rank, percentage-point error, dollar difference and interview-finding match answer different questions. Keep each with its task. Combining them requires a justified common measure.

The twelve normalized scores are reported results. The complete formulas and inputs for each measured outcome are not provided here. You cannot recalculate them from these selected tables alone. Ask Sapien whether those materials are available. Until then, judge the decisions directly supported by the paired results and errors shown here.

Use a disagreement to choose the next step. You may need better audience data, clearer research materials, a narrower claim or a focused human study. Keep the original result when changing the setup so readers can see what changed. Useful validation shows which recommendation the team can act on and which still needs evidence.

What these comparisons show

The closest matches included menu-message selection, EV feature priorities, burger pricing and the simpler savings offer. Sapien chose the same leading menu message and ranked the displayed EV priorities in the same order. Burger willingness to pay differed by an average of $0.24 per pound. Uber Eats was the only break in the relative order of the eight ads.

Among the 12 cases with reported normalized scores, the lowest was 95.3% for brand familiarity and liking. The highest were about 98.3% for EV priorities and offer testing, rounded to one decimal place. Each concerns its own task and measured outcomes. These are not percentages of people predicted correctly. The other two cases use dollar differences and interview matches. Use the paired results to understand agreement in practical units.

The cases cover consumer goods, food and restaurants, cars, financial services and home energy. Choose a similar case to define what a proposed study needs to get right: message order, a price gap, a buyer-group difference or an offer’s effect on action. Resolve important disagreements before committing resources based on a simulated result.

For a product-concept example, see our illustrative Red Bull protein study. It uses unofficial concepts and simulated consumers; it was not a project commissioned by Red Bull.

If you’re considering Sapien, start with the closest paired case above. Ask which audience, stimulus, outcome and failure check a proposed study would use for your decision. Explore the research you can scope with Sapien.