What Is Silicon Sampling? How AI Populations Work
Silicon sampling uses language models to simulate survey answers. Learn what human comparisons show, where the method fails and how to check your planned study.
Silicon sampling uses a language model, guided by information about a target population, to generate survey answers intended to stand in for those people's responses. It can help screen ideas when comparisons with suitable human evidence support that specific task.
In this guide's invented teaching example, rewording one prompt flips which of two product concepts wins, with the same audience profile and concept cards.
Book a demo to discuss the audience data and assumptions for your question. See our comparisons with human research and check results for the particular simulated task you need.
What the studies tested
The studies below test different tasks and populations. Our synthetic-user evidence guide shows human and simulated results side by side, including important misses. Read each result in its own setting; combining them into one accuracy percentage would hide those differences.
Research | Details |
|---|---|
Argyle and colleagues, Out of One, Many, published 2023 | Construction and evaluation described by the authors: GPT-3 conditioned on sociodemographic backstories from participants in US surveys; silicon samples compared with human response patterns Scope to preserve: Evidence about that conditioning, model and social-science context, rather than every topic or model |
Construction and evaluation described by the authors: Interview-grounded agents representing 1,052 people, evaluated on survey responses and other tasks Scope to preserve: Individual interview grounding and the stated evaluation tasks; the reported 85% GSS figure is normalized to human retest performance |
Argyle and colleagues gave GPT-3 demographic and social profiles from U.S. survey participants, then compared its answers with human response patterns, including patterns within groups. That result applies to their model, profiles and social-science questions. It cannot tell you whether a demographic profile alone can answer any product question in any market.
Park and colleagues’ linked November 2024 preprint used interviews to build agents representing 1,052 people. Its reported 85% GSS result is measured against how consistently each person answered again after two weeks. It is not “85% correct” on any marketing question. The paper was later revised, including in June 2026. The figure here comes from the linked first version.
The studies use different information about people and different tests. Use them to ask better questions about validation, not to borrow one accuracy score. Neither study evaluated Sapien. Ask which audience, data and task match the study you want to run.
Chen, Zhu and Zheng’s 2026 preprint compares demographic prompting with methods that do not use a language model, using GSS and WVS data. Those alternatives are fitted and evaluated on different respondents. They include a question-marginal rule, demographic lookup, logistic regression and random forest. Matching an overall pattern can still hide errors about individuals or groups. Ask for separate checks of overall distributions and targeting decisions. The tested methods do not evaluate Sapien.
Chidichimo and colleagues’ 2026 preprint sets out eleven tests of how answers are produced, how measures relate and how results match human responses, groups and experimental effects. It compares nine models across twenty human sources and keeps track of estimates that are unavailable. For segmentation, ask whether the method keeps the group differences and relationships your decision needs. A matching average is not enough. The paper tests its listed settings and sources, not Sapien.
Validation starts with the decision
Before testing a model, define the audience, market, exact product cards, what each person sees and the main answer you need. A realistic-sounding response is not enough. Ask whether the recommendation matches suitable human evidence and whether an error could change which offer you develop.
Use human comparison data kept out of the model’s preparation where the test requires it. Save recruitment details, what people saw and which answers counted. A result for one recruited sample is not automatically a result for the whole population. Too few people in a critical group leave that group untested, even if the overall result matches.
Keep test answers out of the information used to build the model. If those answers also supply its data, matching them may reward access to the solution. Record what information was allowed, what was kept for testing and any changes made after seeing the results.
Also name a simple comparison method, or baseline. For a two-option concept task, a majority rule based on permitted training data may be useful. Another task may need a simple statistical model. Choose it before checking the held-out results and give it the same information restrictions as the language model.
Choose measures that reflect the decision. For answer shares, check the full distribution and the size of share errors. For concepts, check rankings and changes in the recommended winner. For open answers, check important and conflicting themes. A good overall average can hide the error that changes what you should develop.
What to ask for in a validation report
Record field | What should be inspectable |
|---|---|
Task and decision | Audience, stimuli, instrument, exposure, primary measure and action the result would support. |
Human reference | Recruitment, field date, valid bases, important subgroup coverage and known limitations. |
Model construction | Version, grounding inputs, prompts, configuration, permitted information and preparation date. |
Evaluation | Held-out boundary, named baseline, prespecified metrics, invalid-output handling and all tested conditions. |
Recommendation | Where errors change the action; which audience/tasks are supported; unresolved evidence and failure response. |
Example: changing one prompt changes the leading concept
The example CSV and standard-library Python summary reproduce all five conditions below. This is a hypothetical dataset, including the “human reference.” Every row has 100 valid answers. No human recruitment or model run took place.
The reference gives A 52 answers and B 48. Prompt 1/profile 1 gives A 60. Changing only the prompt gives A 45, so B leads. Changing the profile under prompt 1 ties the concepts at 50–50. Repeating the baseline gives A 58. Keep all five conditions: choosing only the favorable first result hides the reversal.
To adapt the exercise, keep the questions and cards fixed. Compare two reasonable neutral prompts, two supported profiles and a repeated baseline. One prompt could ask the model to answer using the supplied profile. Another could explicitly limit it to the profile and card, leaving unknown information unknown. These are proposed checks of sensitivity, not an established validation protocol.
A prompt change should not change the offer, add a favorable fact or tell the model which concept to prefer. Test reasonable ways of describing the same task. Build alternative profiles from permitted evidence, not convenient stereotypes. Save the exact prompts and inputs so you can investigate a reversal.
One prompt change reverses the leading concept
The reference favors A by four percentage points: 52−48. A’s share is two points above the 50% choice boundary. The prompt change moves A from 60% to 45%, a fifteen-point swing across that boundary. Averaging the two conditions would hide that the winner depends on how the responses were generated.
Against the reference, A’s share errors are +8 points for the first version, −7 for the changed prompt, −2 for the changed profile and +6 for the repeated baseline. These are arithmetic differences within the five-condition dataset. They are not confidence intervals, estimates of population error or proof that one prompt is correct.
The repeated baseline changes A from 60% to 58%. One repeat shows only one contrast for that setup. It cannot describe the full range of run-to-run variation or validate the profile. Changing the profile checks a different dependency, so report it apart from the repeat.
Judge variation by what it would change. A small ranking error may matter little when both concepts clearly qualify for the next stage. The same shift may decide a close choice. Set the useful difference and what you will do if the method fails before seeing results. Use your commercial decision, not this example’s numbers, to set the tolerance.
The failure rule belongs in the study plan
Agree what would stop the project moving forward: a reasonable setup changes the winner, an important buyer group differs from the human comparison or too many outputs cannot answer the question. Decide whether that calls for better audience data, a smaller scope, a revised setup or direct human research.
Keep refused, malformed, conflicting and unanswerable outputs and explain how each was handled. A percentage calculated only from usable answers can hide poor coverage. Report attempted and valid outputs with the main result. Generating more answers does not give the precision of an independent human sample.
Do not choose the run closest to the held-out human result after seeing every run and call it an untouched test. If you adjust the model using that result, record the adjustment and use fresh evaluation data for the revised claim. Ask the provider to make this clear in the report.
Check important gaps before buying more generated responses. More responses may reduce random variation within one setup while errors in prompts, audience data or measurement remain. If the audience data lacks a niche buying situation, generating more rows cannot supply that experience.
A’s first lead is not enough to move forward
Do not choose A just because the first simulated condition gives it 60%. Find out why a reasonable prompt change reverses the choice, then use suitable human evidence to resolve the real comparison. The example shows a validation approach; its invented reference cannot decide a product launch.
For a proposed study, ask for validation on a similar task and the sensitivity checks beside the recommendation. Our managed research service compares options with simulated audiences. The proposal should explain which audience data and validation support the conclusion. Claims about a group not covered or physical product use need their own evidence.
A later successful test supports the audiences, tasks and changes actually tested. Use it to decide what the simulated comparison can answer, then name the evidence still needed for launch, behavior or a missing buyer group. More targeted human research can be a useful recommendation.
Questions about silicon sampling
Silicon sampling vs synthetic panels: what is the difference?
Silicon sampling generates responses intended to represent a population. A synthetic panel is the simulated group used for a study or follow-up. The terms overlap. Neither tells you which data built the audience or whether its answers to your question have been checked against suitable human evidence.
The reversal, with data and code
Save the complete CSV as silicon-sensitivity.csv and the Python code as summarize-sensitivity.py in the same folder. Run python3 summarize-sensitivity.py to print A’s share of valid answers, its percentage-point difference from the example reference and the number of invalid answers for every condition. You can also pass a compatible CSV path as the first argument.
All five condition records
status,condition,a_chosen,b_chosen,invalid
fictional teaching data,human-reference,52,48,0
fictional teaching data,prompt1-profile1,60,40,0
fictional teaching data,prompt2-profile1,45,55,0
fictional teaching data,prompt1-profile2,50,50,0
fictional teaching data,prompt1-profile1-repeat,58,42,0
Full sensitivity summary
"""Summarize the adjacent fictional teaching CSV, or pass a compatible CSV path."""
import csv
import sys
from pathlib import Path
path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(__file__).with_name('silicon-sensitivity.csv')
with path.open(newline='') as handle:
rows = list(csv.DictReader(handle))
shares = {}
for row in rows:
a, b, invalid = (int(row[key]) for key in ('a_chosen', 'b_chosen', 'invalid'))
if min(a, b, invalid) < 0 or a + b == 0:
raise ValueError('Counts must be nonnegative and include at least one valid response')
shares[row['condition']] = 100 * a / (a + b)
reference = shares['human-reference']
for row in rows:
share = shares[row['condition']]
print(f"{row['condition']}: A={share:.2f}%; difference={share-reference:+.2f} percentage points; invalid={row['invalid']}")
Further reading
For a scoped Sapien study discussion, name the choice that would change if the ranking reversed. Ask which validation and sensitivity checks support that choice and which question still needs human evidence.
