When social scientists want to understand what changes people's minds or behaviour, they run experiments on real participants — work that costs considerable time and money. A new study in Nature suggests large language models could ease part of that burden by forecasting, in advance, how such experiments are likely to turn out.
A team led by Harvard psychologist Ashwini Ashokkumar, with collaborators at Stanford, assembled an archive of 70 preregistered, nationally representative survey experiments conducted in the United States — 469 measured effects across nearly 120,000 participants. They asked GPT-4 to simulate how representative samples of Americans would react to each experiment's messages and questions, then compared the model's estimates against the real results.
The predictions tracked the actual treatment effects closely, reaching accuracy similar to pooled human forecasters. The correlation held even for studies that appeared after the model's training cutoff — work it could not have memorised — and for several prominent open-weight models. In a harder second archive of 15 "megastudies" with 606 effects, accuracy fell but stayed comparable to expert forecasters.
The authors are careful about what this does and does not show. GPT-4 was good at ranking which interventions would work better, yet it systematically overstated how large the effects would be — often by roughly double. A model that predicts responses, they stress, is not one that understands people; "synthetic respondents" are no substitute for real populations, and accuracy varied somewhat across demographic groups.
A tool for planning, not replacing
The researchers, who also surveyed 460 social scientists about likely uses and perceived risks, frame the models as a way to augment research rather than replace it. Cheap forecasts could guide pilot testing, help select the most promising interventions, and flag findings most in need of replication — steering scarce human effort to where it matters. Tellingly, combining model predictions with human forecasts beat either on its own. A public demo at treatmenteffect.app lets researchers try the approach for themselves.
The team is equally clear about the risks, from reproducing dominant patterns in the training data to helping optimise persuasive but harmful messaging, and it calls for safeguards that go beyond simply blocking obvious prompts. Used with those limits in mind, the researchers argue, language models could become a genuinely useful new instrument in the social scientist's toolkit.