Large language models can predict how Americans will respond to social science experiments with an accuracy that rivals pooled human forecasters, according to a study published in Nature.
Researchers built an archive of 70 preregistered, nationally representative survey experiments conducted in the United States, encompassing 469 distinct experimental effects and 119,330 participants. Rather than running new experiments on human subjects, the team prompted an LLM to simulate how representative samples of Americans would react to experimental stimuli, then compared the simulated responses across different conditions to infer treatment effects.
The predictions came from GPT-4, whose training data cutoff predated the publication of many of the studies included in the archive. Despite this, the model's forecasts were strongly correlated with the actual treatment effects measured in the original experiments — a level of accuracy similar to that achieved by pooling forecasts from human experts.
Notably, the correlations held up even for studies that had not been published or publicly posted before the model's training cutoff date, suggesting the results were not simply a product of the model having seen the studies during training. The findings also extended to predictions generated by other prominent open-weight language models, not just GPT-4.
One consistent limitation emerged: even when directionally accurate, the model's predictions systematically overestimated the size of the effects found in the actual experiments.
Testing across a second archive
To probe how well this approach generalizes, the researchers also tested it against a secondary archive of 15 "megastudies" comprising 606 additional effects. Here, the correlations between predicted and actual outcomes were lower than in the primary archive, but still comparable to the performance of pooled expert forecasters.
To understand how such tools might be used — and misused — in practice, the researchers surveyed 460 social scientists about the likely applications and perceived risks of this approach. Using their archives, they assessed potential uses including pilot testing of experimental designs, selecting among candidate interventions, and identifying which effects are most in need of replication. They also examined risks such as bias and potential misuse of the technology.
The authors conclude that large language models could serve as a meaningful complement to traditional experimental methods in social science research and applied practice, while cautioning that their responsible use requires careful attention to the risks identified in the study. The researchers have also made a web-based demonstration available that generates AI-based forecasts of experimental effects using the method described in the paper.