- An international collaboration of 865 academics sampled claims from 3,900 research papers in behavioural and social science
- The researchers, including Aston’s Dr James Reynolds and Dr Natalia Stanulewicz-Buckley, assessed research repeatability, including reproducibility, robustness, and replicability
- There is no single indicator of credibility, but the results will help to strengthen how research is interpreted and communicated
New research carried out by an international collaboration of 865 researchers has published three papers in Nature, as well as five pre-prints, looking at the credibility of research findings in behavioural and social science.
Funded by the US Defense Advanced Research Projects Agency (DARPA), the Systematizing Confidence in Open Research and Evidence (SCORE) programme examined multiple dimensions of research repeatability, including reproducibility, robustness, and replicability, and assessed how well people and machine methods can predict whether findings will replicate. SCORE was coordinated by the Center for Open Science (COS).
To support consistent interpretation of these results, SCORE also released a short preprint that explains the standardised terminology used in the programme. In this research, “reproducibility” refers to re-running the same analysis on the same data; “robustness” tests whether conclusions hold under reasonable alternative analyses of the same data; and “replicability” refers to whether findings hold up when tested with new data.
Human expert assessments were conducted by two independent teams, repliCATS and Replication Markets. Three teams led by researchers at Pennsylvania State University, TwoSix Technologies, and the University of Southern California implemented machine-learning and algorithmic approaches to predicting replicability, and the Metascience Lab at Eötvös Loránd University coordinated robustness assessments.
SCORE sampled claims from 3,900 papers published between 2009 and 2018 in 62 journals spanning criminology, economics, education, health, management, psychology, political science, sociology, and other fields.
The programme’s outcomes will contribute to strengthening how research is interpreted and communicated, work that supports authors, reviewers, funders, policymakers, and readers’ understanding and use of research evidence. Improving credibility assessment will help direct attention and resources for further research to where they have the greatest impact in accelerating production of knowledge and solutions.
Key findings include:
-
About half of tested findings replicated, consistent with prior large-scale replication efforts: Of 164 papers subjected to replication attempts, 49% replicated by the most common criterion (statistical significance with the same pattern as the original study).
-
Field differences were limited: No social and behavioural science field showed consistently higher repeatability overall, although variation in data availability and sharing practices contributed to observed differences in reproducibility across fields.
-
Humans forecast replication reasonably well; tested machine methods did not: Human assessments achieved 76% and 78% success rates by the best-performing metric for the two methods, respectively. Three distinct machine-based methods were tested, and none were effective at predicting which claims would replicate successfully or not.
-
Different analyses led to different results: 72% of reproduction tests reproduced at least approximately, and 53% reproduced precisely. But, in robustness testing of 100 papers, 34% of independent reanalyses matched the original within a narrow tolerance (± 0.05 Cohen’s d units), and 57% matched within a wider tolerance.
-
There is no single indicator of credibility: The papers emphasise that credibility assessments are diverse and no singular measure captures credibility. Repeatability is just one type of credibility indicator, and outcomes varied substantially within different repeatability assessments: reproducibility, robustness, and replicability.
Dr James Reynolds, a senior lecturer in psychology at Aston University, was involved in the research, after responding to a call out for people with analytical experience to test the robustness of the analysis done in existing research. He was allocated a paper to analyse the data and also contributed to the editing/reviewing of the manuscript.
Dr Reynolds said:
"Our work highlights that research published across the behavioural and social sciences cannot be simply trusted to be accurate. Although around three-quarters of findings were replicated when we reanalysed the original data, that leaves around a quarter of results where different researchers came to different conclusions when analysing the data. This work also provides a clear strategy for improving the quality of data analysis."
Dr Tim Errington, senior director of research at COS and one of the SCORE project leaders, said:
“The main message of SCORE is a simple one: research is hard. And, in some ways, the hard work begins after making a discovery. A tremendous amount of work is needed to verify and have enough confidence in new discoveries to build foundations for further discovery.”
Visit the SCORE website for an overview of the programme, links to the papers, press releases for each paper, and other context to understand the project, findings, and implications.
- Notes to editors
-
For over 130 years, Aston University has been making our world a better place through education, research and innovation. Our history is intertwined with the remarkable city of Birmingham, once the heartland of the Industrial Revolution and now the thriving base for an innovation ecosystem of global significance, which Aston is co-creating.
Our vision is to be a leading university for science, technology and enterprise, measured by the positive transformational impact we achieve for our people, students, businesses and the communities we serve.
Aston focuses on high-quality, exploitable research that has an impact on society through medical breakthroughs, advancements in engineering, policy and practice in government, and the strategies and performance of business.
The University offers a range of undergraduate and postgraduate degree programmes, as well as continuing professional development solutions.
Thanks to its focus on delivering excellent outcomes for students, Aston University's reputation continues to grow. It was recognised as the Daily Mail University of the Year for Student Success 2025, is second in England for social mobility (2023 HEPI Social Mobility Index), and is top 20 for graduate salaries (2024 Longitudinal Education Outcomes).
Aston University is now defining its place in the Fourth Industrial Revolution (and beyond) within a rapidly changing world.
For media inquiries in relation to this release, contact Helen Tunnicliffe, Press and Communications Manager, on (+44) 7827 090240 or email: h.tunnicliffe@aston.ac.uk.
Be first to get the latest news, research and expert comment from Aston by following us on X
Need an expert for your story? Browse our experts directory.