Week 12 — Replication & cumulative science

Scope & method. Rapid narrative review, not a systematic review. Structured search of the sport-science replication and evidence-synthesis literature (PubMed, Sports Medicine, Systematic Reviews, R. Soc. Open Sci.), including Sports Science Replication Centre (SSRC) papers. Findings reported as the cited studies state them; the psychology reproducibility project is flagged as adjacent.

The principle under review Week 12 · replication and sound synthesis

Week 12 argues that a single study is provisional, that direct and conceptual replication filter chance findings and test generalisation, and that meta-analysis accumulates evidence only when its inputs are sound. The empirical questions are whether the field replicates its findings, what proportion hold up, and whether its syntheses are methodologically sound.

1 · How well is it currently applied? the evidence

Replication was essentially untested, and where measured is low

Until recently the field had run no large replication effort. Mesquida, Murphy, Lakens and Warne (2022) noted that few direct or conceptual replications existed, and Murphy, Mesquida, Caldwell, Earp and Warne (2022) proposed the first collaborative replication protocol for the field precisely because none existed. When that project reported, Murphy and colleagues (2025) found only 28% of 25 findings robustly replicated (7 of 25), with 88% of effect sizes shrinking on replication (median −75%) — somewhat below the ~⅓–½ observed in psychology's reproducibility project (Open Science Collaboration, 2015). The field's cumulative record has therefore been built with very little of the replication that would confirm it.

Replication is obstructed by upstream weaknesses

Reflecting on the project, Murphy, Caldwell and Warne (2025) report that replication was hindered by the poor reporting of statistical information, the frequent unavailability of original raw data, and a prioritisation of feasibility that risks some bias — problems they attribute to the wider field rather than to particular studies. In other words, the transparency and reporting deficits documented in Weeks 9 and 10 directly impede the replication that Week 12 relies on.

The other pillar — synthesis — is only as sound as its inputs

Meta-analysis can accumulate evidence, but only if the studies and the review itself are sound. Oliveira, Botton, De Nardi and Umpierre (2021), assessing 103 systematic reviews with meta-analysis in exercise science, found suboptimal adherence to methodological and reporting standards: only 4.9% provided a publicly available protocol and just 32% discussed the risk of bias in their included studies. Combined with the biased primary literature documented earlier (publication bias, low power, spin), this means syntheses can produce precise estimates of biased values — the “avoidable waste” Chalmers and Glasziou (2009) estimated at around 85% of research investment.

Source	Finding	Implication
Murphy et al. (2025) Sports Medicine	28% of 25 findings robustly replicated; 88% of effects shrank (median −75%)	Most tested findings did not robustly replicate
Mesquida, Murphy, Lakens & Warne (2022) R. Soc. Open Sci.	Few direct/conceptual replications existed in the field	The record was largely unverified by replication
Murphy, Mesquida, Caldwell, Earp & Warne (2022) Sports Medicine	First collaborative replication protocol for the field	Systematic replication is only now beginning
Murphy, Caldwell & Warne (2025) Sports Medicine	Replication hindered by poor reporting and unavailable raw data	Upstream weaknesses obstruct replication
Open Science Collaboration (2015) Science (adjacent)	~⅓–½ of 100 psychology findings replicated	Context for the field's 28%
Oliveira et al. (2021) Systematic Reviews	4.9% of exercise-science meta-analyses had a public protocol; 32% discussed risk of bias	Syntheses are frequently of low methodological quality
Chalmers & Glasziou (2009) The Lancet (adjacent)	~85% of research investment avoidably wasted	Poor primary studies and syntheses compound

2 · Is there cause for concern? objective assessment

Both pillars of cumulative science are currently weak. Direct replication was essentially absent until the past few years and, where measured, only about a quarter of findings robustly held; and evidence synthesis frequently fails basic methodological and transparency standards. Because the primary literature it draws on is itself biased and under-powered (Weeks 1, 7), the field's apparent accumulation of knowledge is less secure than its volume of publications suggests.

This is, however, the area of most active and organised reform. The field has now designed and executed its first large collaborative replication project, produced a selection protocol others can use, and openly analysed the barriers it encountered. Replication rates are a diagnostic to be improved, not a fixed verdict, and every practice in this course is aimed at raising them. The concern is substantial but the corrective machinery is, uniquely for this topic, already in motion.

3 · Immediate steps to improve actionable

1. Value and conduct replications — for the next study, consider whether replicating an important but shaky finding is more useful than another novel one-off (student and multi-site research is well suited to this).
2. Report enough to be replicable — full statistical detail, protocols, and de-identified raw data — since poor reporting is a leading barrier to replication (Weeks 9, 10).
3. Use direct replications to test reality and conceptual replications to test generalisation, and interpret each accordingly.
4. Hold meta-analyses to methodological standards (a public protocol, AMSTAR 2, explicit risk-of-bias assessment), and treat a synthesis of biased studies as a precise estimate of a biased value.
5. Engage with multi-site and replication networks, whose pooled contributions can achieve the power and generality a single laboratory cannot.

References