Replication
in Sports Science
Jenny Murphy
Background
Supervised by Dr. Joe Warne
What is replication?
Replication is retesting a claim using
the same analyses and new data
(Nosek et al, 2020)
Same
methods
New
sample and
data
Compare
results
Reproducibility is retesting a claim
using the same analyses and same data
(Nosek et al, 2020)
Same
data
Compare
results
Types of replication
Exact
replication
Systematic
replication
Conceptual
replication
Direct
replication
Close
replication
Operational
replication
Literal
replication
Why replicate?
Rigorous science is the
“theoretical or
experimental approaches
undertaken in a way that
enhances confidence in
the veracity of their
findings”
Knowledge is
uncertain
Purpose:
• Increase confidence in findings
• Reaffirm findings
(Casadevall and Fang, 2016, p1)
• Update boundaries on external validity
• Identify type 1 errors
The problem with Sports Science
Focus on novel or
exciting effects
Questionable
research practices
Invested interests
Low statistical
power & small
sample sizes
Lack of
transparency
Change is coming
• STORK
• Registered
Reports
• Preregistration
• Data sharing
The Sports Science Replication Centre
Aim
• Evaluate the replicability of sports science research
Objectives
• Establish a collaborative lab network worldwide
• Undertake replication trials of different effects
• Report replication outcomes using meta-analytic procedures
• Present an initial overview of the replicability of sports science
research
Large replication projects
Large
replication
projects
Multiple
Single
replications –
replications –
single effects
multiple effects
Reproducibility
Sports Science
Replication
Centre
Many Labs
Project
Psychology
Timeline (short term)
Summer
2021
Cont.
2022
May
2021
Current
Lab
recruitment
begins
Replication
Selection
Protocol
study
prereg
Replication
study
allocation
Data
collection
phase 1
Survey
Oct
2021
Summer
2021
Current
Lab recruitment
Who?
Where?
Sports and exercise
science researchers
Worldwide!
Suggestions for conducting a replication study:
Conduct it
yourself
Undergrad
project
Research
internship
Postgrad
project
Why become involved?
\1. Contribute towards the first
collaborative replication project
in sports and exercise science
\4. Write up and final analysis
will be completed by the team
leader
\2. Lead the way towardsopen
science in the field
\5. Each lab will receive a study
that is feasible to conduct
\3. All methods and analysis are
preplanned by the team leader
\6.
Each author who makes a
substantial contribution will
receive authorship
Selection Protocol
Year of publication
and journal ranking
• Previous 5 years
• Q1 journals
• Applied sport and
Research disciplines
exercise science
Selection Protocol
• Experimental studies
o Effect of IV on DV
• Means, effect size and
confidence intervals
Study types
• One dependent
variable
• Stated in first
hypothesis or abstract
Research question
and key variable
Selection Protocol
• Sample details
• Instrumentation details
• Boundary conditions
Study methods
Feasibility
• Data collection period
• Statistical analyses
Allocation to lab
Conducting the replication
Statistical Power
1
2
3
Data
sharing
Follow
sample
Local
ethics
approval
• High powered replication
agreement
guidelines
studies (95%)
• Correction for publication
bias
• Correction for effect size
4
5
6
Follow
uncertainty
Send raw
data back
Collect
data
procedural
guidelines
• Anderson and Kelley’s R
package “BUCSS” (2017)
Assessing the outcomes
Multiple inferential strategy
Descriptive information
\1. Replication effect size is
statistically significant
\1. Normalized differences in
the magnitude of the
effect sizes
\2. Same direction as
original effect size
\2. Included and excluded
studies as per selection
protocol
\3. Original effect size falls
into 95% confidence
interval of replication
effect size
\3. Original author response
rates
Reporting the outcome
Successful replication
Original effect is false
Replication is false
Methods differed
Error
Failure to replicate
What this project is not
A personal attack on
researchers in the
field
The final evaluation of
replicabilty
Statistically
significant &
non-
Published &
non-published
research
significant
effects
Future
Focus on specific
Popular effects
Self-replications
sub-disciplines
Personal
interests
Highly cited
effects
Conceptual
replications
Considerations and barriers
Poor
reporting of
results
Response
rates
Unclear
methods
Funding
Conclusion
Help us - volunteer labs needed!
Contact us:
https://ssreplicationcentre.com
Email: x00170612@mytudublin.ie
@JennyMurphy2
Thank you!
https://ssreplicationcentre.com
References
Abt, G., Boreham, C., Davison, G., Jackson, R., Nevill, A., Wallace, E., Williams, M., 2020. Power, precision, and sample
size estimation in sport and exercise science research. https://doi.org/10.1080/02640414.2020.1776002
Brandt, M.J., IJzerman, H., Dijksterhuis, A., Farach, F.J., Geller, J., Giner-Sorolla, R., Grange, J.A., Perugini, M., Spies, J.R.,
van ’t Veer, A., 2014. The Replication Recipe: What makes for a convincing replication? J. Exp. Soc. Psychol. 50, 217–
\224. https://doi.org/10.1016/j.jesp.2013.10.005
Caldwell, A.R., Vigotsky, A.D., Tenan, M.S., Radel, R., Mellor, D.T., Kreutzer, A., Lahart, I.M., Mills, J.P., Boisgontier, M.P.,
Boardley, I., Bouza, B., Cheval, B., Chow, Z.R., Contreras, B., Dieter, B., Halperin, I., Haun, C., Knudson, D., Lahti, J., Miller,
M., Morin, J.B., Naughton, M., Neva, J., Nuckols, G., Peters, S., Roberts, B., Rosa-Caldwell, M., Schmidt, J., Schoenfeld, B.J.,
Severin, R., Skarabot, J., Steele, J., Twomey, R., Zenko, Z., Lohse, K.R., Nunan, D., 2020. Moving Sport and Exercise
Science Forward: A Call for the Adoption of More Transparent Research Practices. Sport. Med. 50, 449–459.
https://doi.org/10.1007/s40279-019-01227-1
Casadevall, A., Fang, F.C., 2016. Rigorous science: A how-to guide. MBio. https://doi.org/10.1128/mBio.01902-16
Collaboration, O.S., 2015. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349, aac4716.
https://doi.org/10.1126/science.aac4716
Halperin, I., Vigotsky, A.D., Foster, C., Pyne, D.B., 2018. Strengthening the practice of exercise and sport-science
research. Int. J. Sports Physiol. Perform. 13, 127–134. https://doi.org/10.1123/ijspp.2017-0322
References
Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B., Bahník, Š., Bernstein, M.J., Bocian, K., Brandt, M.J., Brooks, B.,
Brumbaugh, C.C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W.E., Devos, T., Eisner, M., Frankowska, N., Furrow, D.,
Galliani, E.M., Hasselman, F., Hicks, J.A., Hovermale, J.F., Hunt, S.J., Huntsinger, J.R., Ijzerman, H., John, M.-S.S., Joy-Gaba,
J.A., Kappes, H.B., Krueger, L.E., Kurtz, J., Levitan, C.A., Mallett, R.K., Morris, W.L., Nelson, A.J., Nier, J.A., Packard, G., Pilati,
R., Rutchick, A.M., Schmidt, K., Skorinko, J.L., Smith, R., Steiner, T.G., Storbeck, J., Van Swol, L.M., Thompson, D., Van ’T
Veer, A.E., Vaughn, L.A., Vranka, M., Wichman, A.L., Woodzicka, J.A., Nosek, B.A., 2014. Investigating variation in
replicability: A “many labs” replication project. Soc. Psychol. (Gott). 45, 142–152. https://doi.org/10.1027/1864-
9335/a000178
LeBel, E.P., Vanpaemel, W., Cheung, I., Campbell, L., 2019. A Brief Guide to Evaluate Replications. Meta-Psychology 3.
https://doi.org/10.15626/mp.2018.843
Nosek, B.A., Errington, T.M., 2020. What is replication? PLoS Biol. 18, 1–8.
https://doi.org/10.1371/journal.pbio.3000691
Sainani, K.L., Borg, D.N., Caldwell, A.R., Butson, M.L., Tenan, M.S., Vickers, A.J., Vigotsky, A.D., Warmenhoven, J., Nguyen,
R., Lohse, K.R., Knight, E.J., Bargary, N., 2020. Call to increase statistical collaboration in sports science, sport and
exercise medicine and sports physiotherapy. Br. J. Sports Med. 0. https://doi.org/10.1136/bjsports-2020-102607
Stroebe, W., 2019. What Can We Learn from Many Labs Replications? Basic Appl. Soc. Psych. 41, 91–103.