What you will be able to do

The course covers the full research process, from an initial idea to a published, defensible conclusion, with the emphasis on the decisions that determine whether a finding will hold up. Each principle is illustrated with published sport and exercise science examples rather than invented ones.

By the end you will be able to: state a claim precisely enough to be tested and derive a directional hypothesis from it; define constructs and choose valid, reliable measures; size a study so it can actually answer its question; preregister and analyse a study without inflating the error rate; report and interpret results in proportion to the evidence; and appraise published work critically, including your own.

Three parts, thirteen weeks

The weeks are ordered to follow a single study from question to conclusion, and a running example develops across them. Select any week to jump to it.

Four components per week

Getting the most from the course

1. Work through one week at a time, in order. Each week builds on the previous one, and a single running example develops across the course.
2. Read or listen — the content is the same. Many people do both: read the module, then listen to the episode to consolidate it.
3. Do the task before moving on. The course is designed around applying each idea to a study you care about; the tasks accumulate into a plan for a well-designed study.
4. Read the Current State review to see how the principle plays out across the published field, not only in theory.
5. It is self-paced. Your current week and your task answers are stored in your browser, so you can stop and resume at any point.

Every empirical claim in the course is cited; illustrative figures are flagged as such. The course is free and produced for the Sports Science Replication Centre.

A method for detecting error

Ask what science is and the common answer is “a body of knowledge” — facts about oxygen uptake, muscle protein synthesis, training load. Science produces such facts, but it is not the facts. Science is the process: a disciplined way of finding things out that is unusually good at one thing in particular — catching its own errors.¹

Every method we have of learning about athletes is fallible. Measurements carry error; samples are small and unrepresentative; reasoning is biased; chance plays tricks. Science does not remove these problems; it uses systematic methods to detect them: controls, comparison groups, statistics, replication, and transparency. It does not certify claims as true — it provides procedures for identifying error.

Scientific progress consists of ruling out more of the ways a result could be mistaken, rather than proving a claim true.

Take a concrete claim: a new recovery device improves next-day sprint performance. The unscientific way to believe it is to try it, feel fresher, and sprint well. The scientific way is a sequence of error-catching questions — fresher compared to what? Would the athlete have sprinted well anyway? Did they know they had the device, so expectation alone could explain it? Is one good session simply noise? Each question tests for a specific error before the claim is accepted.

Evidence, and belief

The distinction is not about how certain a claim feels — beliefs often feel more certain than evidence does. It is about where the confidence comes from, and whether it could have been shown wrong. A belief is a claim you hold from experience, tradition, or a plausible mechanism; it may even be right, but it has not faced a fair test that could have falsified it. Evidence is what a claim earns by surviving exactly that kind of test.

This is humbling for practitioners, because experience is a superb generator of hypotheses and an unreliable judge of them.² In the field you never see the control condition; you never re-run the same week without the intervention. You remember the athlete who took the supplement and broke through, and forget the three who took it and did not. So grade any claim by how much it has actually earned your trust:

A frequent error in applied practice is acting on the lower levels of evidence as though they were the higher ones. Matching the strength of a claim to the strength of its evidence is the central skill.

“One study showed”

A single study is one data point measured with error, not a final result. Reviews of the sport and exercise science literature document small samples, an over-representation of statistically significant “positive” results, and low rates of data-sharing — indicators associated with publication bias and low replicability.³ Consider a well-studied real case — dietary nitrate, from beetroot juice.

When a striking result appears — a small sample, a large effect, a strong conclusion — the appropriate response is to evaluate it: how large was the sample; was there a control condition; how large was the effect, and does that size matter in practice; has it been replicated? Knowledge accumulates across many studies, replications, and meta-analyses rather than resting on any single study.

Your calibration exercise

For one week, replace “a study showed” with “one study suggested,” and notice how it changes your thinking.

The Current State review can also be read online.

Clear enough to be wrong

A claim that is precise enough to be shown false is scientifically useful; a vague concept that is compatible with any result is not. When the concepts are defined precisely, everything downstream — question, hypothesis, measure — becomes clearer. When they are left vague, no amount of later methodological rigour can correct the study.

Compare two versions of the same idea. “Managing training load optimises adaptation” forbids nothing — whatever happens, it survives; it is a slogan. “In these athletes, weekly load increases above about 10% are associated with more injuries the following week” specifies who, what, how much, and when — a study could refute it.¹

The jingle & jangle fallacies

Two naming errors undermine comparisons across studies. The jingle fallacy: assuming one word means one thing. “Fatigue” covers peripheral muscle-fibre fatigue, perceived fatigue, and chronic over-reaching — Enoka & Duchateau (2016) split it into performance fatigability (objective decline) and perceived fatigability (the sensations).² The jangle fallacy is the mirror image: assuming different words mean different things — “wellness”, “readiness”, and “recovery” may index much the same thing.

“Training load” is the standing example in our field: external load (the work done — distance, speed, weight) versus internal load (the biological response — heart rate, RPE). Impellizzeri, Marcora & Coutts (2019) re-published to re-clarify their own terms — a model of a field doing conceptual clarity out loud.

Description is not explanation

A pattern is not a cause. Suppose athletes with lower “wellness” scores tend to have poorer sessions. Before that observation licenses an intervention, rule out the alternatives — because only one of them means that fixing wellness would help.

Description requires observation; causation requires intervention and comparison. A descriptive finding should not be treated as a causal recommendation.

Sharpen one of your concepts

The Current State review can also be read online.

Descriptive or inferential?

A good research question fixes who you study, what you compare, which outcome counts, and what kind of question it is. The first fork is the one most often skipped. A descriptive question asks “what is the case?” — variation, typical error, normal ranges — and needs only careful observation. An inferential question asks “does X relate to, or affect, Y?” — a comparison that generalises beyond the sample, and needs a comparison group, ideally an intervention.

The classic error is to run a descriptive, observational study and then claim an inferential or causal effect it was never built to detect. Match the question to the design from the very start, because the design you can afford may decide which question is even answerable.¹

Confirmatory vs exploratory

Both modes are legitimate and essential.³ A confirmatory test states a prediction in advance and looks at it once. An exploratory analysis searches the data for patterns and generates hypotheses; it is a primary source of new ideas. The problem is not exploring — it is reporting exploratory work as if it had been confirmatory.

Correlate fifteen monitoring variables — sleep, HRV, load, jump height — against match performance and you will find “significant” hits by chance alone. That is fine as exploration. Writing “we hypothesised that HRV predicts performance, and confirmed it” is the error: a chance pattern presented as a prediction. It appears more trustworthy than it is, and it does not replicate.

Build your question

The Current State review can also be read online.

A good bet is specific

A good hypothesis is falsifiable, directional, tied to a theory, and specific about how big an effect would matter — and it is written before you see the data. Each property does real work, and dropping any one weakens the test.

The null hypothesis — usually “no effect” — is a reference point, not a belief. The test asks how surprising the observed data would be if there were no effect. Understanding it as a reference point, rather than a claim to argue against, is necessary for interpreting statistics correctly (Week 8). One qualification: an exactly-zero effect is rarely literally true, which is why the magnitude of the effect matters.

HARKing

HARKing — Hypothesizing After the Results are Known — is presenting a hypothesis you formed after seeing the data as if you'd predicted it all along.¹ The honest path: predict “HRV → performance”, collect data, test that one prediction once, report at face value. The HARKed path: collect fifteen variables, correlate them all against performance, find the one chance “hit”, and write the introduction as if that was the prediction — hiding the fourteen you ignored.

Why it is so damaging: the statistics assume one prediction tested once. HARKing makes a lucky pattern look like a survived prediction, so it appears far more trustworthy than it is — and it doesn't replicate.

What else could explain a “yes”?

Before you run it, list the rival explanations for the result you hope to see — and how the design rules each out. If sprinters improve, it could be a training effect over four weeks (ruled out by a comparison group doing the same training), placebo or expectation (ruled out by blinding), or regression to the mean from picking slow starters (ruled out by randomisation and not selecting on baseline). A hypothesis you can defend against its alternatives is one worth testing.

Write it so specifically that a stranger could see exactly what would prove you wrong.

The Current State review can also be read online.

Validity and reliability are two questions

Every measure faces two separate tests. Validity: does it measure the construct of interest? Reliability: does it give the same value on repeated measurement? They are independent — a measure can be highly reliable and still invalid, consistently measuring the wrong thing. Measurement precedes analysis: if the measure is poor, no analysis can recover valid conclusions from the data.

A common error is to treat reliability as sufficient. A device that returns a consistent number can appear trustworthy even when it has not been shown to measure the intended construct. Check the two properties separately, and be most cautious where a measure is most consistent.¹

What you cannot see, you cannot study

Every measure has a typical error — the random scatter in an individual's repeated trials, often expressed as a coefficient of variation.² It sets a noise floor: a signal smaller than the noise is invisible, however real. If your test's typical error is 3% and you're chasing a 1% effect, you cannot see it — the study is incapable of answering its own question. So a change has to exceed the typical error before it can be believed, and reliability directly drives the sample size you'll need (Week 7).

A measure whose typical error exceeds the effect being studied can make an otherwise well-designed study incapable of answering its question.

The conclusion can't outrun the method

The strength of causal claim you can make is capped by your design. An observational study watches without intervening — strong for description and hypothesis-generating, rarely for causation (confounding, reverse causation). A quasi-experiment intervenes without full randomisation — causal-ish, but vulnerable to whatever you didn't control. A randomised controlled trial intervenes and randomises, balancing measured and unmeasured factors — the strongest footing for causation. Don't let the conclusion outrun the design (Week 11).

The Current State review can also be read online.

Researcher degrees of freedom

In any real analysis you face dozens of defensible choices: which outliers to drop, which outcome is primary, which covariates to include, when to stop collecting. Made after looking at the data, each in the direction of significance, these choices inflate the false-positive rate.

The scale is startling. With a handful of such flexibilities, the chance of finding a “significant” result when nothing is real climbs from the nominal 5% to well over 60%.¹ This is not fraud — it is flexibility plus a peek, and it feels like diligence the whole way through.

The checklist, and the 21-word solution

A useful preregistration fixes the hypothesis (directional, from theory), the primary outcome (the one that decides it), the design and comparison, the sample size and its justification (Week 7), the exact analysis plan, the exclusion and stopping rules, and — crucially — a clear split between confirmatory and exploratory analyses.

Simmons and colleagues' “21-word solution” is a minimal public commitment: state that you report how the sample size was determined, and all data exclusions, conditions, and measures. It fits in a footnote and closes most of the gaps.

Registered Reports

A Registered Report takes preregistration one step further.² At Stage 1, before data, you submit the introduction, hypotheses and full method; reviewers judge the question and design, and a pass earns in-principle acceptance. At Stage 2 you execute the plan and the paper is published regardless of the results. A well-designed null publishes with equal status — which decapitates publication bias (Week 1) and removes the incentive to p-hack. A growing number of sport journals now offer them.

And the common objection — doesn't this kill exploration? No. Preregistration protects exploration by naming it. Explore as freely as you like; just label confirmatory (predicted, tested once) versus exploratory (discovered — a hypothesis for next time). You lose only the freedom to misrepresent a discovery as a prediction.

The Current State review can also be read online.

Two errors, and power

Two errors apply to every test. A Type I error is a false positive — claiming an effect when there is none. A Type II error is a false negative — failing to detect a real effect. Power is one minus the Type II rate: the probability of detecting a real effect if it is present.

Numbers make it concrete. A modest true effect studied in about a dozen athletes often leaves power at or below 30% — the study is more likely to miss a real effect than to find it. A non-significant result from such a study then teaches you almost nothing: absence of evidence is not evidence of absence.

The winner's curse

The mechanism is counter-intuitive.¹ In a small, noisy study, an estimate has to be large to reach significance at all. If the true effect is modest, the only way a small study crosses the line is when random noise happens to push the estimate way up. So the significant results that small studies produce are systematically inflated — often by two- to threefold — and occasionally point the wrong way entirely.

This is the mechanism behind the “decline effect”: a small exciting study reports a huge effect, gets published because it is huge, and bigger replications find it much smaller or absent. It didn't decline — the original was inflated by low power.

A priori power, and a SESOI

Do the power analysis a priori: given the effect you'd care to detect and your error rates, compute the N you need, and commit to it in your preregistration. Avoid post-hoc or “observed” power — computed from the effect you just saw, it is circular, merely a re-expression of the p-value, and always low after a null, so it can't excuse one. If you want to interpret a completed study, run a sensitivity analysis instead: given the N you had, what size of effect could you reliably have detected?

The maths needs an effect size, and the right one is your smallest effect of interest — the smallest change that would matter to an athlete — not the (probably inflated) published effect, and not “whatever is significant.”² That turns the question from “is there any effect?” into “is there an effect big enough to matter?”

Pool, pilot, or acknowledge

Three honest routes when the numbers won't come. Pool: a multi-site collaboration — twenty labs at twelve athletes each is 240 no one could recruit alone. The SSRC's Meta-Student initiative is built on exactly this model, pooling standardised data collected by students across many institutions to reach sample sizes no single lab could achieve. Pilot: reframe the study as an effect-size estimate to plan a larger one — and report it that way. Acknowledge: report the power you had, a sensitivity analysis, and the winner's-curse risk on any significant finding.

The Current State review can also be read online.

One sentence, held precisely

A p-value is the probability of data at least as extreme as what you observed, if the null hypothesis — usually “no effect” — were true. It is a statement about your data under an assumption, and that is all. Nearly every abuse comes from asking it to be something bigger.¹

Four misreadings do most of the harm. It is not the probability that the null is true (it is computed assuming the null, so it cannot tell you the probability of that assumption). It is not the probability the result is “due to chance.” One minus p is not the probability the effect is real or will replicate — p-values swing wildly from study to study. And a small p does not mean a big or important effect: with a large sample, a trivial effect is highly significant.

“p < .05” does not mean “95% likely to be real.” It means: if there were truly no effect, data like ours would occur less than 5% of the time. That's all.

Estimation is richer and more honest

Reporting an estimate with its interval says everything the dichotomy hides.² Compare: “the improvement was significant (p = .03)” stops there; “the improvement was about 4 s, 95% CI 1 to 7 s” gives direction, a likely size, and its uncertainty — it could be as small as 1 s (trivial?) or as large as 7 s (a real edge), and that honest “we don't yet know which” is the finding.

Procedural control of the error rate

One test carries about a 5% false-positive rate. Twenty tests, and you should expect roughly one “significant” result by chance even if nothing is real — the mechanism underlying researcher degrees of freedom (Week 6). The more outcomes, subgroups and time-points you test, the more your true error rate inflates. The protection is procedural, not a magic correction: run your preregistered confirmatory analyses first, report them as primary with the error rate intact; then conduct further analyses, labelling them exploratory. This preserves the error rate for the confirmatory test while still allowing exploration.

Practice on a paper: cover the p-value, look only at the effect size and CI, and ask “how big, how sure?” You'll often find the interval so wide it rules little out.

The Current State review can also be read online.

Two words often confused

Reproducibility and replicability are not the same thing. Reproducibility is computational integrity: the same data plus the same analysis should give the same numbers. Replicability (Week 12) is a new study, new participants, testing whether the effect holds at all. Sharing is what makes the first possible and the second interpretable.¹

The uncomfortable finding is that many results aren't even reproducible: undocumented processing, lost files and quiet errors mean people often can't re-run the author's own analysis on the author's own data. That is a problem before replication even begins.

As open as possible, as closed as necessary

Sport data can be sensitive, so share responsibly rather than not at all. De-identify; where raw data can't be open, share a subset, a summary, the code and materials, or offer controlled access. The principle — “as open as possible, as closed as necessary” — asks you to say why for anything you restrict, and not to use privacy as a blanket excuse to share nothing. A reproducible workflow helps here too: keep raw data untouched, do all cleaning in a script rather than by editing cells, organise the project folder, and comment the logic. It benefits you first — a reviewer's query or an extension becomes minutes of work, not a week of archaeology.

The Current State review can also be read online.

Completeness and calibration

Good reporting is two habits: report everything you found, and describe it in language calibrated to the evidence. Selectivity — spotlight the winners, bury the nulls, reach for stronger words than the data support — each feels like telling “a clear story,” and collectively tilts the published record toward effects that are bigger and surer than reality.

Start with completeness: report every preregistered outcome, including the nulls. This is the antidote to selective outcome reporting. If you tested five things and report only the one that “worked,” it is probably chance — but the reader can't tell. Preregistration lets a reader check your paper against your plan, and lets you present a planned null as the real, publishable finding it is.

Checklists, effect sizes, and the abstract test

Reporting-guideline checklists catch what memory misses and make selectivity harder. For a randomised controlled trial, work the CONSORT checklist as you write; for observational designs, STROBE.¹ Make effect sizes with confidence intervals — interpreted against your SESOI — the centrepiece, not bare p-values, and report nulls fully and unapologetically; each is a small correction to publication bias (Week 1).

Read your abstract alone: if this were all someone saw, would they believe something stronger than your data support? If yes, remove the spin.

The Current State review can also be read online.

Calibrate the claim to the evidence

Everything so far protects the study; this week protects the sentence people actually repeat. A finding can be clean and still be mis-stated the moment it leaves the results section — usually by growing a little beyond what was shown. Four disciplining questions keep the claim the right size.

That number sets the default attitude to any single fresh finding, including your own: humility. On the field's own evidence, a lone result is probably an over-estimate.¹

A causal claim licenses an action

The fourth question has direct practical consequences. A causal claim licenses an intervention — “do X to obtain Y.” If the causal relationship does not hold, coaches and athletes may spend training time and resources on an effect that was only an association. A causal verb should not be attached to non-causal evidence. A claim stated within its evidential limits is more accurate, and more useful, than one that overstates the finding.

The Current State review can also be read online.

Direct and conceptual

A direct (or close) replication repeats a study as faithfully as possible with new participants — it tests whether the effect is real or a chance result. A conceptual replication tests the same hypothesis with different methods or populations — it tests whether the effect generalises. Both are needed, and sport and exercise science has conducted few of either.

They fail differently, which matters for interpretation. A failed conceptual replication is ambiguous — the effect might be false, or your method simply didn't capture it. A failed direct replication points more cleanly at fragility in the original. Reach for direct replications when the question is “can I trust this at all?”

A meta-analysis is only as good as its inputs

A meta-analysis pools many studies into one precise estimate — ten underpowered studies can reach the power of one large one. But it is only as trustworthy as its inputs: a biased literature (publication bias, selective reporting, p-hacking, the winner's curse) yields a precise estimate of a biased value, expressed as a narrow confidence interval. This is why preregistration, complete reporting, transparency and adequate power matter — they maintain the quality of the literature. The more durable fixes are prospective: Registered Reports and multi-site studies — including the SSRC's Meta-Student initiative, in which students across many institutions collect and pool standardised data, building the large, replicated, combinable evidence base that cumulative science depends on.

Chalmers and Glasziou estimate that around 85% of research investment is wasted. Because much of this waste is avoidable, the practices covered in this course can yield substantial improvements. Each stage of the course corresponds to avoiding a specific form of waste.

The Current State review can also be read online.

Science is done by people in a system

Every practice in this course is known and freely available, yet the field-wide numbers are stubborn. The reason is not ignorance — it is incentives. Science is done by people whose jobs, grants and reputations depend on a reward system, and if that system rewards the wrong things, even well-informed, well-intentioned researchers get pulled toward worse practice.

This is a structural observation, not a moral one: the individually rational career move and the collectively beneficial scientific practice can point in different directions, and the effective response is to change the incentive structure rather than to blame individuals.

Leverage at every level

The cycle is broken by many people each doing one more thing right — and the leverage differs by role.

These changes are measurable. Over roughly a decade, preregistration has moved from uncommon to widely used; Registered Reports are offered at hundreds of journals; open data is increasingly expected; sport and exercise science has published transparency guidelines and conducted replication projects; and metascience — the empirical study of research methods — has developed as a field. These are documented changes in research practice.

The common thread

Each method in this course addresses a specific, identifiable way that research can produce mistaken conclusions.

Calibration of confidence, clear concepts, falsifiable pre-specified hypotheses, valid and reliable measures, adequate power, pre-specified analysis, complete reporting, appropriately bounded claims, transparency, and cumulative interpretation each address a particular source of error. Beyond the methods, sustained improvement also depends on incentives and norms that reward accuracy rather than publication volume.

Research culture changes through many individual changes in practice. — End of the course.

The Current State review can also be read online.