What can a confidence interval or hypothesis test tell you?

For: Readers who encounter p-values and confidence intervals but want to interpret them rather than memorise a threshold.

A study reports one set of observations, but a statistical procedure describes how results could vary under a model of repeated sampling. Start there. Variation between people is not the same as uncertainty about their average. The sampling-distribution lectures make that distinction visible and explain why the assumptions behind an approximation matter. A larger sample can improve precision without repairing an inappropriate measure or an unrepresentative comparison. NPTEL hypothesis-testing course.

A p-value is calculated under a specified null hypothesis and the other assumptions of the test. It describes how unusual the observed test statistic, or something more extreme in the specified direction or directions, would be under that model. It is not the probability that the null is true or that the result was “just chance”. The ASA’s six principles also distinguish statistical significance from effect size and practical importance. Interpretation needs the design, the analysis choices and the size of the estimated effect.

A confidence interval contributes a different view of uncertainty. In the frequentist construction discussed here, the confidence level belongs to the method: under its assumptions, repeated samples would produce intervals covering the fixed population quantity at the stated long-run rate. The realised interval either covers that quantity or does not. It is not a range containing a stated percentage of the individual observations. The NIST handbook explanation connects this interpretation to the uncertainty of a mean and to a corresponding test.

Consider an invented comparison of two learning methods. The estimated difference is two recalled words. An interval from minus one to five words leaves a different practical uncertainty from an interval from 1.8 to 2.2 words. Ask what size of improvement would matter before interpreting either result. The first example allows both a modest disadvantage and a useful benefit; its inclusion of zero does not demonstrate equivalence. The second is precise under its assumed model, but precision alone does not establish a meaningful learning gain or a causal interpretation.

Power adds another conditional question: how often would a particular test detect a particular alternative? Its inputs include effect size, variability, sample size and the test rule. The power lectures are useful for planning and for recognising why weak evidence against a null is not automatically strong evidence for it. They do not turn a completed study’s significant result into a probability of being correct.

For a reading exercise, replace “the result was significant” with three sentences. State the estimated effect in its original units, describe the uncertainty, and identify one design or modelling assumption needed for the interpretation. Then write a separate sentence about practical importance. If the paper gives only a threshold label, record what cannot be recovered from that label. A good interpretation makes the unanswered questions visible.

Next step

Read a paper’s results alongside its methods and ask whether its interval, test and scientific claim address the same quantity.

Limits of this route

The video descriptions were checked against identified official-transcript excerpts, not complete viewing. An accessible official media endpoint is not evidence that every frame played. Teaching examples are not independent validation of the underlying scientific or clinical claims. This guide concerns common frequentist interpretations; it is not a complete treatment of Bayesian inference, multiplicity or equivalence testing. The numerical intervals are invented teaching examples.