How can you tell whether a measure is reliable and valid?

For: Readers encountering questionnaires, behavioural scores or performance tasks.

A number acquires meaning through the procedure that produced it. A “concentration score” might count missed targets, average reaction times, or add responses to a questionnaire. These are observations with different demands. Before asking whether the score is good, describe the task, scoring rule, time window and intended interpretation. The opening video supplies a vocabulary for categories, ordered values and numerical measurements; it does not itself establish a psychological test’s validity.

Reliability concerns consistency, but the relevant consistency depends on the task. Agreement between observers matters when behaviour must be coded. Agreement across items matters when several questions contribute to a score. Stability over time matters when the underlying characteristic is expected to remain stable. A changing mood score over a month need not signal measurement failure. The measurement chapter explains these distinctions and why reliability alone cannot establish that a score measures the intended construct.

Validity requires evidence for the interpretation. Do the observations cover the intended characteristic? Do scores relate to other measures in ways the theory predicts, while remaining distinguishable from different constructs? A familiar label or a plausible-looking question is insufficient. Treat the evidence as an argument to inspect rather than a single number that makes a measure permanently valid. The same chapter gives accessible introductions to content, criterion, convergent and discriminant evidence. Reliability and Validity of Measurement.

Use an original thought experiment to make the distinction concrete. Suppose an app counts how often someone unlocks a phone and calls the total “distractibility”. The count might be captured very consistently. Yet a high count could also reflect work duties, navigation or messages from family. Write three interpretations compatible with the same count. Then propose one additional observation that would help separate them. This is an argument about what would be informative, not a claim that any particular app has been evaluated.

The statistical videos help inspect the score’s visible behaviour. A mean can conceal unusual values; a median and a spread measure reveal different features. Percentile rank answers where someone stands in a reference distribution, not how much of a trait they possess. A scatterplot can reveal a pattern that a lone correlation hides. Missing observations need an explicit rule: a software option for omitting them does not explain why they were missing. These checks support scrutiny of a measure; they cannot replace validation studies. Descriptive Statistics with R.

Finish by writing a score interpretation that you would be willing to defend. Replace “this measures concentration” with a precise statement about the observations, setting and intended use. List the evidence available and the evidence still needed. If the available material only describes the scoring procedure, leave reliability and validity unresolved rather than filling the gap with an impressive-looking coefficient.

Next step

Choose one published measure and locate its actual reliability and validity evidence before comparing group averages.

Limits of this route

The video descriptions were checked against identified official-transcript excerpts, not complete viewing. An accessible official media endpoint is not evidence that every frame played. Teaching examples are not independent validation of the underlying scientific or clinical claims. The statistical lectures support inspection of data and scoring; the conceptual reliability/validity claims come from the linked reading. No diagnostic instrument is recommended.