Hypothesis Testing

Mathematical Foundations and Hypothesis Construction

Hypothesis testing is a formal decision procedure in which observed data are compared with the probability distribution expected under a specified model. For Primary FRCA purposes, the key is not merely that a p value is small, but that the rejection rule has been constructed in advance to control the long-run probability of a false positive conclusion.

Null and alternative hypotheses

Let θ denote the parameter of clinical interest: a mean arterial pressure difference, a relative risk, a hazard ratio, an ED₅₀, or a regression coefficient. The null hypothesis, H₀, defines the parameter values compatible with no effect, no difference, or a prespecified clinically acceptable boundary. The alternative hypothesis, H₁, defines the parameter values for which the investigator wishes to claim evidence.

Mathematically, H₀ and H₁ partition, or partially partition, the parameter space Θ. A point null might be H₀: θ = 0 for a mean difference, or H₀: risk ratio = 1. A composite null might be H₀: θ ≤ 0, used when only benefit in one direction is considered. In a non-inferiority trial, the null is reversed conceptually: H₀: θ ≤ −Δ, where Δ is the non-inferiority margin, and rejection supports that the new treatment is not worse than control by more than Δ.

Clinical question	Parameter θ	Typical H0	Typical H1
Does drug A reduce postoperative morphine use?	Mean difference, A − control	θ = 0	θ ≠ 0 or θ < 0
Is a supraglottic airway non-inferior to tracheal intubation for first-pass success?	Difference in success proportions	θ ≤ −Δ	θ > −Δ
Does tranexamic acid reduce mortality?	Risk ratio or hazard ratio	θ = 1	θ ≠ 1, usually two-sided

Probability density under H₀

A test statistic T is a function of the data, chosen so that its sampling distribution is known, or asymptotically known, under H₀. If comparing two independent means with known or large-sample variance, a standardised statistic may be written as Z = (observed estimate − null value) / standard error. Under H₀, Z approximately follows a standard normal distribution with probability density function f₀(z) = 1/√(2π) exp(−z²/2). The p value is the probability, calculated under f₀, of observing a statistic at least as extreme as that obtained.

For a two-sided normal test, p = P(|Z| ≥ |z_obs| | H₀) = 2[1 − Φ(|z_obs|)]. For a one-sided upper-tail test, p = P(Z ≥ z_obs | H₀) = 1 − Φ(z_obs). The rejection region R must be defined so that P(T ∈ R | H₀) ≤ α, where α is the Type I error rate. Conventionally, α = 0.05 in many biomedical studies, although confirmatory regulatory trials commonly use two-sided 5% testing or one-sided 2.5% testing for non-inferiority.

One-tailed versus two-tailed tests

A two-tailed test allocates α across both extremes of the null distribution. With α = 0.05 and Z ∼ N(0,1), rejection occurs when |Z| > 1.96, leaving 0.025 in each tail. A one-tailed upper test at α = 0.05 rejects when Z > 1.645. This lower critical value increases power for detecting effects in the prespecified direction, but it provides no formal evidence against H₀ for an effect in the opposite direction.

Design	Rejection rule for Z	Total α under H0	Interpretation
Two-sided, α = 0.05	|Z| > 1.96	0.05	Detects either benefit or harm
One-sided upper, α = 0.05	Z > 1.645	0.05	Detects only upper-direction effect
Post hoc choice of favourable one-sided tail	|Z| > 1.645	0.10	Invalid Type I error inflation

The mathematical danger is tail switching. If investigators inspect the data and then choose the favourable one-sided direction, the effective rejection region becomes both tails beyond ±1.645. Under H₀, P(|Z| > 1.645) = 0.10, doubling the false positive risk from 5% to 10%. Thus, a one-tailed test is not a stylistic reporting choice; it is a design feature that must be specified before enrolment, analysis, unblinding, or interim review.

Justification in pharmacology and clinical trials

In drug and device evaluation, two-sided testing is the default because unexpected harm is clinically important. For example, an anaesthetic adjunct intended to reduce opioid consumption may instead increase sedation, hypoxaemia, or delayed discharge; a two-sided framework preserves the capacity to detect clinically material effects in either direction. CONSORT guidance and ICH E9(R1) emphasise prespecification of estimands, hypotheses, analysis populations, and error control.

One-sided testing is defensible only when effects in the opposite direction would be scientifically irrelevant or when the null is intrinsically directional, as in non-inferiority. Regulatory non-inferiority trials typically use a one-sided α of 0.025, corresponding to demonstrating that the lower bound of a two-sided 95% confidence interval lies above −Δ. The margin Δ must be clinically and historically justified, not chosen for convenience; for mortality or major morbidity endpoints it is often small, whereas for surrogate endpoints it may be larger but less persuasive. In pharmacological dose-response studies, a directional trend test may be justified only if monotonicity is biologically plausible, doses are prespecified, and safety is separately analysed. For FRCA examinations, the safest principle is: unless the protocol can justify ignoring the opposite tail before seeing the data, use a two-tailed test.

Type I and II Errors, Power, and Sample Size Calculation

Hypothesis testing is a formal decision procedure conducted under uncertainty. In the Neyman-Pearson framework, one pre-specifies a null hypothesis, an alternative hypothesis, a test statistic, a rejection region, and acceptable long-run error rates. The result is not a proof of truth or falsehood, but a controlled probabilistic decision: reject or fail to reject the null hypothesis.

The decision matrix

True state of nature	Decision: reject H0	Decision: fail to reject H0
H0 true	Type I error: false positive; probability = α	Correct non-rejection; probability = 1 − α
H1 true	Correct rejection; probability = 1 − β, the power	Type II error: false negative; probability = β

A Type I error is concluding that an effect exists when, in truth, the null hypothesis is correct. The conventional α threshold is 0.05, meaning a 5% long-run probability of falsely rejecting a true null hypothesis. For a two-sided test at α = 0.05, the critical standard normal values are ±1.96; for α = 0.01 they are ±2.576. In regulatory and trial contexts, α is often controlled more stringently: for example, many non-inferiority trials use a one-sided α of 0.025, equivalent to preserving a two-sided 5% error structure.

A Type II error is failing to detect a real effect of a specified magnitude. β is commonly set at 0.20 or 0.10, corresponding to 80% or 90% power respectively. Importantly, β is not a fixed property of the statistical test alone: it depends on the true effect size, variance, sample size, allocation ratio, significance threshold, and whether the test is one- or two-sided.

Statistical power: rigorous definition

Power is the probability of rejecting H₀ when a specified alternative hypothesis is true:

Power = 1 − β = P(reject H₀ | H₁ true).

More precisely, power is a function of the true parameter value. For a test statistic T with rejection region R, the power function is π(θ) = P_θ(T ∈ R). Under θ = θ₀, π(θ₀) = α; under clinically relevant alternatives, π(θ) should usually be at least 0.80. This distinction is examined frequently: a non-significant P value does not prove no effect; it may reflect low power, excessive variance, poor compliance, dilution by crossover, or an effect smaller than the trial was designed to detect.

Sample size for comparing two means

For two independent groups with equal allocation, equal population variance σ², a two-sided α, desired power 1 − β, and a clinically important absolute difference δ = μ₁ − μ₂, the approximate required sample size per group is:

n = 2σ²(Z_1−α/2 + Z_1−β)² / δ².

Using Cohen's standardised effect size d = δ / σ, this becomes:

n = 2(Z_1−α/2 + Z_1−β)² / d².

Design choice	Numerical implication	Effect on sample size
α = 0.05, two-sided	Z1−α/2 = 1.96	Baseline conventional threshold
Power 80%	Z1−β = 0.84	Lower n than 90%, but higher false-negative risk
Power 90%	Z1−β = 1.28	Approximately 34% higher n than 80% power for same α and d
Effect size halved	d reduced by 50%	n increases fourfold because n ∝ 1/d2
Standard deviation doubled	σ doubled	n increases fourfold because n ∝ σ2

For example, with α = 0.05 two-sided and 80% power, Z_1−α/2 + Z_1−β = 1.96 + 0.84 = 2.80. If d = 0.5, then n = 2 × 2.80² / 0.5² = 62.7, so approximately 63 participants per group before inflation for attrition. At 90% power, n = 2 × 3.24² / 0.25 = 84.0 per group. For d = 0.2, often labelled small, approximately 392 participants per group are required at 80% power; for d = 0.8, approximately 25 per group suffice. Cohen's descriptors, small 0.2, medium 0.5, large 0.8, are statistical conventions rather than substitutes for a clinically important difference.

Practical examination points

• α is chosen by the investigator; it is not the probability that the null hypothesis is true after observing a significant result.
• β is conditional on a specified alternative; power is meaningless unless the target difference δ is stated.
• Multiplicity inflates Type I error; repeated endpoints, interim analyses, and subgroup testing require adjustment, such as Bonferroni correction or group-sequential boundaries.
• Variance is pivotal; underestimating σ in planning produces an underpowered study despite apparently adequate recruitment.
• Loss to follow-up requires inflation; if calculated n is 100 per group and expected attrition is 10%, recruit 100 / 0.90 = 112 per group.
• Unequal allocation is inefficient; for allocation ratio r = n₂/n₁, total N = ((1 + r)²/r)σ²(Z_1−α/2 + Z_1−β)²/δ², minimised at r = 1.

The Statistical Selection Algorithm: Parametric vs. Non-Parametric Frameworks

Selection of a statistical test in clinical research is not a matter of preference; it follows from the measurement scale, study design, distributional assumptions and variance structure. In Primary FRCA statistics questions, the safest algorithm is: define the outcome variable first, then the number of groups, then whether observations are independent or paired/repeated, then assess whether parametric assumptions are defensible.

Strict criteria for parametric test validity

Parametric tests such as the Student t-test and ANOVA model data using parameters of an assumed distribution, usually the mean and variance of a Gaussian distribution. Their validity rests on three core assumptions.

• Appropriate measurement scale: the dependent variable should be continuous on an interval or ratio scale. Examples include arterial pressure in mmHg, plasma propofol concentration in µg ml−1, PaCO2 in kPa, or time to extubation in minutes. Ordinal pain scores, ASA physical status and Likert scales are not truly interval data, even if often summarised numerically.
• Approximate normality: the residuals, not necessarily the raw observations, should be approximately normally distributed. For a paired t-test, the distribution of the within-subject differences must be normal. For ANOVA, residuals within each group should be approximately Gaussian. With sample sizes above about 30 per group, the central limit theorem improves robustness for inference on means, but it does not rescue severe skew, heavy tails or influential outliers.
• Homoscedasticity: variances should be similar between independent groups. A pragmatic rule is that the largest group variance should not exceed the smallest by more than about 3- to 4-fold, especially when group sizes are unequal. Levene’s or Brown-Forsythe tests assess equality of variance; if violated for two groups, Welch’s t-test is usually preferable to the classic pooled-variance t-test.

Independence is also fundamental: measurements from one patient must not determine measurements from another. Clustered ICU data, bilateral limb measurements, repeated blood gases or multiple embryos from one patient violate simple independence and may require mixed-effects models rather than the elementary tests listed below.

Mathematical assessment of normality

Normality should be assessed graphically and analytically. Histograms and Q-Q plots show skewness, kurtosis and outliers; formal tests quantify departure from a reference Gaussian distribution but are sample-size dependent.

Shapiro-Wilk test

The Shapiro-Wilk statistic is

W = (Σa_ix_(i))² / Σ(x_i − x̄)²

where x_(i) are ordered observations and a_i are constants derived from expected normal order statistics and their covariance matrix. W approaches 1 when data are compatible with normality; smaller values imply departure from normality. The null hypothesis is that the sample comes from a normally distributed population. Thus, p < 0.05 rejects normality at the conventional 5% significance threshold. Shapiro-Wilk is generally more powerful than Kolmogorov-Smirnov for small to moderate samples and is commonly implemented for n from 3 to 5000.

Kolmogorov-Smirnov test

The Kolmogorov-Smirnov statistic is

D = sup_x|F_n(x) − F₀(x)|

where F_n(x) is the empirical cumulative distribution function and F₀(x) is the specified theoretical cumulative distribution function. It tests the maximum vertical distance between observed and theoretical distributions. When the mean and variance are estimated from the sample rather than specified a priori, the Lilliefors correction is required. Kolmogorov-Smirnov is less sensitive in the tails than Shapiro-Wilk. In very large samples, trivial deviations may yield p < 0.05; in very small samples, important non-normality may be missed. Therefore, formal normality tests should not be used mechanically.

Clinical research design to statistical test mapping

Clinical design or question	Outcome type	Groups or measurements	Parametric test	Non-parametric or categorical alternative
Mean induction dose of propofol in males versus females	Continuous, approximately normal	Two independent groups	Unpaired Student t-test; Welch t-test if variances unequal	Mann-Whitney U test if ordinal, skewed or outlier-prone
PaO2 before and after recruitment manoeuvre in the same patients	Continuous paired differences	Two paired measurements	Paired Student t-test	Wilcoxon signed-rank test if paired differences are non-normal but symmetrically distributed
Mean morphine consumption across three analgesic regimens	Continuous, approximately normal	More than two independent groups	One-way ANOVA with post hoc Tukey testing if global p < 0.05	Kruskal-Wallis test with Dunn-type post hoc comparisons
Cardiac output measured at baseline, post-induction and post-intubation	Continuous repeated measures	More than two related measurements	Repeated-measures ANOVA; requires sphericity, assessed by Mauchly’s test; Greenhouse-Geisser correction if violated	Friedman test for non-parametric repeated measures
Incidence of postoperative nausea with volatile versus TIVA anaesthesia	Categorical binary	Independent proportions	Not applicable	Chi-squared test if expected cell counts are adequate; Fisher’s exact test for small samples
Distribution of Cormack-Lehane grades across devices	Ordinal categorical	Independent groups	Usually inappropriate	Chi-squared test for association; consider ordinal methods if directionality is important

For the Chi-squared test, expected frequencies should generally be at least 5 in each cell, or no more than 20% of cells below 5 and none below 1. Fisher’s exact test is preferred for 2 × 2 tables with small expected counts. Non-parametric tests do not compare medians in all circumstances; more precisely, they compare rank distributions. If distributions have the same shape, this may be interpreted as a location shift. Multiple pairwise testing after ANOVA, Kruskal-Wallis or Friedman procedures requires correction to control type I error, commonly Bonferroni, Holm or Tukey methods.

Interpreting Results: P-values, Effect Sizes, and Diagnostic Metrics

P-values: definition and limitations

A p-value is a tail probability calculated under a specified null hypothesis. Formally, for a test statistic T, the two-sided p-value is P(|T| ≥ |t_obs| | H₀ true), or the probability of observing data at least as incompatible with H₀ as those obtained, assuming the null model and all test assumptions are correct. Thus p = 0.03 does not mean there is a 3% probability that the null hypothesis is true, nor a 97% probability that the treatment works.

For Primary FRCA, the key interpretation is that p-values quantify compatibility of data with a model, not clinical importance. A very small effect may be statistically significant in a large trial, while an important effect may be non-significant in an underpowered trial. The conventional α = 0.05 threshold is arbitrary; repeated interim analyses, multiple endpoints, subgroup testing and selective reporting inflate type I error unless corrected, for example by Bonferroni adjustment, hierarchical testing, or prespecified analysis plans. Modern reporting standards, including CONSORT for trials and STARD 2015 for diagnostic accuracy studies, emphasise estimation with confidence intervals rather than isolated p-values.

Confidence intervals and effect sizes

A 95% confidence interval is a procedure that, in repeated identical sampling, would contain the true parameter in 95% of intervals. It is not strictly a 95% probability that this particular interval contains the truth, although it is often used pragmatically as a range of plausible values. CIs integrate magnitude and precision: a mean difference in propofol induction dose of −0.05 mg kg⁻¹ with 95% CI −0.09 to −0.01 may be statistically significant but clinically trivial; a mortality risk ratio of 0.80 with 95% CI 0.62 to 1.03 may be inconclusive but potentially important.

Hypothesis testing and CIs are mathematically linked. For a two-sided α = 0.05 test, a 95% CI excluding the null value corresponds to p < 0.05. The null value is 0 for differences, such as mean difference or absolute risk reduction, and 1 for ratios, such as relative risk, odds ratio or hazard ratio. Wider CIs indicate less precision, commonly due to small sample size, low event rate, high biological variability, or measurement error.

Diagnostic test statistics from a 2 × 2 table

	Disease present	Disease absent	Total
Test positive	a true positive	b false positive	a + b
Test negative	c false negative	d true negative	c + d
Total	a + c	b + d	n

• Sensitivity = a/(a + c): probability of a positive test if disease is present. A highly sensitive test has few false negatives and is useful for ruling out disease when negative.
• Specificity = d/(b + d): probability of a negative test if disease is absent. A highly specific test has few false positives and is useful for ruling in disease when positive.
• Positive predictive value = a/(a + b): probability of disease given a positive test.
• Negative predictive value = d/(c + d): probability of no disease given a negative test.
• Likelihood ratio positive = sensitivity/(1 − specificity). Values >10 usually provide strong rule-in evidence.
• Likelihood ratio negative = (1 − sensitivity)/specificity. Values <0.1 usually provide strong rule-out evidence.

Predictive values are prevalence-dependent. Consider a test with sensitivity 90% and specificity 90%. In 1000 patients with 10% prevalence: diseased = 100, non-diseased = 900, so true positives = 90, false negatives = 10, false positives = 90, true negatives = 810. PPV = 90/(90 + 90) = 50%; NPV = 810/(810 + 10) = 98.8%; LR+ = 0.90/0.10 = 9; LR− = 0.10/0.90 = 0.11. If prevalence falls to 1%, PPV becomes 9/(9 + 99) = 8.3% despite unchanged sensitivity and specificity. This is why indiscriminate screening in low-risk populations produces many false positives.

Likelihood ratios allow direct Bayesian updating. Pre-test odds = prevalence/(1 − prevalence). Post-test odds = pre-test odds × LR. Post-test probability = odds/(1 + odds). With prevalence 10%, pre-test odds = 0.1/0.9 = 0.111. A positive test with LR+ 9 gives post-test odds 1.0, hence post-test probability 50%, matching the PPV above.

ROC curves and AUC

A receiver operating characteristic curve plots sensitivity against 1 − specificity across all possible thresholds of a continuous test, such as high-sensitivity troponin, D-dimer, lactate, or a risk score. Lowering the threshold increases sensitivity but reduces specificity; raising it does the reverse. The area under the curve is the probability that a randomly selected diseased patient has a more abnormal result than a randomly selected non-diseased patient. AUC 0.5 indicates chance discrimination, 0.7–0.8 acceptable, 0.8–0.9 good, and >0.9 excellent, though these categories are approximate. ROC analysis assesses discrimination, not calibration, clinical utility, or prevalence-dependent predictive value. Threshold choice should therefore reflect clinical consequences: in anaesthesia and peri-operative medicine, missing malignant hyperthermia susceptibility, difficult airway risk, or postoperative myocardial injury may be more harmful than additional confirmatory testing.