A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
Statistical conclusion validity concerns the accuracy of conclusions about whether an observed covariation between variables is genuine. This depends on proper assumptions including independent observations, homogeneity of variance, appropriate distribution forms, and adequate statistical power. Violations of assumptions can lead to inflated or deflated Type I and Type II error rates, producing biased conclusions. Researchers must verify statistical assumptions through diagnostic tests and use appropriate statistical techniques (e.g., nonparametric alternatives, robust estimators) when assumptions are violated.
Conduct analyses assuming violated assumptions to observe how conclusions change. Practice diagnostic tests (Q-Q plots, Levene's test, independence checks) on real datasets.
If p < .05, the conclusion is definitely correct (violating assumptions can bias p-values). Statistical tests are robust to all assumption violations (actual robustness depends on specific assumptions, effect sizes, and sample sizes).
From your study of hypothesis testing and statistical power, you know that a statistical test can produce two kinds of error: a Type I error (a false positive — you conclude there is an effect when there isn't) and a Type II error (a false negative — you miss a real effect). You also know that power is the probability of detecting a true effect. Statistical conclusion validity is the umbrella question: *can you trust the conclusion your statistical test produced?* It is threatened whenever the test's assumptions are violated, because those violations silently change the actual Type I and Type II error rates away from what you thought you had set.
Every parametric statistical test is built on assumptions. The t-test and ANOVA assume that observations are independent of each other (no clustering), that residuals are approximately normally distributed, and that group variances are roughly equal (homogeneity of variance). These are not arbitrary formalities — the math that produces the p-value you observe is derived under these conditions. When the conditions do not hold, the null distribution changes shape, and the critical value you used to decide whether to reject H₀ is no longer correct. A test that nominally operates at α = .05 might, under severe assumption violations, actually produce false positives at α = .15 — or, if the violation pushes in the other direction, at α = .01. You no longer know what you have.
The most consequential assumption in practice is independence of observations. Clustering — measuring multiple students in the same classroom, multiple patients from the same clinic, multiple observations from the same person over time — introduces positive dependence within clusters. Standard errors computed under the independence assumption are too small, p-values are too small, and Type I error rates are inflated. The fix is to use multilevel models or cluster-robust standard errors that account for the nested structure. Independence violations are especially insidious because they are invisible in raw data — you have to know the data collection procedure to spot them.
Non-normality of residuals matters most in small samples. With sample sizes above roughly 30–40 per group, the central limit theorem means that sampling distributions of means are approximately normal even if the raw data are not — this is what people mean when they say ANOVA is "robust to non-normality." But this robustness is conditional on adequate sample size and does not apply to all statistics (e.g., tests involving variances are less robust). Heterogeneity of variance is more troubling when combined with unequal group sizes: if the large group also has the larger variance, Type I error is inflated; if the large group has the smaller variance, it is deflated. Welch's t-test and Welch's ANOVA correct for unequal variances and should be used by default rather than the standard versions.
The practical discipline of statistical conclusion validity is running diagnostic checks before interpreting results. Q-Q plots assess normality of residuals; Levene's test or Bartlett's test assesses homogeneity of variance; intraclass correlations detect clustering. When assumptions are violated, the response is not to run the test anyway and hope — it is to choose a procedure whose assumptions match your data: nonparametric alternatives (Wilcoxon, Kruskal-Wallis) when normality is badly violated; robust estimators (bootstrap confidence intervals, heteroskedasticity-consistent standard errors) when variance is unequal; multilevel models when data are nested. The goal is not a specific p-value, but a p-value you can interpret as meaning what it is supposed to mean.