Misconceptions and useful counterexamples in statistics
Table of Contents
This page contains an assortment of useful or informative examples/counterexamples in probability and statistics. I plan to continue updating this page as I come across new examples.
A Bayes factor which contradicts the posterior
The following is an example of what can happen when Bayes factors are used to test point nulls (their only use in psychology).
Courtesy of Stone (1997), consider a binomial experiment involving trials with successes, where we want to test the null-hypothesis that . Assuming a uniform prior, the posterior distribution is , yielding a Bayes factor of 8.11, which Jeffreys calls “substantial” evidence for the null hypothesis. But what does the posterior actually look like?
This would seem to definitively exclude (in fact, with certainty). A classical hypothesis test likewise rejects with a p-value of .
A p-value which doesn’t approximate the probability of a type-1 error
The p-value is the probability of obtaining an effect greater than or equal to the size of the observed effect if the null hypothesis were true. Unfortunately, this is not very well understood in psychology, and p-values are widely misinterpreted as the probability that the null-hypothesis is true (some research suggests that close to 80% of psychology students and researchers hold some variant of this misconception).
Consider screening for breast cancer in a population of 1,000,000 people, where only 100 members of the population actually have cancer. Suppose further that our test is 99% accurate in the sense that, if a person has breast cancer, then we will detect it 99% of the time, and if a person does not, then we will give them a clean bill of health 99% of the time (so the sensitivity and specificity of the test are both 99%). Define the null hypothesis to be that a person is cancer-free. Suppose that we pick a person at random and obtain a positive test result — what is the p-value associated with this outcome? If were true, such a result would only be obtained 1% of the time, so our p-value is .01, which is highly significant. What is the probability that is actually true? Bayes theorem gives us
So we obtain a significant result with , and yet the probability that we have made a type-1 error is 0.99.
A confidence interval which can’t contain the true value
Confidence intervals, much like p-values, are widely misinterpreted in a Bayesian way. In particular, it is common to interpret a 95% confidence intervals for a parameter as having a 95% chance of containing the true value of that parameter, but this is not true for all kinds of reasons (mostly because parameters are not generally considered to be random variables in frequentist statistics).
A confidence interval for a parameter is, simply, a random interval that will contain the true value of some specified percentage of the time. Consider the standard 95% confidence interval for a normal mean — if we we continue drawing random samples and computing confidence intervals, then 95% of the confidence intervals will contain the true value of the mean. This is not the same as saying that some specific interval has a 95% chance of containing the true value (the probably is technically 0 or 1 by frequentist reasoning — it either contains the true value or it doesn’t). The difference is hard to appreciate when it comes to the standard confidence interval for a mean, so consider the following example:
Construct a confidence interval for the mean of a normally distributed random variable as follows: Take a biased coin (with of returning heads) and flip it. If the coin returns heads, define , otherwise, define (the empty set). If we draw random samples and flip our coin over and over again, then 95% of our confidence intervals will contain the true value (since they contain everything), so this is a 95% confidence interval. What happens if the coin lands on tails? It’s still a 95% confidence interval, but it cannot possibly contain the true value, since it contains nothing!
Independence implies uncorrelatedness, but not vice versa. For example, let be standard normal, and let . Then and are dependent, but uncorrelated, since
One of the few situations in which the reverse holds is when a set of random variables are jointly normally distributed, in which case they are independent if they are uncorrelated.
Correlation is not transitive
If is correlated with , and is correlated with , it does not follow that is correlated with . For example, let and be independent (and thus uncorrelated) normal random variables, and define . Then both and are positively correlated with , but the correlation between and is zero.