Researchers and students alike often mistake any overlap among confidence intervals to denote a statistically non-significant p value. However, confidence intervals can overlap and still correspond to a statistically significant p value for an independent sample t test. The WISE confidence interval applet can help people understand the relationship between confidence interval overlap and statistical significance.
The applet simulates a comparison of the confidence intervals for two group means. The means are displayed as a bar graph with confidence intervals around each group mean. The user can 'grab' one of the means and slide it up or down to change the amount of overlap of the two confidence intervals. The applet displays the p value associated with an independent samples t test for the difference between the two population means.
A common misperception is that statistical significance with p=.05 is attained when the two 95% confidence intervals just touch, but that statistical significance is lost when the intervals overlap. First time users will be surprised to see that the p value is only about .005 when the intervals just touch.
To facilitate an understanding of why the p value is so small when the intervals just touch, the confidence intervals in the display include a representation of the underlying normal sampling distributions. Now it is apparent that when the two intervals just touch, only the very thin tails overlap, and it is highly unlikely that a mean drawn from one distribution would be mistaken for a mean drawn from the other distribution.
Manipulation of the applet allows the user to gain an accurate understanding of how the degree of overlap between confidence intervals is associated with p values for the test of the difference between means. The amount of overlap for p=.05 is likely to be surprising at first encounter.
A Demonstration Guide is linked to the applet.
