Hypothesis Testing

Hypothesis testing involves a statistical analysis that attempts to determine whether a result is due to chance or due to some real cause or phenomenon.

These concepts are also presented in section 6.2 of the text.

Examples

• Did Mervin the Mystic guess the color of 60 out of 100 card draws due to chance? Or does he have some power (or trick) for knowing the color of a card before it is revealed?
• Why did BestPurchase.com generally have better prices than CrazyPrices.com for a set of selected items? Was it due to the chance of selecting particular items? Or does BestPurchase generally offer lower prices?
• When users complete tasks faster on the new Web design, is it because it's a better design? Or is it simply because more fast users were randomly selected for the new design?

Terms

• Null hypothesis -- The hypothesis that the observed result is due to chance. Also called H₀.
• Alternate hypothesis -- The hypothesis that the observed result is due to something other than chance (some underlying pattern, effect, power, etc). Also called H_a or the experimental hypothesis.
• p value -- The probability that the observed result could be obtained assuming the null hypothesis. If this value is small enough, we reject the null hypothesis.
• alpha criterion for statistical significance -- This value sets the level at which we decide to reject the null hypothesis. If the p-value is below this level, we decide to reject the null hypothesis and accept the alternate hypothesis. Often the convention is to have an alpha of 0.05, but some scientific communities prefer a lower alpha.
• One-tailed test -- This kind of analysis only counts the probability of the result being extreme in one direction.
• Two-tailed test -- This kind of analysis calculates the probability of the result being extreme in either direction. It's harder to reject the null hypothesis when using a two-tailed test than a one-tailed test. Typically, two-tailed tests are used.