Any decision made based on a hypothesis test may be incorrect. In the framework of hypothesis tests there are two types of errors: Type I error and type II error. A type I error occurs if a true null hypothesis is rejected (a “false positive”), while a type II error occurs if a false null hypothesis is not rejected (a “false negative”). In other words, a type I error is detecting an effect that is not present, while a type II error is failing to detect an effect that is present.

$\begin{array}{l|ll} & H_0 \text{ is true} & H_0 \text{ is false} \\ \hline \text{Do not reject } H_0 & \text{Correct decision} & \text{Type II error} \\ \text{Reject } H_0 & \text{Type I error} & \text{Correct decision} \\ \hline \end{array}$

If you are confused about the type I and type II errors you may find this illustration helpful (here).

Conducting a hypothesis test always implies that there is a chance of making an incorrect decision. The probability of the type I error (a true null hypothesis is rejected) is commonly called the significance level of the hypothesis test and is denoted by $$\alpha$$. The probability of a type II error (a false null hypothesis is not rejected) is denoted by $$\beta$$. Keep in mind that for a fixed sample size, the smaller we specify the significance level, $$\alpha$$, the larger will be the probability, $$\beta$$, of not rejecting a false null hypothesis (Weiss 2010).

The outcome of a hypothesis test is a statement in favor of the null hypothesis or in favor of the alternative hypothesis. If the null hypothesis is rejected, the data does provide sufficient evidence to support the alternative hypothesis. If the null hypothesis is not rejected, the data does not provide sufficient evidence to support the alternative hypothesis. If the hypothesis test is performed at the significance level $$\alpha$$, one may state that the test results are statistically significant at the $$\alpha$$ level. When the null hypothesis is not rejected at the significance level $$\alpha$$, one may state that the test results are not statistically significant at the $$\alpha$$ level.