In order to make a decision whether to reject the null hypothesis, a test statistic is calculated. The decision is made on the basis of the numerical value of that test statistic. There are two approaches how to arrive at that decision: The critical value approach and the p-value approach.
The critical value approach
By applying the critical value approach it is determined, whether or not the observed test statistic is more extreme than a defined critical value. Therefore, the observed test statistic (calculated on the basis of sample data) is compared to the critical value (a kind of cutoff value). If the test statistic is more extreme than the critical value, the null hypothesis is rejected. If the test statistic is not as extreme as the critical value, the null hypothesis is not rejected. The critical value is computed based on the given significance level \(\alpha\) and the type of probability distribution of the idealized model. The critical value divides the area under the probability distribution curve in rejection region(s) and non-rejection region.
The following three figures show a right tailed test, a left tailed test and a two-sided test. The idealized model in the figures, and thus \(H_0\), is described by a bell-shaped normal distribution curve.
In a two-sided test the null hypothesis is rejected, if the test statistic is either too small or too large. Thus, the rejection region for such a test consists of two parts: one on the left and one on the right.
For a left-tailed test the null hypothesis is rejected, if the test statistic is too small. Thus, the rejection region for such a test consists of one part, which is left from the center.
For a right-tailed test the null hypothesis is rejected, if the test statistic is too large. Thus, the rejection region for such a test consists of one part, which is right from the center.
The p-value approach
For the p-value approach the likelihood (p-value) of the numerical value of the test statistic is compared to the specified significance level (\(\alpha\)) of the hypothesis test.
The p-value corresponds to the probability of observing sample data at least as extreme as the actually obtained test statistic. Small p-values provide evidence against the null hypothesis. The smaller (closer to 0) the p-value, the stronger is the evidence against the null hypothesis.
If the p-value is less than or equal to the specified significance level \(\alpha\), the null hypothesis is rejected. Otherwise, the null hypothesis is not rejected.
Note: if \(p \le \alpha\), reject \(H_0\); otherwise, if \(p > \alpha\), do not reject \(H_0\)
In consequence, by knowing the p-value any desired level of significance may be assessed. For example, if the p-value of a hypothesis test is 0.01, the null hypothesis can be rejected at any significance level larger than or equal to 0.01. It is not rejected at any significance level smaller than 0.01. Thus, the p-value is commonly used to evaluate the strength of the evidence against the null hypothesis without reference to significance level.
The following table provides guidelines for using the p-value to assess the evidence against the null hypothesis (Weiss, 2011) ):
\[ \begin{array}{l|l} \hline \text{p-value} & \text{Evidence against } H_0 \\ \hline \ p > 0.10 & \text{Weak or no evidence} \\ \ 0.05 < p \le 0.10 & \text{Moderate evidence} \\ \ 0.01 < p \le 0.05 & \text{Strong evidence} \\ \ p \le 0.01 & \text{Very strong evidence} \\ \hline \end{array} \]
Citation
The E-Learning project SOGA-R was developed at the Department of Earth Sciences by Kai Hartmann, Joachim Krois and Annette Rudolph. You can reach us via mail by soga[at]zedat.fu-berlin.de.
You may use this project freely under the Creative Commons Attribution-ShareAlike 4.0 International License.
Please cite as follow: Hartmann, K., Krois, J., Rudolph, A. (2023): Statistics and Geodata Analysis using R (SOGA-R). Department of Earth Sciences, Freie Universitaet Berlin.