Pearson product moment correlation coefficient
Correlation is a commonly used method to examine the relationship between quantitative variables. The most commonly used statistic is the linear correlation coefficient, \(r\), which is also known as the Pearson product moment correlation coefficient in honor of its developer, Karl Pearson. It is given by
\[r = \frac{\sum_{i=1}^n(x_i- \bar x)(y_i - \bar y)}{\sqrt{\sum_{i=1}^n(x_i- \bar x)^2}\sqrt{\sum_{i=1}^n(y_i- \bar y)^2}}=\frac{s_{xy}}{s_x s_y}\text{,}\]
where \(s_{xy}\) is the covariance of \(x\) and \(y\), \(s_x\) and \(s_y\) are the standard deviations of \(x\) and \(y\), respectively. By dividing by the sample standard deviations, \(s_x\) and \(s_y\), the linear correlation coefficient, \(r\), becomes scale independent and takes values between \(-1\) and \(1\).
The linear correlation coefficient measures the strength of the linear relationship between two variables. If \(r\) is close to \(\pm 1\), the two variables are highly correlated, and when plotted on a scatter plot, the data points cluster around a line. If \(r\) is far from \(\pm 1\), the data points are more widely scattered. If \(r\) is near \(0\), the data points are essentially scattered around a line, indicating that there is almost no linear relationship between the variables.
An interesting property of \(r\) is that its sign reflects the slope of the linear relationship between two variables. A positive value of \(r\) suggests that the variables are positively linearly correlated, indicating that \(y\) tends to increase linearly as \(x\) increases. A negative value of \(r\) suggests that the variables are negatively linearly correlated, indicating that \(y\) tends to decrease linearly as \(x\) increases.
There is no unambiguous classification rule for the quantity of a linear relationship between two variables. However, the following table may serve as a rule of thumb for how to address the numerical values of the Pearson product moment correlation coefficient:
\[\begin{array}{lc} \hline \ \text{Strong linear relationship} & r > 0.9 \\ \ \text{Medium linear relationship} & 0.7 < r \le 0.9\\ \ \text{Weak linear relationship} & 0.5 < r \le 0.7 \\ \ \text{No or doubtful linear relationship} & 0 < r \le 0.5 \\ \hline \end{array}\]
Pearson’s correlation assumes the variables to be roughly normally distributed, and it is not robust in the presence of outliers.
In a later section on linear regression, we discuss the coefficient of determination, \(R^2\), a descriptive measure for the quality of linear models. There is a close relation between \(R^2\) and the linear correlation coefficient, \(r\). The coefficient of determination, \(R^2\), equals the square of the linear correlation coefficient, \(r\):
\[\text{coefficient of determination }(R^2) =r^2 \]
Pearson correlation coefficient: An example
In order to get some intuition, we calculate the Pearson product
moment correlation coefficient in an example. Therefore, we load the
students data set into our workspace (You may download the
students.csv file here
or load it directly using read.csv()).
students <- read.csv("https://userpage.fu-berlin.de/soga/data/raw-data/students.csv")The students data set consists of 8239 rows, each of them representing a particular student, and 16 columns, each of them corresponding to a variable/feature related to that particular student. These self-explanatory variables are: stud.id, name, gender, age, height, weight, religion, nc.score, semester, major, minor, score1, score2, online.tutorial, graduated, salary.
In this example, we assess the linear relationship between
the weight and the height of students. For this, we randomly
pick 37 students and extract the weight and
height variables from the data set.
n <- 37
sample_idx <- sample(1:nrow(students), size = n)
weight <- students[sample_idx, "weight"]
height <- students[sample_idx, "height"]
plot(height, weight)
The scatter plot indicates that there exists a linear relationship between the two variables under consideration.
For the sake of this exercise we calculate the linear correlation
coefficient by hand at first and then we apply the cor()
function in R. Recall the equation from above:
\[r = \frac{\sum_{i=1}^n(x_i- \bar x)(y_i - \bar y)}{\sqrt{\sum_{i=1}^n(x_i- \bar x)^2}\sqrt{\sum_{i=1}^n(y_i- \bar y)^2}}=\frac{s_{xy}}{s_x s_y}\]
x <- height
y <- weight
x_bar <- mean(height)
y_bar <- mean(weight)
sum((x - x_bar) * (y - y_bar)) / (sqrt(sum((x - x_bar)^2)) * sqrt(sum((y - y_bar)^2)))## [1] 0.9743696As as sanity check we calculate the ratio of the covariance of \(x\) and \(y\) and the standard deviations of \(x\) and \(y\):
\[r = \frac{s_{xy}}{s_x s_y}\]
cov(x, y) / (sd(x) * sd(y))## [1] 0.9743696Finally, we apply the in-build cor() function:
cor(x, y)## [1] 0.9743696Perfect. The three calculations yield the exact same result! The linear correlation coefficient evaluates to \(r = 0.9743696\). Thus, we may conclude that there is a strong linear correlation between the height and the weight of a student.
Of course a correlation analysis is not restricted to two variables. Thanks to statistical software packages, such as R, we are able to conduct a pairwise correlation analysis for more than two variables. Let us first prepare the data set. For a better visualization experience we draw 100 randomly picked students from the students data set. Then we select a number of variables to perform the correlation analysis on.
n <- 100
sample_idx <- sample(1:nrow(students), size = n)
vars <- c("height", "weight", "nc.score", "score1", "score2", "salary") # select variables
cor(students[sample_idx, vars])## height weight nc.score score1 score2 salary
## height 1.00000000 0.94526224 -0.02448925 NA NA NA
## weight 0.94526224 1.00000000 -0.04048844 NA NA NA
## nc.score -0.02448925 -0.04048844 1.00000000 NA NA NA
## score1 NA NA NA 1 NA NA
## score2 NA NA NA NA 1 NA
## salary NA NA NA NA NA 1The cor() function returns a nice table, also called
correlation matrix, with the pairwise Pearson
correlation coefficients. Obviously, some variables contain missing
values, denoted as NA. We exclude those values from the
analysis by adding the argument
use = 'pairwise.complete.obs' to the function call.
cor(students[sample_idx, vars], use = "pairwise.complete.obs")## height weight nc.score score1 score2
## height 1.00000000 0.94526224 -0.024489250 0.195887582 0.16769035
## weight 0.94526224 1.00000000 -0.040488441 0.241458260 0.21921260
## nc.score -0.02448925 -0.04048844 1.000000000 -0.008946945 -0.02146809
## score1 0.19588758 0.24145826 -0.008946945 1.000000000 0.87080991
## score2 0.16769035 0.21921260 -0.021468095 0.870809912 1.00000000
## salary 0.51781672 0.36597798 0.283531817 0.381895086 0.15137427
## salary
## height 0.5178167
## weight 0.3659780
## nc.score 0.2835318
## score1 0.3818951
## score2 0.1513743
## salary 1.0000000A table is a nice representation for a correlation analysis, but a
figure would of course improve the interpretability. R provides the
pairs() function for plotting correlation matrices.
pairs(students[sample_idx, vars])
Immediately, we realize that the majority of the variables does not
appear to be linearly correlated. In contrast, the variable pairs
height and weight, as well as
score1 and score2 appear to be positively
correlated.
Another recommendable function for pair plots is provided by the
psych package. The function pairs.panels() is
very flexible and comes with many nice plotting features. Visit the help
page or type ?pairs.panels into your console for more
information on the function.
library(psych)
pairs.panels(students[sample_idx, vars],
smooth = FALSE,
ellipses = FALSE,
main = "Pearson product moment correlation coefficient"
)
The function returns a plot, which shows scatter plots, the variable histograms and the density curves as well as the correlation coefficients.
Note: Correlation functions implemented in R, such as
cor()orpairs.panels(), include different types of correlation coefficients such as Pearson’s, Spearman’s and Kendall’s correlation coefficients. To pick one particular formula you add the argumentmethodto the function call. Pearson’s correlation coefficient is the default setting.
pairs.panels(students[sample_idx, vars],
smooth = FALSE,
ellipses = FALSE,
method = "spearman",
main = "Spearman's rank correlation coefficient"
)
We will have a closer look at Spearman’s rank correlation coefficient on the following page.
Note: It is always recommended to advance with a statistical test, in order to assess whether the result is statistically significant or whether the variation is just due to chance. Check out the sections on Hypothesis Testing for further information!
Spurious correlations
A spurious correlation occurs, when there are initial dependencies between two variables, which are not founded by the measured values. In other words: When the observed correlation is rooted in common dependencies of both variables or complete coincidence, rather than a relation in causality.
A good example to emphasize this is the occurrence of childbirths and stork sightings. Both the stork population and the birth rates of humans peak in spring to summer and therefore correlate in timing. Whatsoever, both variables have no effect on each other of course Sappsford & Jupp.
A spurious correlation can occur for a number of reasons:
For once, the stork example shows, that the similar timing of unrelated events can cause a spurious correlation. Here, the reoccurring seasons are the common, governing factor for both variables.
Secondly, a constant sum causes spurious correlations. This is commonly seen in concentrations, like oxygen or carbon-dioxide concentrations in air. If the concentration of one gas in the air mixture is reduced, all other gases will rise in in concentration, as the volume will stay constant at \(100\)%. This is, of course not, for the absolute mass.
A third, common reason is the skewness of the variables. Positive
skewness will lead to exaggerated correlations, due to the distributions
shape. The correlation itself is calculated by the arithmetic mean, which is very sensitive to positive
outliers. Such are commonly given in positively skewed
distributions.
Correlation, cosine and scalar product
As already intruduced in the previous section about the covariance, we may consider two variable \(x\) and \(y\), both as vectors in the n-dimensional real Euclidian vector space \(\mathbb R^n\) (sample space): \[\vec x' = (x_i - \bar x; i=1,...,n )\qquad and \qquad \vec y' = (y_i - \bar y; i=1,...,n )\].
In this scenario, the correlation can be interpreted as cosine of the angle between \(\vec x\) and \(\vec y\):
\[\cos\!\big(\vec{x'},\vec{y'}\big)= \frac{\langle \vec{x'},\vec{y'}\rangle}{\|\vec{x'}\|\|\vec{y'}\|}= \frac{\sum_{i=1}^n x_i' y_i'}{\sqrt{\sum_{i=1}^n x_i'^2}\sqrt{\sum_{i=1}^n y_i'^2}}= \frac{\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^n (x_i-\bar{x})^2}\sqrt{\sum_{i=1}^n (y_i-\bar{y})^2}}= r\]
By this relationship, we can examine three extreme cases for alternating \(\vec y'\) in relation to \(\lambda\vec x'\):
1. \(r=0\) : \(\cos\!\big(\angle\vec{x},\vec{y_1}\big)=0 \rightarrow\angle(\vec{x},\vec{y_1}\big)=90^\circ\)
==> both variable vectors are rectangular to each other!
2. \(r = 1\) : \(\cos\!\big(\angle\vec{x},\vec{y_2}\big)=1\rightarrow \angle(\vec{x},\vec{y_2}\big)=0^\circ\)
==> both variable vectors point to the same direction!
3. \(r=-1\) : \(\cos\!\big(\angle\vec{x},\vec{y}\big)=-1\rightarrow \angle(\vec{x},\vec{y}\big)=180^\circ\)
both variable vectors point in the exact opposite direction of each other! ***
Let us visualize this aspect on a 2D-projection plane:
Here we see, that the correlation coefficient \(r\) is only meaningful in a data space with an orthonormal base (Euclidean data space) as the cosine and sine is only defined in a right triangle!
Under these requirements the correlation coefficient \(r=cos \angle(\vec x',\vec y')\) provides a measure of joint direction:
Since the correlation coefficient is the normalized covariance:
\[ cos^2+sin^2=r^2+(1-r)^2=1 \qquad\text{and}\] Thus, the coefficient of determination (COD) also has a geometric meaning and we can define the squared sine as the Coefficient of non-determination (COnD) and split the variance meaningful into a “common portion” and a “unique portion”! > However, covariance, correlations, and COD/COnD together with their geometric implications are only meaningfull in a reference system with an orthogonal base (= linear independent variable space)! In cases of inherent dependencies through constant sum constraints, compositional dependencies or external common relations (e.g. seasonal or spatial dependencies resp. temporal/spatial autocorrelation), the reference system becomes oblique!
If two variables are spurious correlated (linear dependent), it means that their bases become oblique to each other through a common perspective (variable). Imagine: it is impossible to project a three-axis rectangular object onto a 2D plane preserving right angles.
This leads to the \(\cos\), the correlation coefficient \(r\) and the coefficient of determination \(R^2\) to become meaningless:
Important conclusion:
- Spurious correlation indicates an oblique projection of our data
- Spurious correlations obscure the message in our data
- Spurious correlations produce artificial patterns
- Spurious correlations obscure the data space!
- Correlation analysis needs interval scales!
- Correlation on ratio scales is meaningless without proper
transformation!
- Correlation of variables in a constrained data space is
nonsense!
- Variables with units of %, mg/g, ppm, etc. have to be transformed
before applying correlation/covariance analyses.
Citation
The E-Learning project SOGA-R was developed at the Department of Earth Sciences by Kai Hartmann, Joachim Krois and Annette Rudolph. You can reach us via mail by soga[at]zedat.fu-berlin.de.
You may use this project freely under the Creative Commons Attribution-ShareAlike 4.0 International License.
Please cite as follow: Hartmann, K., Krois, J., Rudolph, A. (2023): Statistics and Geodata Analysis using R (SOGA-R). Department of Earth Sciences, Freie Universitaet Berlin.