A regression model relates the dependent variable (a.k.a. response variable), \(y\), to a function of independent variables (a.k.a. explanatory or predictor variables), \(x\), and unknown parameters (a.k.a. model coefficients) \(\beta\). Such a regression model can be written as
\[ y = f(x; \beta)\text{.}\]
The goal of regression is to find a function such that \(y \approx f (x;\beta)\) for the data pair \((x; y)\). The function \(f(x;\beta)\) is called a regression function, and its free parameters (\(\beta\)) are the function coefficients. A regression method is linear if the prediction function \(f\) is a linear function of the unknown parameters \(\beta\).
By extending the equation to a set of \(n\) observations and \(d\) explanatory variables, \(x_1,..., x_d\) the regression model can be written as
\[ \begin{align} y_i & = \beta_0 + \sum_{j=1}^dx_{ij}\beta_j + \epsilon_i \\ & = \beta_0 + \beta_1x_{1i} + \beta_2x_{2i} +...+ \beta_dx_{di} + \epsilon_i \text{,} \quad i = 1,2,...m\text{, } x\in \mathbb R^d\text{, } \end{align} \]
where \(\beta_0\) corresponds to the intercept, sometimes referred to as bias, shift or offset and \(\epsilon\) corresponds to the error term, referred to as residuals.
A regression model based upon \(m\) observations (measurements) consists of \(n\) response variables, \(y_1, y_2,...y_m\). For the ease of notation we write the response variables as a one-dimensional column vector of the size \(\mathbf{y}_{m \times 1}\).
\[ \begin{align} \mathbf y_{m \times 1}= \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{m} \\ \end{bmatrix} \end{align} \]
Moreover, for each particular observation \(x_i\) \((x_1, x_2,..., x_m)\) we represent the \(d\) associated explanatory variables as a column vector as well.
\[ \begin{align} \mathbf x_{i}= \begin{bmatrix} x_{i1} \\ x_{i2} \\ \vdots \\ x_{id} \\ \end{bmatrix} \end{align} \text{(e.g.)} \Rightarrow \begin{bmatrix} \text{height} \\ \text{weight} \\ \vdots \\ \text{age} \\ \end{bmatrix} \]
Further, by transposing \(\mathbf{x}_i\) we stack a set of \(m\) observation vectors into a matrix \(\mathbf{X}\) of the form \(\mathbf{X}_{m \times d}\):
\[ \begin{align} \mathbf X_{m \times d}= \begin{bmatrix} \mathbf x_{1}^T \\ \mathbf x_{2}^T \\ \vdots \\ \mathbf x_{m1}^T \\ \end{bmatrix} = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1d} \\ x_{21} & x_{22} & \cdots & x_{2d} \\ \vdots & \vdots & \ddots & \vdots \\ x_{m1} & x_{m2} & \cdots & x_{md} \\ \end{bmatrix} \end{align}. \]
This matrix notation is very similar to a spreadsheet representation, where each row corresponds to an observation and each column to a feature. Please note that we assume that all features are continuous-valued \((\mathbf x \in \mathbb R^d)\) and that there are more observations than dimensions \((m > d)\).
Citation
The E-Learning project SOGA-R was developed at the Department of Earth Sciences by Kai Hartmann, Joachim Krois and Annette Rudolph. You can reach us via mail by soga[at]zedat.fu-berlin.de.
You may use this project freely under the Creative Commons Attribution-ShareAlike 4.0 International License.
Please cite as follow: Hartmann, K., Krois, J., Rudolph, A. (2023): Statistics and Geodata Analysis using R (SOGA-R). Department of Earth Sciences, Freie Universitaet Berlin.