The Geometry of Ordinary Least Squares

A linear model can be expressed in the form:

$$\begin{align} \pmb y = X \pmb b + \pmb e \end{align}$$

Where $\pmb y \in \mathbb{R}^N$, $X \in \mathbb{R}^{N\times K}$, $\pmb b \in R^K$, and $\pmb e \in \mathbb{R}^N$.

Mathematically, the equation $\pmb y = X\pmb b + \pmb e$ is nothing more than a statement until we define what the problem is. In OLS, we’re given $\pmb y$ and $X$ and asked to find $b$. What criteria defines $\pmb b$? We say we want to minimize the squared error terms.

Minimizing the squared error terms by select $\pmb b$ can be thought of in two ways.

1. Selecting $\pmb b$ to minimize $\sum\limits_{i=1}^N e_i^2 = \sum\limits_{i=1}^N (y_i - \pmb x_i'\pmb b)^2 = \sum\limits_{i=1}^N (y_i - \hat y_i)^2$. We write this problem as:

$$\begin{align} \argmin_b \ \sum\limits_{i=1}^N e_i^2 \end{align}$$

2. Selecting $b$ to minimize $\pmb e'\pmb e = (\pmb y - X\pmb b)'(\pmb y - X\pmb b) = (\pmb y - \pmb {\hat y})'(\pmb y - \pmb{\hat y})$. We write this as:

$$\begin{align} \argmin_{\pmb{b}} \ \pmb e'\pmb e \end{align}$$

I’ll refer to the setup (1) as the row way of thinking about the problem and (2) as the column way of thinking about the problem.

The Row Interpretation (Elementary)

Thinking about the problem as minimizing $\sum\limits_{i=1}^N e_i^2 = \sum\limits_{i=1}^N (y_i - \pmb x_i'\pmb b)^2$ is intuitive and, consequentially, the first way students are taught to think about regression. Consider row $i$ of the $N$ rows that form the $\pmb y = X\pmb b + \pmb e$.

By convention $X$ with each row representing and “observation” with each column representing some “feature” of the observation. Each $x_i$ has the corresponding scalar outcome $y_i$. We can make a tuple $(\pmb x_i, y_i$) and plot it in the space $\mathbb{R}^{K+1}$.

One features case (K=1)

When we only have one feature our points live in $\mathbb{R}^2$. This is our standard xy-coordinate space from middle school. After we put on all the dots on our paper defined by $(x_i,y_i)$, our goal is draw a line that minimizes the sum of squared y-distances from $(x_i,y_i)$ to the line defined by $(x_i,x_ib) = (x_i, \hat y_i)$. Since $e_i^2 = (y_i - \hat y_i)^2$ we can think of each $e_i^2$ as a collection of squares of which we’re trying to minimize the total area.

Two feature case (K=2)

When $x \in \mathbb R^2$ we now have points that live in $\mathbb R^3$. Writing down each point as the tuple $\left([x_{1i},x_{2i}],y_i\right) = (\pmb x_i, y_i)$ should convey the idea that the part $x$ defines the location of the point along the “base” of the graph, i.e. the xy-coordinates, while the $y$ coordinate pins it down to a specific location along the z-axis. Since $e_i = y_i - \pmb x_i'\pmb b$ we see that $x'b$ defines the plane of possible predictions and $e_i$ is the vertical distance along the z-axis to the plane. Again, these squared vertical distances define a collection of squares whose aggregate area we want to minimize.

Higher dimensions (K>2)

I wrote out case K=2 in a way that visualizing dimensions that aren’t visualizable should be intuitive. Recapping, $x$ defines the “base” of the point given by $(x_i,y_i)$ while $y$ pins the point to an exact location along that final orthogonal (“vertical”) axis. We find the $b$ that traces out that best hyperplane (the surface of the hyperplane is given by $x'b$). In least squares land, the “best” hyperplane is the hyperplane where no other hyperplane could make the area of our collection of squares any smaller.

The Column Interpretation (Advanced)

Minimizing $\pmb e' \pmb e$

What does it mean to minimize $\pmb e'\pmb e$? Note that minimizing $\sqrt{\pmb e'\pmb e}$ is the same as minimizing $\pmb e'\pmb e$ because the square-root operator is a monotonic transformation on $\pmb e'\pmb e$. A monotonic transformation is any transformation $T(\cdot)$ that preserves the order of the set it’s acting upon. In our case, if $\pmb e_1'\pmb e_1 > \pmb e_2'\pmb e_2$ and $T(\cdot)$ is monotonic transformation then $T(\pmb e_1'\pmb e_1) > T(\pmb e_2'\pmb e_2)$. Thus,

$$\begin{align} \argmin_{\pmb b} \ \pmb e'\pmb e \equiv \argmin_{\pmb b} \ \sqrt{\pmb e'\pmb e} \end{align}$$

Now, $\sqrt{\pmb e'\pmb e} = \sqrt{\sum e_i}$ which is the definition of euclidean length. Euclidean length is just an extension of the idea of length in $\mathbb R^2$ and $\mathbb R^3$. Therefore we know we’re trying to minimize the length of the error vector.

Choosing $\pmb b$

The following derivation should make it clear that $\pmb b$ multiplies the columns of $X$ to produce a vector of predictions $\hat{\pmb y}$ that is close to the observed outcomes $\pmb y$.

Recall our original set up:

$$\begin{align} \begin{bmatrix}y_1 \\ y_2 \\ \vdots \\ y_n\end{bmatrix} &= \begin{bmatrix}x_{1,1} & x_{1,2} & \dots & x_{1,k} \\ x_{2,1} & x_{2,2} & \dots & x_{2,k} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n,1} & x_{n,2} & \dots & x_{n,k} \end{bmatrix} \begin{bmatrix}b_1 \\ b_2 \\ \vdots \\ b_k\end{bmatrix} +\begin{bmatrix}e_1 \\e_2 \\ \vdots \\ e_n \end{bmatrix} \end{align}$$

Which is equivalent to:

$$\begin{align} \pmb y &= \begin{bmatrix}\pmb x_1 & \pmb x_2 & \dots & \pmb x_k\end{bmatrix} \begin{bmatrix}b_1 \\ b_2 \\ \vdots \\ b_k\end{bmatrix}+ \pmb e \\\\ &=b_1 \pmb x_1 + b_2 \pmb x_2 + \dots + b_k \pmb x_k + \pmb e \\ &= \pmb{ \hat y } + \pmb e \\ \implies \pmb e &=\pmb y - \pmb{ \hat y } \\ \pmb e &=\pmb y + (-\pmb{ \hat y }) \end{align}$$

Multiplying a vector by a scalar has the action of scaling the vector. Vector addition has the action of putting vectors “tip-to-tail”. Thus, the goal of OLS is to select $\pmb b$ that orients the column vectors of $X$ in the way that minimizes the length of the vector $\pmb e$. Where $\pmb e$ is the vector that connects the prediction vector, $\pmb{ \hat y }$, to the outcome vector, $\pmb y$.

OLS Projection Matrices

There are a couple famous projection matrices associated with OLS.

The first projection matrix is called the prediction maker.

Recall $\pmb b = (X'X)^{-1}X'\pmb y$. Now:

$$\begin{align} \pmb{ \hat y}&= X\pmb b \\ &= X(X'X)^{-1}X'\pmb y \\ &= P\pmb y \end{align}$$

Where $P = X(X'X)^{-1}X'$ is the prediction maker.

We can verify $P$ is projection matrix. $PP = [X(X'X)^{-1}X'][X(X'X)^{-1}X']= X(X'X)^{-1}X'=P$. We call it the prediction maker because $P\pmb y$ generates our vector of predictions.

Learn to recognize $P$ in terms of $X$.

The second projection matrix is called the residual maker.

$$\begin{align} \pmb e &= \pmb y - \pmb{\hat y} \\ &= \pmb y - P\pmb y \\ &= (I-P)\pmb y \\ &= M \pmb y \end{align}$$

Where $M =I-P = I - X(X'X)^{-1}X'$ is the residual maker.

We can quickly verify that $M$ is a projection matrix. $MM = (I-P)(I-P)= I -2P + P = I-P = M$. We call $M$ the residual maker because $M\pmb y= \pmb e$.

Orthogonality of $\pmb{\hat y}$ and $\pmb e$.

We saw that $\hat{ \pmb y} = P\pmb y$ and that $\pmb e = M\pmb y = (I - P)\pmb y$ where $P$ and $M$ are projection matrices. It’s easy to see that $PM = MP = 0$. Therefore the projection matrices map to orthogonal subspaces.

Orthogonality is the generalization of perpendicular. One interpretation for perpendular is that the vectors for a right-angle with each other. Noting that $\hat{\pmb y} \perp \pmb e$ and that $\pmb y = \hat{\pmb y} + \pmb e$, it becomes clear that $\pmb y$, $\hat{\pmb y}$, and $\pmb e$ form a right triangle in $\mathbb R^N$ with $\pmb y$ as the hypotenuse.

In conclusion, we can say that $\pmb b$ maps the column vectors of $X$ to a single vector whose tip is directly “below” $\pmb y$.