The Modeling Tradeoff | Logan Nielsen

In modeling theory, we observe something due to some generation process described by $f(x)$ .

Though there’s some process that produces a phenonemon, there’s always some disturbance that we call $\epsilon$ that effects what we observe. Then

\begin{align} y = f(x) + \epsilon\end{align}

Let’s talk a little more about $\epsilon$ . If we run the same experiment many times, we’ll expect what we see on average to be what the true process is. That is to say $E[\epsilon] = 0$ . We also assume that this noise is entirely random and independent of the experiment. That is to say that $E[\epsilon | \text{experiment}] = 0$ which also implies $Cov(\epsilon, \text{experiment}) = 0$ .

With these assumptions, we’re ready to figure out what we expect $y$ to be on average.

\begin{align} E[y] &= E[f(x) + \epsilon ]\\ &= E[f(x)] + E[\epsilon] \\ &= E[f(x)]\end{align}

The goal of machine learning is to come up with $\hat f(x) \approx f(x)$ . However we do not observe $f(x)$ , we observe $y = f(x)+\epsilon$ . Lets see what the implications of this are for our attempts at modeling. Remember $E[\epsilon]=0$ and $E[\epsilon | \text{experiment}] = 0$ . We probably need to spend some time defining what we mean by “experiment” which would require its own discussion but let’s just say this last statement is equivalent to $0 = E[\epsilon|x]$ .

Note: To avoid unnecessary clutter, in the following derivation we write $\hat f(x)= \hat f$ and $f(x) =f$ . We also write $E[\cdot |x] =E[\cdot]$

\begin{align*} E[(y - \hat f)^2] &= E[(f + \epsilon - \hat f)^2] \\ &= E[(f-\hat f)^2 + \epsilon^2 + 2 \epsilon (f-\hat f)] \\ &= E[(f-\hat f)^2] + \underbrace{E[\epsilon^2]}_{\text{irreduscible Error = $\sigma^2$}} + 2 E[\epsilon (f-\hat f)] \\ &= E[(f-\hat f)^2] + 2 E[\epsilon (f-\hat f)] + \sigma^2 \\ &= E[(f-\hat f)^2] + 2 \underbrace{E[\epsilon]E[f-\hat f]}_{\text{By Independence}} + \sigma^2 \\ &= E[(f - E[\hat f] + E[\hat f] -\hat f)^2] + 2(0)E[f-\hat f] + \sigma^2 \\ &= E[(f - E[\hat f])^2] + \underbrace{E[(\hat f - E[\hat f])^2]}_{\text{Variance}} + 2E[(f-E[\hat f])(E[\hat f]- \hat f )] + \sigma^2 \\ &= \underbrace{(f - E[\hat f])^2}_{\text{$f$ and $E[\hat f]$ are constants}} + \text{Variance} + \underbrace{2(f-E[\hat f])E[E[\hat f]- \hat f ]}_{\text{$f$ and $E[\hat f]$ are constants}} + \sigma^2 \\ &= \underbrace{(f - E[\hat f])^2}_{\text{Bias}^2} + \text{Variance} + 2(f-E[\hat f])\underbrace{(E[\hat f] - E[\hat f])}_{\text{linearity}} + \sigma^2 \\ &= \text{Bias}^2 + \text{Variance} + 2(f-E[\hat f])(0) + \sigma^2 \\ &= \text{Bias}^2 + \text{Variance} + \sigma^2 \end{align*}

If MSE is chosen as the loss function to minimize, then in machin learning the goal is to find the model that will minimize $E[(y-\hat f)^2]$ for out of sample fit. Since we’ve shown that MSE is a function of Bias, model variance, and noise variance.

Econometrics chooses not to minimize MSE, but rather to minimize bias. Better predictions does not mean that bias is minimized, nor does it mean that the model can be used for inference on treatment effects.