Best predictor to minimize absolute deviation.

We observe $y=f(x) +\epsilon$ . As usual, $y$ is what we observe, $f(x)$ is the true data generating process, and $\epsilon$ is random noise uncorrelated with the data generating process. Note that $y$ is really a function of $x$ . In other words, $y = y(x)$

Our model, $\hat f(x)$ generates predictions $\hat y$ .

We choose to minimize the absolute deviation of our model’s predictions with the outcomes. Put mathematically,

\argmin_{\hat f(x)} E[\ |y(x) -\hat f(x) |\ ] \equiv \argmin_{\hat y} E[\ |y-\hat y |\ \vert x]

Note that we are evaluating at a specific value of $x$ . Many data scientists will omit this detail but it’s necessary for a correct understanding of what the model is optimizing. We don’t want a model that gives the median value of $y$ , we want one that gives the median value of $y$ at a particular value of $x$ .

For sake of not being over encumbered, we Write $E[\ |y-\hat y|\ x] = E[\ |y - \hat y|\ ]$ . In other words, it’s implied that we’re considering a single slice at $x$ .

Finally, remember that our model is deterministic and is not subject to noise. Therefore, we’re finding the best $\hat y$ over all possible values of $y$ . Let $y$ by a continuous random variable, then:

\begin{align*} E[\ |y - \hat y|\ ] &= \int_{-\infty}^\infty \underbrace{|y - \hat y|}_{\text{loss}}\ \underbrace{p(y)}_{\text{probability of loss}}\ dy \end{align*}

Our goal is to find the optimal $\hat y$ over this, this can be done by solving first order conditions but we don’t know how to take the derivative of a function with an absolute value. Let’s get rid of the absolute value.

\begin{align*} \int_{-\infty}^\infty |y - \hat y|\ p(y)\ dy &= \int_{-\infty}^{\hat y} -(y-\hat y) p(y)\ dy + \int_{\hat y}^\infty (y-\hat y)p(y) dy \\ \end{align*}

Now we can take the derivative and set it equal to zero to find the optimal $\hat y$ .

\begin{align*} \frac{\partial}{\partial \hat y}\left( \int_{-\infty}^{\hat y} -(y-\hat y) p(y)\ dy + \int_{\hat y}^\infty (y-\hat y)p(y) dy \right) &= \int_{-\infty}^{\hat y} \frac{\partial}{\partial \hat y}\bigg(-(y-\hat y) p(y)\ dy \bigg) + \int_{\hat y}^\infty \frac{\partial}{\partial \hat y} \bigg( (y-\hat y)p(y) \bigg) dy && \text{By Liebniz Rule} \\ &= \int_{-\infty}^{\hat y} p(y)\ dy - \int_{\hat y}^\infty p(y)\ dy \\ &= 0 \\ \implies \int_{-\infty}^{\hat y} p(y)\ dy &= \int_{\hat y}^\infty p(y)\ dy \\ Pr(y \geq \hat y) &= Pr(y \leq \hat y) \end{align*}

This last line is the definition of of the median.

Therefore, we want our model $\hat f(x)$ to produce predictions $Median\big(y(x)\big)$ .