A Brief Introduction to Deep Learning

Deep learning a subfield of machine learning. Basic knowledge of machine learning is a helpful foundation for deep learning but you’ll pick these up as you learn about deep learning.

Neural Networks as Universal Function Approximators

Artificial neural networks (ANNs) are universal function approximators. Theoretically, with proper design, they can take any input space, and map it to any output space. Typically, an ANN can be described as $f:\mathbb{R^n} \mapsto \mathbb{R}^m$ where $n$ is the dimensionality of the input and $m$ is the dimensionality of the output.

\begin{align} \underbrace{\hat y}_\text{output} = f(\underbrace{X}_\text{input}) \end{align}

Objective Functions

While neural networks can appear quite magical to people not familiar with them, conceptually they’re actually pretty straightforward. Loosely speaking, because neural networks have some much flexibility in the outputs they can produce, they need a lot of data to be trained on in order to be able to produce reasonable outputs.

Instead of providing a neural network with how we want it to find the answer, only tell it what the answer is. When the neural network produces an answer close to what is desired, we don’t tell it to change but when it performs poorly, we tell it to change a lot. The number that determines how much the neural network needs to change is called the cost function. Cost functions and objective functions are closely related. By minimizing a cost function, you are maximizing your implied objective function.

To train a neural network we need input-output pairs like this: $(X_1, y_1), (X_2, y_2), \dots, (X_N,y_N)$ . The loss function if we were minimizing squared error would look like this:

\begin{align} L(X, y) = (f(X) - y)^2 \end{align}

Which would imply our objective function is something like:

\begin{align} \argmin_{A_l} L(X,y) = \sum_{i=1}^N(f(X_i) - y_i)^2 \end{align}

We’ll explain what $A_l$ is in the next section.

Typical Form for $f$

Deep neural networks take in an input, multiply it by a linear transformation followed by a non-linear activation before being “passed to the next layer”. The next layer treats the output from the previous layer as an input and repeats the same process. The non-linear transformations in addition to incorporating many layers are what give artificial neural networks their flexibility.

Let $A_l \in \mathbb{R}^{\dim(l) \times \dim(l+1)}$ describe the linear transformation that layer $l$ performs on the input and $\dim(l)$ be the dimension of that input with $\dim(l+1)$ being the dimension of the output at layer $l$ which is also the input for layer $l+1$ . Furthermore, let $\phi_l(\cdot)$ be some function that is a non-linear mapping from each scalar in the input to another scalar. Finally let $X$ be our input. Then for an $L$ layer ANN we can describe it with a function:

\begin{align} \hat y = \phi_L(A_L(\dots\phi_2 (A_2\phi_1(A_1X))) \end{align}

If this is your first time seeing this, it probably looks like gibberish which is why networks are usually diagrammed in a form that resembles this:

\begin{align} \underbrace {X}_{\text{input}} \to \underbrace{\phi_1(A_1X)}_{\text{layer 1}} \to \underbrace{\phi_2(A_2X_1)}_{\text{layer 2}} \to \dots \to \underbrace{\phi_L(A_LX_{L-1})}_{\text{layer M}} = \underbrace{\hat y}_{\text{output}} \end{align}

Where $X_l$ is the output from the previous layer.

Training, Validating, and Testing

When training a neural network, you have to make sure that your the training data your using is representative of the population for which you want to generate predictions.

Generally data will be used as following.

First we find $\{N\}$ observations to use for training out model. Then we partition this bank of observations into three sets, $\{N\} = \{Train\}+\{Validate\} + \{Test\}$ .

The $\{Train\}$ set is what we’re allowed to use in our quest to tune the weights ( $A_l$ ) of our model.

The $\{Validate\}$ set is what we use during training to make sure we’re not overfitting.

Finally, the $\{Test\}$ set should not be touched until after the model has been fully trained. Seeing how to the model performs on the $\{Test\}$ set should give us an idea of how well we might expect our model to perform on brand-new data once we use it to make predictions in the real world.

Some Neural Network Anatomy

The entire in the matrices $A_l$ are called “weights” these are what get adjusted when a neural network is being trained.

Each $\phi(\cdot)$ is usually picked from a fairly small set of what are called “activation functions”. Some population activation functions include the sigmoid, ReLU, Leaky ReLU, and TanH. As was mentioned earlier, activation functions are what provide neural networks their flexibility because activation function are non-linear. Otherwise, an entire network could be described perfectly by one weight matrix: $(A_3A_2A_1)X = BX$

Why learn the “math” of neural networks and not just the intuition?

Though diagrams are helpful for understanding the general design of a network, they leave the details of how the network functions ambiguous. Typically, diagrams are used to make deep learning more approachable but for mathematically inclined individuals, it leaves details confused which can make replication of published models more difficult than necessary.

To increase clarity of various neural network designs, I plan to cover some canonical neural network designs with fairly rigorous mathematical notation.