Overview: Neural networks

ANNs: Overview

Neuron

image credit: https://en.wikipedia.org/wiki/Biological_neuron_model

Neuron

Each neuron receives the weight sum of activations from the previous layer (or the input values) and applies a non-linear activation function

• Fully connected layer¹: \[z = \sum_{i=1}^{4} (w_i x_i) + b\]
• A common activation function is the Sigmoid function

\[f(z) = \frac{1}{1 + e^{-z}}\]

Activation functions

• Activation functions decide whether a neuron should be “fired” (activated) by calculating the weighted sum and adding bias.

Representation power

A neural network with at least one hidden layer is a universal approximator in that it can represent any function

Forward pass

Backward pass

Applications in bio-medicine

Figure 2: Weiss R, et al (2022) Applications of Neural Networks in Biomedical Data Analysis. Biomedicines. 10(7):1469.

• There is hardly any topic in biomedicine that has not been modeled with ANNs of one type of another
• Deep understanding of the basics of (simple) feed-forward networks is a prerequisite for understanding more sophisticated models in use today

Overview: Neural networks

Review: ANN Architecture and Matrix Representation

ANN Architecture

• Input: 8x8 grey-scale images of handwritten digits
• Goal: Determine which digit the image represents (\(0,\ldots, 9\))
• Input layer: \(X^{64\times 1}\)
• Output layer: \(Y^{10\times 1}\)
• Hidden layer(s): three hidden layers (128, 64, 32 neurons)

Figure 3: ANN Architecture: \(64 \to 128 \to 64 \to 32 \to 10\)

Representing the network using matrices

\[ \begin{bmatrix} w_{1,1} & w_{1,2} & \cdots & w_{1,64} \\ w_{2,1} & w_{2,2} & \cdots & w_{2,64} \\ & & & \\ \vdots & \vdots & \ddots & \vdots \\ & & & \\ w_{128,1} & w_{128,2} & \cdots & w_{128,64} \\ \end{bmatrix}^{128\times 64} \begin{bmatrix} x_1\\ x_2\\ \vdots \\ x_{64} \end{bmatrix}^{64\times 1} + \begin{bmatrix} b_1\\ b_2\\ \; \\ \vdots \\ \; \\ b_{128} \end{bmatrix}^{128\times 1} = \begin{bmatrix} z_1\\ z_2\\ \; \\ \vdots \\ \; \\ z_{128} \end{bmatrix}^{128\times 1} \] followed by activation \[ \begin{bmatrix} \sigma(z_1)\\ \sigma(z_2)\\ \; \\ \vdots \\ \; \\ \sigma(z_{128}) \end{bmatrix}^{128\times 1} = \begin{bmatrix} a_1\\ a_2\\ \; \\ \vdots \\ \; \\ a_{128} \end{bmatrix}^{128\times 1} \]

• The \(w_{j,i}\) values are called the weights. They are learnable parameters
• The \(z_j\) values are called the weighted sums or the pre-activations.
• The \(a_j\) values are called the activations and these are passed on as the input to the next layer

The Forward Pass: Step-by-Step

A single “layer” of computation follows a predictable three-step cycle:

Neurons as functions

Each neuron in the hidden layers can be thought of as a function that takes the results of all neurons in the previous layer as input and returns a number between 0 and 1.

The entire network can be thought of as a function that takes 64 numbers as input and returns 10 numbers as output.

Backpropagation

• The forward operation of each layer computes an output based on its input and the current values of the parameters (\(\theta\))
• The learning problem is to find the parameters that achieve a desired mapping.
• We do this via gradient descent.
• But how are the gradients computed?

Backprop: Toy example

• An algorithm that efficiently calculates the gradient of the loss with respect to each parameter in a computation graph.

• In backpropagation, we need to know how the Loss (\(L\)) changes with respect to the weights.
• We will start off by calculating the backprop by hand with this toy network

Backprop by hand

• Our network will have a sigmoid activation function in the hidden layer, no activation in the output layer:

\[ \sigma(x) = \dfrac{1}{1+e^{-x}} \]

• We will use a squared-error loss function (where \(y\) is the label, zero or one) and \(a_2\) is the output of the ANN for inputs \(x_1\) and \(a_3\): \[ \mathcal{L} = \dfrac{1}{2}(y-a_3)^2 \]

Backprop by hand

Chain rule (review)

• To understand backprop, it is essential to be familiar with the chain rule

Calculating partial derivatives

To calculate the gradient of the loss function, \(\nabla (\Theta; \mathbf{x};y)\), we need all the partial derivatives – for all weights and biases. Let’s do this step by step

• derivative of the sigmoid activation function \[ \begin{eqnarray*} \dfrac{d\sigma(x)}{dx} =\dfrac{d}{dx} \dfrac{1}{1+e^{-x}} &=& \hspace{1cm} \dfrac{-1}{(1+e^{-x})^2}\cdot (-e^{-x})&& \text{chain rule} \\ &=&\dfrac{e^{-x}}{(1+e^{-x})^2} && \hspace{1cm} \text{simplify}\\ &=&\dfrac{1}{1+e^{-x}} \cdot \dfrac{e^{-x}}{1+e^{-x}}&& \hspace{1cm} \text{rearrange}\\ &=&\sigma(x) \cdot \dfrac{e^{-x}}{1+e^{-x}}&& \hspace{1cm} \text{definition of $\sigma(x)$}\\ &=&\sigma(x) \cdot \dfrac{1 + e^{-x} - 1}{1+e^{-x}}&& \hspace{1cm} \text{add and substract 1}\\ &=&\sigma(x) \cdot \left( \dfrac{1 + e^{-x} }{1+e^{-x}} -\dfrac{1}{1+e^{-x}} \right)&& \hspace{1cm} \text{rearrange}\\ &=&\sigma(x) \cdot \left( 1 -\sigma(x) \right)&& \hspace{1cm} \text{simplify using definition of $\sigma(x)$}\\ \end{eqnarray*} \]

• The forward pass calculates \(\sigma(x)\). Using this derivation, the backward pass can calculate the partial derivate easily using the value of \(\sigma(x)\).

Derivative of the loss function

• In our toy network, \(a_3\) was the output of the network (i.e., the prediction).
• \(y\) is the true value
• The squared error loss function is thus

\[ \mathcal{L}=\dfrac{1}{2}(y-a_3)^2 \]

• The first derivative of the loss function with respect to the output is then \[ \dfrac{\partial \mathcal{L}}{\partial a_3} = 2\cdot \dfrac{1}{2}(y-a_3)\cdot(-1) = a_3 - y \]

How does the loss change as \(w_6\) changes?

How does the loss change as \(b_3\) changes?

How does the loss change as \(b_2\) changes?

How does the loss change as the other parameters change?

• HOMEWORK You will be asked to derive expressions for \(w_1, w_2, w_3, w_4, w_5, b_1, b_2\).
• If we change the network architecture, the activation function, or the loss function, we would need to recalculate these expressions
• This manual procedure is tedious and would not scale to exploring various network structures for realistic network sizes.
• We will see it is possible to automate this procedure, but to understand that, we will show how it is possible to use matrix calculus to abstract away some of the fidgety work with the indices

Matrix calculus: Vector function with scalar argument

Tangent vector

• Here, \(\mathbf{f}\) is bold - the function returns a vector, and \(x\) is normal font - the function takes a scalar.

Matrix calculus: Scalar function with vector argument

Matrix calculus: Vector function with vector argument

The numerator layout convention yields a column vector for the derivative of \(\mathbf{f}(x)\) (a function that takes a scalar argument and returns a vector). Correspondingly, if \(\mathbf{f}\) is a vector and \(\mathbf{x}\) is a vector, the derivative is a matrix where each row corresponds to an element of \(\mathbf{f}\) and each column to an element of \(\mathbf{x}\).

• For \(\mathbf{f}: \mathbb{R}^m \rightarrow \mathbb{R}^n\), the first derivative is called the Jacobian:

\[ \dfrac{\partial \mathbf{f}}{\partial \mathbf{x}} = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \dfrac{\partial f_1}{\partial x_2} & \cdots & \dfrac{\partial f_1}{\partial x_m} \\ \dfrac{\partial f_2}{\partial x_1} & \dfrac{\partial f_2}{\partial x_2} & \cdots & \dfrac{\partial f_2}{\partial x_m} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial f_n}{\partial x_1} & \dfrac{\partial f_n}{\partial x_2} & \cdots & \dfrac{\partial f_n}{\partial x_m} \\ \end{bmatrix}^{n\times m} \tag{6}\]

Matrix calculus: Vector function with vector argument

• We can think of Equation 7 as a stack of transposed gradient vectors (c.f., Equation 5)

Matrix calculus: Scalar function with matrix argument

• let \(f\) be a function that takes an \(n\times m\) matrix as input and produces a scalar value
• \(f: \mathbb{R}^{n\times m}\rightarrow \mathbb{R}\)
• In the Numerator Layout, the rule is that the derivative \(\frac{\partial f}{\partial \mathbf{X}}\) should have the shape of the transpose of \(\mathbf{X}\).
• If \(\mathbf{X}\) is \(n \times m\) then \(\frac{\partial f}{\partial \mathbf{X}}\) must be \(m \times n\)

\[ \dfrac{\partial f}{\partial \mathbf{X}} = \begin{bmatrix} \frac{\partial f}{\partial x_{11}} & \frac{\partial f}{\partial x_{21}} & \cdots \frac{\partial f}{\partial x_{n1}} \\ \frac{\partial f}{\partial x_{12}} & \frac{\partial f}{\partial x_{22}} & \cdots \frac{\partial f}{\partial x_{n2}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial f}{\partial x_{1m}} & \frac{\partial f}{\partial x_{2m}} & \cdots \frac{\partial f}{\partial x_{nm}} \\ \end{bmatrix}^{m\times n} \tag{8}\]

Matrix calculus: Scalar function with matrix argument

• Similar to our motivation for the dimensions of the derivative of a vector function with vector arguments, we can intuit that the transpose is necessary by comparing to the definition of \(\frac{\partial d}{\partial \mathbf{x}}\)
• We defined \(\frac{\partial d}{\partial \mathbf{x}}\) as a row vector, using \(\mathbf{x}^T\) as the ordering
• Therefore, consistency demands that the \(\dfrac{\partial f}{\partial \mathbf{X}}\) be ordered as \(\mathbf{X}^T\)

Backpropagation with matrix notation

• Each layer of a fully connected feed forward network (FFN) can be thought of as a vector function \(f: \mathbb{R}^n \rightarrow \mathbb{R}^m\):

\[ \mathbf{y}^{m\times 1} = \mathbf{f}(\mathbf{x}^{n\times 1}) \]

• The input to the layer is an \(n\)-dimensional vector \(\mathbf{x}^{n\times 1}\). This vector is either the actual input to the FFN or (if we are working with a hidden layer), the output of the previous layer.

• The forward pass runs through each layer of the FFN. \(\mathbf{y}^{(i)}\) becomes \(\mathbf{x}^{(i+1)}\) (output of layer \(i\) becomes the input to layer \(i+1\))

• The output of the final layer is used to calculate the loss. Letting this layer be called \(\mathbf{h}\), we have \[ \mathcal{L} = L(\mathbf{h}, \mathbf{y}_{true}) \]

• The loss \(\mathcal{L}\) is a scalar value.

• We need to move this error back through the network to perform back propagation

FFN forward pass in matrix notation

• For layers with an input vector coming from the previous layer, we have \[ \mathbf{y}^{m\times 1} = \mathbf{W}^{m\times n}\mathbf{x}^{n\times 1} + \mathbf{b}^{m\times 1} \]

• For activation layers, we have \[ \mathbf{y}^{m\times 1} = \mathbf{\sigma}(\mathbf{x}^{m\times 1} ) \]

• In this notation, \(\mathbf{\sigma}\) is a vector-valued function (and is shown in bold)

\[ \mathbf{\sigma}(\mathbf{x}^{m\times 1} ) = \begin{bmatrix} \sigma(x_1) \\ \sigma(x_2) \\ \vdots \\ \sigma(x_m) \\ \end{bmatrix} \]

Backprop derivatives

• The derivative of the loss function with respect to the network output is the first error term
• To pass this back through the network, we must determine how the error term changes with a change to the input of a layer with respect to how the error changes with respect to the lyers output.

Backprop algorithms

• \(\dfrac{\partial E}{\partial \mathbf{x}}\) for layer \(i\) becomes \(\dfrac{\partial E}{\partial \mathbf{y}}\) for layer \(i-1\) as we move back through the FFN

The entire backprop algorithm can be summarized as follows

1. Run a forward pass to map \(\mathbf{x}\rightarrow \mathbf{y}\) layer for layer, and get the final output \(\mathbf{h}\)
2. Calculate the value of the derivative of the loss function using \(\mathbf{h}\) and \(\mathbf{y}_{true}\). This becomes \(\frac{\partial E}{\partial \mathbf{y}}\) for the output layer
3. Repeat for all layers: calculate \(\frac{\partial E}{\partial \mathbf{x}}\) from \(\frac{\partial E}{\partial \mathbf{y}}\) with \(\frac{\partial E}{\partial \mathbf{x}}\) of layer \(i\) becoming \(\frac{\partial E}{\partial \mathbf{y}}\) for layer \(i-1\)

Backprop algorithms: Derivative of activation layer

• \(\frac{\partial E}{\partial \mathbf{y}}\) is the derivative of the error with respect to the output. This term will depend on the specific loss function. We will see an example in the final section of this lecture.
• In our toy network, we are using the mean squared error (MSE) loss function:
• The cost function for a batch of \(n\) samples is the average of individual errors: \[ E_{batch} = \frac{1}{n} \sum_{i=1}^{n} E_i = \frac{1}{2n} \sum_{i=1}^{n} (y_i - t_i)^2 \tag{9}\]

The derivative is: \[ \frac{\partial E_{batch}}{\partial \mathbf{y}} = \frac{1}{n} (\mathbf{y} - \mathbf{t}) \tag{10}\]

The Jacobian of the Activation

• In our example, the final hidden layer is connected to the output via a sigmoid activation function
• Given \(\mathbf{y} = \sigma(\mathbf{x})\), the derivative \(\frac{\partial \mathbf{y}}{\partial \mathbf{x}}\) is a Jacobian Matrix. Since \(y_i\) only depends on \(x_i\) (and no other \(x_j\)), the matrix is diagonal:

\[ \frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \begin{bmatrix} \frac{\partial y_1}{\partial x_1} & 0 & \dots \\ 0 & \frac{\partial y_2}{\partial x_2} & \dots \\ \vdots & \vdots & \ddots \end{bmatrix} = \begin{bmatrix} \sigma'(x_1) & 0 & \dots \\ 0 & \sigma'(x_2) & \dots \\ \vdots & \vdots & \ddots \end{bmatrix} \]

Applying the Chain Rule

The gradient of the error \(E\) with respect to the input \(\mathbf{x}\) is defined by the matrix product of the upstream gradient and the Jacobian:

\[ \begin{aligned} \frac{\partial E}{\partial \mathbf{x}} &= \frac{\partial E}{\partial \mathbf{y}} \cdot \frac{\partial \mathbf{y}}{\partial \mathbf{x}} \\ &= \begin{bmatrix} \frac{\partial E}{\partial y_1} & \dots & \frac{\partial E}{\partial y_m} \end{bmatrix} \begin{bmatrix} \sigma'(x_1) & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & \sigma'(x_m) \end{bmatrix} \\ &= \begin{bmatrix} \frac{\partial E}{\partial y_1}\sigma'(x_1) & \dots & \frac{\partial E}{\partial y_m}\sigma'(x_m) \end{bmatrix} \end{aligned} \]

• This matrix multiplication is computationally equivalent to the Hadamard Product (\(\odot\)):

\[ \frac{\partial E}{\partial \mathbf{x}} = \frac{\partial E}{\partial \mathbf{y}} \odot \sigma'(\mathbf{x}) \tag{11}\]

Backprop algorithms: Derivative of fully connected layer

Derivative of fully connected layer - derivation

If \(\mathbf{y} = \mathbf{Wx} + \mathbf{b}\), then: \[ y_1 = w_{11}x_1 + w_{12}x_2 + \dots + w_{1n}x_n + b_1 \]

Taking the derivative of the first output (\(y_1\)) with respect to the first input (\(x_1\)): \[ \frac{\partial y_1}{\partial x_1} = w_{11} \]

• Therefore, if we take the derivative of output \(i\) (\(y_i\)) with respect to input \(j\) (\(x_j\)) we get \(\frac{\partial y_i}{\partial x_j} = w_{ij}\)
• When we fill out the entire matrix, therefore, we get the original weight matrix \(\mathbf{W}^{m\times n}\)

Partial derivatives of the bias

• The previous slides told us how to pass the error term backward through the network from layer to layer.
• The goal of backprop is to get first derivatives for tthe bias and weight terms (how would a change in the bias/weight affect the error?) to enable gradient descent.
• We need to be able to calculate \(\frac{\partial E}{\partial \mathbf{b}}\) and \(\frac{\partial E}{\partial \mathbf{W}}\) given that we have \(\frac{\partial E}{\partial \mathbf{y}}\) and \(\frac{\partial E}{\partial \mathbf{x}}\) from previous calculations

\[ \begin{eqnarray*} \dfrac{\partial E}{\partial \mathbf{b}} &=& \dfrac{\partial E}{\partial \mathbf{y}} \dfrac{\partial \mathbf{y}}{\partial \mathbf{b}} \\ &=& \dfrac{\partial E}{\partial \mathbf{y}} \dfrac{\partial \mathbf{Wx+b}}{\partial \mathbf{b}}\\ &=& \dfrac{\partial E}{\partial \mathbf{y}} (\mathbf{0} + \mathbf{I} )\\ &=& \dfrac{\partial E}{\partial \mathbf{y}} \end{eqnarray*} \tag{13}\]

Partial derivatives of the weights

Gradient descent