Approaches to biomedical knowledge

Overview: RNNs and LSTMs

RNNs: Overview

• Mienye ID et al (2024) Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications Information, Information 15(9), 517.

RNN Architecture: hidden layer

RNNs are designed to process sequential data by maintaining a hidden state that captures information about previous inputs.

The basic architecture consists of an input layer, a hidden layer, and an output layer.

• Unlike feedforward neural networks, RNNs have recurrent connections, allowing information to cycle within the networks. At each time step, \(t\), the RNN takes an input vector, \(x_t\) , and updates its hidden state, \(h_t\) using the following equation:

\[ \mathbf{h}_i = \sigma_h(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{b}_h) \tag{1}\]

where

• \(\mathbf{x}_t\): Input at time \(t\)
• \(\mathbf{W}_{xh}\): is the weight matrix between the input and hidden layer
• \(\mathbf{W}_{hh}\) the weight matrix for the recurrent connection
• \(\mathbf{b}_h\): the bias vector
• \(\sigma_h\): the activation function, typically the hyperbolic tangent function (tanh) or the rectified linear unit

RNN Architecture: output

The output at each time step, \(t\), is given by the following: \[ \mathbf{y}_i = \sigma_y(\mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y) \tag{2}\]

where

• \(\mathbf{W}_{hy}\): the weight matrix between the hidden and output layers,
• \(\mathbf{b}_y\): the bias vector,
• \(\sigma_y\) the activation function for the output layer.

Recursivity

• RNNs can be portrayed as (equivalent) folded or unfolded networks

::: ::::

Activation Functions

• As we discussed in previous lectures, the activation function plays a crucial role by introducing non-linearity that enables the network to learn and represent complex patterns.
• One commonly used activation function in RNNs is the hyperbolic tangent (tanh).

RNN Architecture

• Compared to ANNs, the significant change for RNNs is the new set of weights that connect the hidden layer from the previous time step to the current hidden layer (\(\mathbf{W}_{hh}\) the weight matrix for the recurrent connection ).
• \(\mathbf{W}_{hh}\) determines how the network makes use of past context in calculating the output for the current input.
• These weights are also trained by backpropagation
• The activation function for the output layer can be a sigmoid function (for binary classification) or a soft-max function for categorical classfication).

RNN Forward Pass Algorithm

Inputs:

• A sequence of input vectors \(\mathbf{X} = \langle\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T\rangle\)
• Network parameter matrices (\(U, W, V\)) and bias vectors (\(\mathbf{b}_h, \mathbf{b}_y\))
• Activation functions \(g\) (hidden layer, e.g., \(\tanh\)) and \(f\) (output, e.g. softmax)

Output: A sequence of predicted output vectors \(\mathbf{Y} = \langle\mathbf{y}_1, \mathbf{y}_2, \dots, \mathbf{y}_T\rangle\)

Algorithm:

\[ \begin{aligned} &\text{1. } \mathbf{h}_0 \leftarrow \mathbf{0} && \bullet \text{Initialize the baseline context vector with zeros} \\ &\text{2. } \mathbf{Y} \leftarrow \langle \rangle && \bullet \text{Initialize an empty sequence to store outputs} \\ &\text{3. } \textbf{for } t \leftarrow 1 \textbf{ to } T \textbf{ do} && \\ &\text{4. } \quad \mathbf{z}_t \leftarrow U\mathbf{h}_{t-1} + W\mathbf{x}_t + \mathbf{b}_h && \bullet \text{Linear combination of past memory and new input} \\ &\text{5. } \quad \mathbf{h}_t \leftarrow g(\mathbf{z}_t) &&\bullet \text{Compute the updated hidden state} \\ &\text{6. } \quad \mathbf{y}_t \leftarrow f(V\mathbf{h}_t + \mathbf{b}_y) && \bullet \text{Generate the prediction for the current time step} \\ &\text{7. } \quad \mathbf{Y} \leftarrow \mathbf{Y} \parallel \langle\mathbf{y}_t\rangle && \bullet \text{Append the current step output vector to the sequence} \\ &\text{8. } \textbf{end for} && \\ &\text{9. } \textbf{return } \mathbf{Y} && \end{aligned} \]

Training RNNs

• There are now three sets of weights to update:
• \(\mathbf{W}_{xh}\): is the weight matrix between the input and hidden layer
• \(\mathbf{W}_{hh}\) the weight matrix for the recurrent connection
• \(\mathbf{W}_{hy}\): the weight matrix between the hidden and output layers,

Types of RNNs

Training of RNNs

Training of RNNs: Forward pass

Backpropagation Through Time (BTT)

• The gradients are calculated layer by layer from the last time step towards the initial time step.
• We will not present a derivation of the gradients, which is mainly analogous to the derivation we presented for ANNs
• The one exception is the fact that the gradients for the hidden state weights depend on previous time points

\[ \begin{align*} \mathbf{h}_i &= \sigma_h(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{b}_h) \\ &=\sigma_h(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\sigma_h(\mathbf{W}_{xh}\mathbf{x}_{t-1} + \mathbf{W}_{hh}\mathbf{h}_{t-2} + \mathbf{b}_h) + \mathbf{b}_h) \\ &=\sigma_h(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\sigma_h(\mathbf{W}_{xh}\mathbf{x}_{t-1} + \mathbf{W}_{hh}\sigma_h(\mathbf{W}_{xh}\mathbf{x}_{t-2} + \mathbf{W}_{hh}\mathbf{h}_{t-3} + \mathbf{b}_h) + \mathbf{b}_h) + \mathbf{b}_h) \\ &=\ldots \end{align*} \]

The Vanishing and Exploding Gradient Problems

• It can be shown that when calculating the gradients of such “telescoped” terms, we encounter the product of many Jacobian matrices: \[ \frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-n}} = \prod_{k=t-n}^{t-1} \mathbf{J}_k \]

where \(\mathbf{J}_k\) is the Jacobian matrix of the hidden state at time step \(k\). If the eigenvalues of \(\mathbf{J}_k\) are less than 1, the product of these matrices will tend to zero as \(n\) increases, leading to vanishing gradients. If they are greater than one, the gradients will explode.¹

• The vanishing gradient problem prevents the network from effectively learning long-term dependencies, as the gradient signal becomes too weak to update the weights meaningfully for earlier layers.
• The exploding gradient problem can cause the model to converge too quickly to a poor local minimum or make the training process fail entirely due to excessively large updates.

Why tanh?

• Why do we tend not to use the sigmoid activation function in RNNs?
• This has to do with the vanishing gradient problem
• The maximum value of the sigmoid derivative is exactly \(0.25\)
• Recall that the sigmoid is \(\sigma(z) = \frac{1}{1+e^{-z}}\) and the first derivative is \(\sigma^{\prime}(z) = \sigma(z)(1-\sigma(z))\)
• Using the chain rule on \(\sigma(z) - \sigma(z)^2\) to calculate the second derivative, we get \[ \begin{align*} \sigma^{\prime\prime}(z) &= \sigma^{\prime}(z) - 2\sigma(z)\sigma^{\prime}(z) \\ &= \sigma^{\prime}(z)(1-2\sigma(z)) \\ \end{align*} \]
• set \(\sigma^{\prime\prime}(z) = 0\) to find crtical points: \(\sigma^{\prime}(z)(1-2\sigma(z)) = 0\). We note that the slope of the sigmoid function (\(\sigma^{\prime}\)) is never zero for finite values, and so we solve the other term.
• \(1-2\sigma(z) = 0\) implies \(\sigma(z) = \frac{1}{2}\) (Maximum slope of the sigmoid function occurs when \(\sigma(z) = \frac{1}{2}\) )
• It is easy to show that if \(\frac{1}{1+e^z} = \frac{1}{2}\), then \(z=0\). The slope of the sigmoid has its max at the center point, \(z=0\)
• We can now calculate the maximum value of the slope: \(\sigma^{\prime}(z)(0) = \sigma(z)(0)(1-\sigma(z)(0)) = 0.5(1-0.5)=0.25\)

What tanh? (2)

• We can also show that \(\tanh\)’s derivative has its maximum at 1.0 (when the input is 0).
• Therefore, the sigmoid will tend to reduce gradients substantially more than the tanh during backpropagation in time

Additionally

• Sigmoid restricts outputs strictly between \(0\) and \(1\). Because the outputs are always positive, the gradients for the weights in the next layer will all carry the same sign (either all positive or all negative). This forces the gradient descent updates to violently “zig-zag” during optimization, slowing down convergence.
• \(\tanh\) outputs range from \(-1\) to \(1\). Because it is zero-centered, the average output is close to zero. This allows the weights to be updated in both positive and negative directions smoothly, which acts as a natural regularizer and helps the model converge much faster.
• A number of techniques have been developed to mitigate the unstable gradient problem including gradient clipping and LSTMs

TODO update above text. - HOMEWORK You will be asked to implement RNN learning using pytorch etc TODO

Long Short-Term Memory Networks (LSTMs)

• The key innovation in LSTM is the use of gating mechanisms to control the flow of information through the network.

• LSTMs divide the context management problem into two subproblems: r

  • Remove information no longer needed from the context,
  • Add information likely to be needed for later decision making

• Let’s go through step by step

LSTM Gates

LSTMs have three gates:

• forget gate
• add gate
• output gate

Each gateconsists of a feedforward layer, followed by a sigmoid activation function, followed by a pointwise multiplication with the layer being gated.

• The sigmoid activation function tends to push its outputs to either 0 or 1.
• Combining this with a pointwise multiplication has an effect similar to that of a binary mask.
• Values in the layer being gated that align with values near 1 in the mask are passed through nearly unchanged; values corresponding to lower values are essentially erased.

Forget gate

• The forget fate deletes information from the context that is no longer relevant
• It computes a weighted sum of the previous state’s hidden layer and the current input and passes that through a sigmoid.
• This mask is then multiplied element-wise by the context vector to remove the information from context that is no longer required.
• Element-wise multiplication of two vectors (represented by the operator , and sometimes called the Hadamard product) is the vector of the same dimension as the two input vectors, where each element i is the product of element i in the two input vectors:

\[ \begin{align*} \mathbf{f}_t &= \sigma(\mathbf{U}_t\mathbf{h}_{t-1} + \mathbf{W}_f\mathbf{x}_t) \\ \mathbf{k}_t &= \mathbf{c}_{t-1} \odot\mathbf{f}_t \end{align*} \tag{4}\]

Cell input

• Then we compute the actual information we need to extract from the previous hidden state and current inputs—

\[ \mathbf{g}_t = \tanh(\mathbf{U}_g\mathbf{h}_{t-1} + \mathbf{W}_g\mathbf{x}_t) \]

Add gate

Next, we generate the mask for the add gate to select the information to add to the current context.

\[ \begin{align*} X\end{align*} \tag{5}\]

it = σ(Uiht−1 +Wixt) (13.23) jt = gt it (13.24) Next, we add this to the modified context vector to get our new context vector. ct = jt +k