Recurrent Neural Networks (RNNs) and extensions such as LSTMs introduced the concept of processing sequential data (like clinical notes, time-series measurements, or genetic sequences).
a patient’s visit history (Sequence of observations and measures at different times)
Sequences can have wildly different lengths from one example to the next.
It is not straightforward to model sequences with standard ANNs, which take a fixed-dimensional input vector
RNN: Basic idea
Key idea: RNNs have an internal (hidden) state that is updates as a sequence is processed \[
\mathbf{h}_t = f_W(\mathbf{h}_{t-1}, \mathbf{x}_t)
\]
\(\mathbf{h}_t\): New state
\(f_W\): function with learnable parameters W
\(\mathbf{h}_{t-1}\): Prior state
\(\mathbf{x}_{t}\): Input at time \(t\)
U, V, and W are learnable matrices
Image credit: Wikipedia
RNN: Output
There is a separate function for the output of an RNN
Key idea: RNNs have an internal (hidden) state that is updates as a sequence is processed \[
\mathbf{y}_t = f_W_{hy}(\mathbf{h}_{t})
\]
The output at time \(t\), $_t $, is calculated from the hidden state
\(f_W_{hy}\): function with learnable parameters W_{hy}
\(\mathbf{h}_{t}\): hidden state at time \(t\)
Convert the dimension of the hidden state to the dimension of the output
Image credit: Wikipedia
Unfolding the RNN
This unfolded view is equivalent
The parameters \(\mathbf{U}\), \(\mathbf{V}\), and \(\mathbf{W}\) are learnable weight matrices that are the same for each time step
RNN types
Many-to-one: e.g., sequence of words, predict whether text is positive or negative (sentiment analysis
One-to-many: e.g., image captioning. From one image, create a sequence of words to describe the image
Many-to-many (synched): one output for each input, e.g. part-of-speech prediction
Many-to-many (encoder-decoder): e.g., translation
See: Das S, et al (2023) Recurrent Neural Networks (RNNs): Architectures, Training Tricks, and Introduction to Influential Research. PMID:37988518.
RNNs: Overview
Recurrent Neural Networks
Unlike standard feed-forward networks, Recurrent Neural Networks (RNNs) introduce a hidden state (\(\mathbf{h}_t\)) that acts as a “memory,” carrying information from time step \(t-1\) to time step \(t\).
possess both current and past features of the temporal sequences
adapt to the long-term historical changes in the data
store the past information to solve context-dependent tasks
make predictions simultaneously with existing observations.
RNNs are designed to process sequential data by maintaining a hidden state that captures information about previous inputs.
The basic architecture consists of an input layer, a hidden layer, and an output layer.
RNNs have recurrent connections, allowing information to cycle within the networks. At each time step, \(t\), the RNN takes an input vector, \(x_t\) , and updates its hidden state, \(h_t\) using the following equation:
\(\mathbf{W}_{xh}\): is the weight matrix between the input and hidden layer
\(\mathbf{W}_{hh}\) the weight matrix for the recurrent connection
\(\mathbf{b}_h\): the bias vector
\(\sigma_h\): the activation function, typically the hyperbolic tangent function (tanh) or the rectified linear unit
RNN Architecture: output
The output at each time step, \(t\), is given by the following: \[
\mathbf{y}_i = \sigma_y(\mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y)
\tag{2}\]
where
\(\mathbf{W}_{hy}\): the weight matrix between the hidden and output layers,
\(\mathbf{b}_y\): the bias vector,
\(\sigma_y\) the activation function for the output layer.
Recursivity
RNNs can be portrayed as (equivalent) folded or unfolded networks
The hidden layer from the previous time step provides a form of memory, or context, that encodes earlier processing and informs the decisions to be made at later points in time.
This approach does not impose a fixed-length limit on this prior context; the context embodied in the previous hidden layer can include information extending back to the beginning of the sequence.
Adapted from Jurafsky D and Martin JH (2026) Speech and Language Processing (3rd ed. draft)
Activation Functions
As we discussed in previous lectures, the activation function plays a crucial role by introducing non-linearity that enables the network to learn and represent complex patterns.
One commonly used activation function in RNNs is the hyperbolic tangent (tanh).
tanh
The tanh function squashes any real-valued input down to a symmetric, zero-centered range of \([-1, 1]\).
Compared to ANNs, the significant change for RNNs is the new set of weights that connect the hidden layer from the previous time step to the current hidden layer (\(\mathbf{W}_{hh}\) the weight matrix for the recurrent connection ).
\(\mathbf{W}_{hh}\) determines how the network makes use of past context in calculating the output for the current input.
These weights are also trained by backpropagation
The activation function for the output layer can be a sigmoid function (for binary classification) or a soft-max function (for categorical classfication).
Understanding the hidden state
To demonstrate that a hidden state can actually “remember” previous steps, let us construct by hand an RNN that outputs \(x_t\) if \(x_t > x_{t-1}\), otherwise it outputs zero.1
The general equation for the hidden state pre-activation is: \[
z_t = w_{hh} h_{t-1} + w_{xh} x_t
\]
In this toy example, we define the hidden state to be a \(2\times 1\) column vector that memorizes both the current input and the immediate history. \[
\begin{align*}
z_t^{(2\times 1)} &= w_{hh}^{(2\times 2)} h_{t-1}^{(2\times 1)} + w_{xh}^{(2\times 1)} x_t^{(1\times 1)} \\
h_t^{(2\times 1)} &= \mathrm{relu}(z_t^{(2\times 1)})
\end{align*}
\]
That is, Combined State Pre-Activation (\(z_t\)) consists of two numbers: the current input (\(x_t\)) and the first element of the previous hidden state (\(h_{1, t-1}\))
This corresponds to \[
y_t = \text{ReLU}\left( w_{yh} h_t \right)
\] - \(w_{yh}\) is a 1D row weight vector (or 1D array of shape \((2,)\)). - \(h_t\) is a column vector of shape \((2, 1)\):\[h_t = \begin{bmatrix} h_{1, t} \\ h_{2, t} \end{bmatrix} = \begin{bmatrix} x_t \\ x_{t-1} \end{bmatrix}\] - Expanding the dot product using the weights \(w_{yh} = \begin{bmatrix} 1 & -1 \end{bmatrix}\):\[\begin{align*} w_{yh} h_t &= \begin{bmatrix} 1 & -1 \end{bmatrix} \begin{bmatrix} h_{1, t} \\ h_{2, t} \end{bmatrix} \\ &= (1 \cdot h_{1, t}) + (-1 \cdot h_{2, t}) \\ &= h_{1, t} - h_{2, t} \\ &= x_t - x_{t-1} \end{align*}\]
Understanding the hidden state (5)
Applying the Activation Function (\(\text{ReLU}\)): - Finally, the scalar result passes through the element-wise \(\text{ReLU}\) function to ensure the output is non-negative: \[y_t = \text{ReLU}(x_t - x_{t-1}) = \max(0, x_t - x_{t-1})\] - If \(x_t > x_{t-1}\), the output is positive (\(x_t - x_{t-1}\)). - If \(x_t \le x_{t-1}\), the output floors at \(0\).
Of course, real RNNs are trained to learn the parameters of the network, but we have shown in this simple example that the architecture of an RNN can be made to remember past information
In the following slides, we will show how RNNs are trained.
RNN Forward Pass Algorithm
Inputs:
A sequence of input vectors \(\mathbf{X} = \langle\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T\rangle\)
It can be shown that when calculating the gradients of such “telescoped” terms, we encounter the product of many Jacobian matrices: \[
\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-n}} = \prod_{k=t-n}^{t-1} \mathbf{J}_k
\]
where \(\mathbf{J}_k\) is the Jacobian matrix of the hidden state at time step \(k\). If the eigenvalues of \(\mathbf{J}_k\) are less than 1, the product of these matrices will tend to zero as \(n\) increases, leading to vanishing gradients. If they are greater than one, the gradients will explode.1
The vanishing gradient problem prevents the network from effectively learning long-term dependencies, as the gradient signal becomes too weak to update the weights meaningfully for earlier layers.
The exploding gradient problem can cause the model to converge too quickly to a poor local minimum or make the training process fail entirely due to excessively large updates.
The Vanishing and Exploding Gradient Problem: Intuition
Let us examine an example to understand why the vanishing gradient problem occurs.
For simplicity, Let’s consider the following symmetric matrix \(A\): \[A = \begin{bmatrix} 0.65 & 0.15 \\ 0.15 & 0.65 \end{bmatrix}\]
To diagonalize \(A\), we first find its eigenvalues \(\lambda\) by solving \(\det(A - \lambda I) = 0\):
The Vanishing and Exploding Gradient Problem: Intuition (2)
Any diagonalizable matrix can be factored as:\[A = P D P^{-1}\]
where \(D\) is a diagonal matrix containing the eigenvalues – in our example we have: \[D = \begin{bmatrix} 0.8 & 0 \\ 0 & 0.5 \end{bmatrix}\]
When we multiple the matrix by itself many times, the intermediate terms cancel out \[\begin{align*} A^n &= (P D P^{-1}) (P D P^{-1}) \dots (P D P^{-1}) \\ &= P D (P^{-1} P) D (P^{-1} P) \dots D P^{-1} \\ &= P D^n P^{-1} \end{align*}\]
Because \(D\) is a diagonal matrix, raising it to the power \(n\) simply means raising each individual eigenvalue to the power \(n\):\[D^n = \begin{bmatrix} 0.8^n & 0 \\ 0 & 0.5^n \end{bmatrix}\]
The Vanishing and Exploding Gradient Problem: Intuition (3)
Because both eigenvalues (\(\lambda_1 = 0.8\) and \(\lambda_2 = 0.5\)) are less than \(1\), raising them to a large power \(n\) forces them to decay exponentially toward zero.
The Vanishing and Exploding Gradient Problem: Intuition (4)
This is the algebraic root of the vanishing gradient problem in RNNs: if the eigenvalues of the hidden-to-hidden Jacobian matrices are less than 1, long-term gradient signals shrink to zero before they can reach earlier time steps.
Conversely, if eigenvalues are greater than 1, \(D^n\) explodes toward infinity, causing exploding gradients.
Why tanh?
Why do we tend not to use the sigmoid activation function in RNNs?
This has to do with the vanishing gradient problem
The maximum value of the sigmoid derivative is exactly \(0.25\)
Recall that the sigmoid is \(\sigma(z) = \frac{1}{1+e^{-z}}\) and the first derivative is \(\sigma^{\prime}(z) = \sigma(z)(1-\sigma(z))\)
Using the chain rule on \(\sigma(z) - \sigma(z)^2\) to calculate the second derivative, we get \[
\begin{align*}
\sigma^{\prime\prime}(z) &= \sigma^{\prime}(z) - 2\sigma(z)\sigma^{\prime}(z) \\
&= \sigma^{\prime}(z)(1-2\sigma(z)) \\
\end{align*}
\]
set \(\sigma^{\prime\prime}(z) = 0\) to find crtical points: \(\sigma^{\prime}(z)(1-2\sigma(z)) = 0\). We note that the slope of the sigmoid function (\(\sigma^{\prime}\)) is never zero for finite values, and so we solve the other term.
\(1-2\sigma(z) = 0\) implies \(\sigma(z) = \frac{1}{2}\) (Maximum slope of the sigmoid function occurs when \(\sigma(z) = \frac{1}{2}\) )
It is easy to show that if \(\frac{1}{1+e^z} = \frac{1}{2}\), then \(z=0\). The slope of the sigmoid has its max at the center point, \(z=0\)
We can now calculate the maximum value of the slope: \(\sigma^{\prime}(z)(0) = \sigma(z)(0)(1-\sigma(z)(0)) = 0.5(1-0.5)=0.25\)
What tanh? (2)
We can also show that \(\tanh\)’s derivative has its maximum at 1.0 (when the input is 0).
Therefore, the sigmoid will tend to reduce gradients substantially more than the tanh during backpropagation in time
Additionally
Sigmoid restricts outputs strictly between \(0\) and \(1\). Because the outputs are always positive, the gradients for the weights in the next layer will all carry the same sign (either all positive or all negative). This forces the gradient descent updates to violently “zig-zag” during optimization, slowing down convergence.
\(\tanh\) outputs range from \(-1\) to \(1\). Because it is zero-centered, the average output is close to zero. This allows the weights to be updated in both positive and negative directions smoothly, which acts as a natural regularizer and helps the model converge much faster.
A number of techniques have been developed to mitigate the unstable gradient problem including gradient clipping and LSTMs
Overview: RNNs and LSTMs
Game plan
Recurrent Neural Networks (RNNs) and extensions such as LSTMs introduced the concept of processing sequential data (like clinical notes, time-series measurements, or genetic sequences).
Accuracy: 73.94% (as compared to the 50% we would have expected by chance, but much worse than the training accuracy of 97.85% - thus, we are overfitting! In the practical, we will discuss some methods for reducing overfitting)
Overview: RNNs and LSTMs
Game plan
Recurrent Neural Networks (RNNs) and extensions such as LSTMs introduced the concept of processing sequential data (like clinical notes, time-series measurements, or genetic sequences).
The key innovation in LSTM is the use of gating mechanisms to control the flow of information through the network.
A challenge for RNNs is to keep track of information from distant parts of the sentence:
The rates the phone company was increasing were already expensive
It is easy to relate “was” to “company” because “was” comes immediately after “company”
It is harder to relate “were” to “rates” because there are five other intervening words
The hidden state of the RNN is being asked to perform two things: inform the current output and to update and carry forward the information needed for future outputs
LSTMS attempt to address the challenge by adding a context layer to the existing hidden layer and by introducing gates to control flow of information
Hochreiter, S. and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780.
LSTMS vs RNNs
All recurrent neural networks have the form of a chain of repeating modules.
In standard RNNs, this module has a simple structure with a single tanh layer and passes on the hidden state
LSTMS vs RNNs
In contrast, LSTM modules have four intacting structures and are connected by the cell state in addition to the hidden state
LSTMS
Let’s review the main ideas of the LSTM step by step
LSTM Gates
LSTMs have three gates:
forget gate
add gate
output gate
Each gate consists of
feedforward layer
followed by a sigmoid activation function
followed by a pointwise multiplication with the layer being gated.
The sigmoid activation function tends to push its outputs to either 0 or 1.
Combining this with a pointwise multiplication has an effect similar to that of a binary mask.
Values in the layer being gated that align with values near 1 in the mask are passed through nearly unchanged; values corresponding to lower values are essentially erased.
Forget gate
The forget fate deletes information from the context that is no longer relevant
It computes a weighted sum of the previous state’s hidden layer and the current input and passes that through a sigmoid.
This mask is then multiplied element-wise by the context vector to remove the information from context that is no longer required.
Element-wise multiplication of two vectors, represented by the \(\odot\) operator,1 results in a vector where element \(i\) is the product of element \(i\) in the two input vectors:
an LSTM accepts as input the context layer (\(\mathbf{c}_{t-1}\)) and hidden layer (\(\mathbf{h}_{t-1}\)) from the previous time step, along with the current input vector (\(\mathbf{x}_{t}\)).
It generates \(\mathbf{c}_{t}\) and \(\mathbf{h}_{t}\) as output that is passed to the next step
The hidden state \(\mathbf{h}_{t}\) is used to generate the output of the network similar to an RNN
LSTMs can be used as drop-in replacements for RNNs in many contexts and often perform better
Overview: RNNs and LSTMs
Game plan
Recurrent Neural Networks (RNNs) and extensions such as LSTMs introduced the concept of processing sequential data (like clinical notes, time-series measurements, or genetic sequences).
that language models predict the next word in a sequence given some preceding context. For example, if the preceding context is “Something is rotten in the state of” and we want to know how likely the next word is “Denmark”1 we would compute:
\[
P(\mathrm{Denmark}\mid \text{Something is rotten in the state of})
\]
That is, we are computing the probability of the word “Denmark” given the previous 7 words.
RNN language models process the input sequence one word at a time, attempting to predict the next word from the current word (\(x_t\)) and the previous hidden state (\(h_{t-1}\)).
Forward inference in an RNN language model
The input is a sequence of words \(\mathbf{X} = \left[x_1;\ldots;x_t\right]\)
Let the Vocabulary be a collection of words of size \(|V|\)
Then each word is represented by a one-hot column vector of size \(|V|\times 1\).
The output prediction \(\hat{\mathbf{y}}\) is a vector representing a probabilites distribution over the vocabulary.
For each step the model uses an embedding matrix \(\mathbf{E}\) to retrieve the embedding for the current word1
The model then multiplies the embedding vector for each word by the weight matrix \(\mathbf{W}\), and then adds it to the hidden layer from the previous step (weighted by weight matrix \(\mathbf{U}\) to compute a new hidden layer. This hidden layer is then used to generate an output layer which is passed through a softmax layer to generate the final prediction.
The softmax operation creates a probability distribution oover the entire vocabulary
The probability that a particular word \(k\) is the next word is then \[
P(w_{t+1}| w_1,\ldots, w_t ) = \hat{\mathbf{y}_t}[k]
\]
\(\hat{\mathbf{y}}\) is a vector with one field for every word in the vocabulary
The output is then the highest-probability word
CE Loss: Review
For a multi-class classification problem, each true label vector \(\mathbf{y}\) is one-hot encoded (meaning \(y_i = 1\) for the correct class and \(y_i = 0\) otherwise),
The model predicts a set of probabilities \(\hat{y}_{i}\) for each class
For a multi-class classification problem with \(K\) classes, the cross-entropy loss is \[
\log L(\theta) = -\sum _{i=1}^{C} y_{i}\log \hat{y}_{i}\\
\tag{8}\]
For the correct class (\(y_k = 1\)), the loss term \(-\log(p_k)\) decreases as \(p_k\) increases.
Weight updates that increase the probability of an incorrect class must decrease the probability of the correct class, thereby increasing the loss
Training RNN language Models
self-supervision: We use a corpus of text as training. The model is trained to predict the next word.
No need for gold-standard labelling.
We minimize therror in predicting the next word with a cross-entropy loss function
In language modelling, the correct prediction is the next word, which is represented by a one-hot vector (so \(\mathbf{y}_t[w]=1\)).
Thus our loss is \[
L_{CE}(\hat{\mathbf{y}_t}, \mathbf{y}_t) = - \sum_{w\in V} \log \hat{\mathbf{y}_t}[w]
\]
Training
figure adapted from Jurafsky and Martin (2026) Speech and Language Processing
At each position \(t\), take correct word \(w_t\) and hidden state \(h_t\), which encodes information from \(w_1,\ldots,w_{t-1}\)
Compute probability distribution over possible next words
Compute loss for the next token
Use the next correct word (not the predicted word - called teacher forcing) for next position
Calculate total loss as average
Perform backpropagation through time
Generative RNNs
figure adapted from Jurafsky and Martin (2026) Speech and Language Processing
RNN-based language models can also be used to generate text (Generative AI)
Autoregressive generation: Iteratively sample the next work based on previous words
This was one of the inspirations for the autoregressive approach used by LLMs
Algorithm:
Sample a word from the softmax distribution.
Use the start of sentence marker <s> as the first input
The word embedding for the current word is used as input to the network to choose the word for the next time step
The output word is used as input for the next step
Continue until end of sentence marker </s> is sampled
The Encoder-Decoder Model
RNNs were among early examples of the encoder-decoder model
encode an input sequence of length \(n\)
aim to output a sequence of length \(m\), where \(n\) and \(m\) may be distinct
input and output sequences do not necessarily align word-to-word
A kind of sequence-to-sequence network
Applications:
summarization
question answering
machine translation
The Encoder-Decoder Model
Encoder decoder models have three major components
Encoder: transform an input sequence, \(x_{1:n}\) into a corresponding sequence of contextualized representations, \(h_{1:n}\).
RNNs, LSTMs, transformers, etc., can be used as encoders
Context: A vector \(c\) that is a function of \(h_{1:n}\) and represents the “meaning” of the input to the decoder
Decoder: takes the context \(c\) and generates a sequence of hidden states \(h_{1:m}\) that can be used to generate a sequence of output states \(y_{1:m}\)
We can adapt the autoregressive RNN
Recall that the hidden state is calculated as \(\mathbf{h}_t = g(\mathbf{h}_{t-1},\mathbf{x}_t)\) and the output is calculated as \(\mathbf{y}_t = \mathrm{softmax}(\mathbf{h}_{t})\)
For training we add a sentence separation marker after the source text (language 1), and then add the target text (language 2)
Ich bin ein Berliner <s> I am a doughnut
The Encoder-Decoder Model
The Encoder-Decoder Model
Note that the order of the words does not have to be the same
Ich habe ein tolles Buch zu Ende gelesen! <s> I finished a great book!
Simplest version of the decoder
Take the context vector (the last hidden state of the encoder): \(c\)
Use the context vector as its own first hidden state: \(\mathbf{h}_{0}^d = \mathbf{c} = \mathbf{h}_n^{e}\)
Generate outputs autoregressively until end of sentence marker is reached
There are many more sophisticated architectures. For insance, we can make the context vector available at each decoding timestep
In the encoder decoder model as described above, the context \(\mathbf{c}\) must represent everything about the input sequence
Information present at the beginning of the input text may not be well represented in the context vector
The attention mechanism was designed as a way to extract information from all hidden states of the input
With attention, the context vector is no longer equal to the final hidden state of the input, instead it is dynamically calculated as a function of the hidden states of the encoder.
The goal is for the weights to focus on (‘attend to’) a particular part of the source text that is relevant for the token \(i\) that the decoder is currently producing.
Note that the context vector is now generated anew for each decoding step \(i\), i.e., we have \(\mathbf{c}_i\) instead of the same \(\mathbf{c}\) for all steps.
Attention: Relevance?
In order to compute \(\mathbf{c}_i\), our goal is to determine how relevant each encoder hidden state is for the current decoder step.
That is, we want to have a function for the relevance of encoder state \(j\) for decoding step \(i\)\[
\mathbf{relevance}(i, j) = f(\mathbf{h}_{i-1}^d, \mathbf{h}_{j}^e)
\]
The simplest function is the dot product, which equates relevance with vector similarity \[
f(\mathbf{h}_{i-1}^d, \mathbf{h}_{j}^de) = \mathbf{h}_{i-1}^d \cdot \mathbf{h}_{j}^e
\]
We calculate one such score for each of the encoder states and use softmax to convert into a probability distribution \[
\begin{align}
\alpha_{ij} &= \mathrm{softmax}(\mathbf{h}_{i-1}^d \cdot \mathbf{h}_{j}^e)\\
&= \dfrac{\exp(\mathrm{softmax}(\mathbf{h}_{i-1}^d \cdot \mathbf{h}_{j}^e))}
{\sum_k \exp(\mathrm{softmax}(\mathbf{h}_{i-1}^d \cdot \mathbf{h}_{k}^e))}
\end{align}
\]
\(\mathbf{c}_i\) is then calculated as the weighted average over these scores \[
\mathbf{c}_i = \sum_j \alpha_{ij}\mathbf{h}_{j}^e
\]
Attention:
\(\mathbf{c}_i\) thus takes information from the entire encoder into account and is dynamically updated at each step of decoding
Attention
The dot product is a relatively simple way of calculating an attention score
We will return to this topic in the lectures about transformers
Sources
Sources for this lecture include
Dan Jurafsky and James H. Martin (2026) Speech and Language Processing (3rd ed. draft) https://web.stanford.edu/~jurafsky/
Alex Sherstinsky (2020) Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network, Physica D: Nonlinear Phenomena, Volume 404,132306
Andrej Karpathy and Justin Johnson and Li Fei-Fei (2015) Visualizing and Understanding Recurrent Networks, arXiv, https://arxiv.org/abs/1506.02078
Aurélien Géron (2025) Hands-On Machine Learning with Scikit-Learn and PyTorch, O’Reilly
