MODELCATALOG

Understand recurrent computation before LSTM, attention and Transformers.

NLP / Neural NLP / Intermediate

6. Simple RNN: Forward Pass, Hidden State and Training

Understand recurrent computation before LSTM, attention and Transformers.

01 Foundations Represent language 02 Classical NLP Build baselines 03 Neural NLP Learn sequence 04 Transformers Learn attention 05 Applications Apply models 06 Engineering Operate safely 07 Capstone Integrate

01 Learning objectives

By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.

  1. Objective 1

    Unroll an RNN over time.

  2. Objective 2

    Understand shared weights.

  3. Objective 3

    Recognize many-to-one and many-to-many patterns.

  4. Objective 4

    See how an RNN classifier is implemented in PyTorch.

02 Unrolling through time

The same recurrent cell is reused at every timestep. The weights are shared.

For The food is good:

t1: The → h1

t2: food + h1 → h2

t3: is + h2 → h3

t4: good + h3 → h4

For sentiment classification, the final state (or pooled states) can feed a classifier.

03 PyTorch example 

import torch
import torch.nn as nn

class SentimentRNN(nn.Module):
    def __init__(self, vocab_size, embed_dim=64, hidden_dim=128):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.RNN(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            batch_first=True
        )
        self.fc = nn.Linear(hidden_dim, 1)

    def forward(self, token_ids):
        x = self.embedding(token_ids)
        output, hidden = self.rnn(x)
        final_hidden = hidden[-1]
        return self.fc(final_hidden)

04 Training idea

For binary classification:

logits → BCEWithLogitsLoss → backpropagation → optimizer.step()

Backpropagation across the unrolled sequence is called Backpropagation Through Time (BPTT).

05 Pitfalls and nuances

These are the mistakes most likely to create a wrong mental model or a weak implementation.

  1. Pitfall 1

    Vanishing gradients on long sequences.

  2. Pitfall 2

    Exploding gradients; gradient clipping may be required.

  3. Pitfall 3

    Padding tokens influencing hidden states if masking/length handling is wrong.

06 Checkpoint questions

Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.

  1. Check 1

    Why are RNN weights shared across timesteps?

  2. Check 2

    What is BPTT?