MODELCATALOG

Understand how gating addresses limitations of vanilla RNNs.

NLP / Neural NLP / Intermediate

7. LSTM and GRU: Better Memory

Understand how gating addresses limitations of vanilla RNNs.

01 Foundations Represent language 02 Classical NLP Build baselines 03 Neural NLP Learn sequence 04 Transformers Learn attention 05 Applications Apply models 06 Engineering Operate safely 07 Capstone Integrate

01 Learning objectives

By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.

  1. Objective 1

    Understand the motivation for gates.

  2. Objective 2

    Contrast RNN, LSTM and GRU.

  3. Objective 3

    Know practical tradeoffs.

02 Why vanilla RNN struggles

During BPTT, gradients may repeatedly shrink or grow across many timesteps. This makes long-range dependencies difficult to learn.

03 LSTM

LSTM introduces a cell state plus gates controlling what to forget, what new information to write and what to expose as output.

Mental model:

memory conveyor belt + gates

04 GRU

GRU simplifies the gating structure and has fewer parameters than LSTM. It often trains faster while retaining much of LSTM's ability to model longer dependencies.

05 PyTorch swap 

self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
# or
self.rnn = nn.GRU(embed_dim, hidden_dim, batch_first=True)

06 Pitfalls and nuances

These are the mistakes most likely to create a wrong mental model or a weak implementation.

  1. Pitfall 1

    Treating LSTM/GRU as modern default for all NLP; Transformers dominate many current tasks.

  2. Pitfall 2

    Ignoring sequence length, padding and batching details.

07 Checkpoint questions

Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.

  1. Check 1

    What problem are LSTM gates trying to solve?

  2. Check 2

    Why might GRU be cheaper than LSTM?