RNN vs LSTM vs GRU: Interview Questions

Why RNN, LSTM, and GRU Still Come Up in Interviews

Even with transformers dominating production NLP, interviewers keep asking about recurrent architectures because the questions test something transformers don't: do you understand why a specific architectural change (a gate, an additive update, a reset mechanism) solves a specific optimization problem. That reasoning skill transfers directly to questions about residual connections, normalization, and other fixes elsewhere in deep learning, which is exactly why interviewers keep this topic in rotation.

This guide covers the four questions that come up together most often: why vanilla RNNs fail on long sequences, how GRU compares to LSTM, what's actually happening inside an LSTM's gates, and how exploding gradients are handled differently from vanishing ones. Each section links a practice problem.

Why Vanilla RNNs Fail on Long Sequences

"Why can't a plain RNN learn dependencies across dozens or hundreds of timesteps?"

A vanilla RNN updates its hidden state at each timestep with h_t = tanh(W_h h_{t-1} + W_x x_t + b), and it's trained with backpropagation through time, which unrolls the network across timesteps and applies the chain rule all the way back to the start of the sequence. That means the gradient flowing back to an early timestep is a product of many repeated factors, one recurrent weight matrix and one activation derivative per timestep in between.

The tanh derivative is at most 1, reached only when the pre-activation is exactly 0, and it shrinks toward zero whenever the pre-activation is large in magnitude, since tanh saturates. Combined with a recurrent weight matrix whose singular values are typically below 1, the repeated per-timestep factors multiply out to something well below 1. Multiply a value below 1 by itself dozens of times and it shrinks toward zero exponentially fast — that's the vanishing gradient problem. In practice it means a vanilla RNN can update its weights based on what happened a few timesteps ago, but the learning signal from something 50 timesteps back is effectively lost by the time it propagates that far, so the network never learns to use it. This single failure mode is the entire reason LSTM and GRU exist. Practice: Vanishing Gradient Problem.

GRU vs LSTM: Which One Do Interviewers Expect You to Know?

"What's the actual difference between a GRU and an LSTM, and when would you pick one over the other?"

Both architectures solve the vanishing gradient problem the same way at a high level: they replace part of the recurrent update with an additive path that lets gradients flow backward without being repeatedly multiplied by a weight matrix and a squashed activation derivative at every step. They differ in how many gates that requires.

LSTM keeps a separate cell state alongside the hidden state and uses three gates to control it: a forget gate decides what to discard from the cell state, an input gate decides what new information to write, and an output gate decides what part of the cell state to expose as the hidden state. GRU simplifies this: it merges the forget and input gates into a single update gate, adds a reset gate that controls how much of the previous hidden state feeds into the candidate update, and drops the separate cell state entirely, so the hidden state alone carries information forward.

The practical trade-off interviewers want you to state plainly: GRU has fewer parameters (two gates instead of three gates plus a cell state), which typically means faster training and lower memory use for the same hidden size, and it performs comparably to LSTM on many tasks. LSTM's extra cell state and third gate give it slightly more expressive control over what to remember versus forget, which can matter on tasks with very long-range dependencies or larger datasets where the extra capacity is worth the cost. Neither one strictly dominates — this is a real engineering trade-off, not a solved question, and stating it that way is usually more convincing than picking a "winner." Practice: GRU vs LSTM.

Inside the LSTM Gates

"Walk me through what each LSTM gate actually computes."

An LSTM's power comes from separating the cell state, c_t, from the hidden state, h_t, and updating the cell state additively instead of through repeated matrix multiplication. Concretely, at each timestep:

• Forget gate: f_t = sigmoid(W_f · [h_{t-1}, x_t] + b_f) decides, per dimension, how much of the previous cell state to keep.
• Input gate: i_t = sigmoid(W_i · [h_{t-1}, x_t] + b_i) decides how much of a new candidate value to write in.
• Candidate values: c̃_t = tanh(W_c · [h_{t-1}, x_t] + b_c) proposes what that new information could be.
• Cell state update: c_t = f_t * c_{t-1} + i_t * c̃_t combines the two using elementwise multiplication and addition, not another matrix multiply.
• Output gate: o_t = sigmoid(W_o · [h_{t-1}, x_t] + b_o) and h_t = o_t * tanh(c_t) decide what part of the cell state becomes the exposed hidden state.

The detail interviewers are actually listening for is the cell state update itself: because c_t is produced by elementwise multiplication and addition rather than a matrix multiplication followed by a nonlinearity, gradients can flow backward through the cell state across many timesteps with far less shrinkage — sometimes called the "constant error carousel." The gates don't eliminate the vanishing gradient problem entirely, but they give the network a learned mechanism to preserve information across long gaps whenever the forget gate decides that information is still relevant. Practice: LSTM Gating Mechanism.

Exploding Gradients and Gradient Clipping

"Your RNN's loss just became NaN a few epochs into training. What happened, and what's the standard fix?"

Exploding gradients are the mirror image of vanishing gradients: instead of the repeated multiplicative factors during backpropagation through time shrinking toward zero, they grow, typically because the recurrent weight matrix has singular values greater than one. Instead of the gradient vanishing, it compounds into extremely large values, causing huge, unstable parameter updates that can overshoot good solutions entirely and, in the worst case, produce NaN losses.

The standard fix is gradient clipping, applied during training right before the optimizer step. The most common version, clip-by-norm, rescales the entire gradient vector so its L2 norm doesn't exceed a chosen threshold, preserving the gradient's direction while capping its magnitude. A simpler variant, clip-by-value, clamps each individual gradient component to a fixed range independently. Clip-by-norm is generally preferred because it keeps the update direction intact instead of distorting it.

It's worth being explicit that clipping is a mitigation for exploding gradients specifically, not a fix for vanishing gradients — capping a large gradient from growing further does nothing for a gradient that's already shrunk toward zero. That asymmetry, and knowing which technique addresses which failure mode, is exactly the kind of distinction interviewers use to separate a memorized answer from real understanding. Practice: Exploding Gradients and Gradient Clipping.

How to Prepare

1. Be able to state the vanishing gradient problem as a chain of repeated multiplications through time, not just "RNNs forget things" — the mechanism is what interviewers are checking for.
2. Practice sketching the LSTM cell state update from memory; the additive combination of the forget and input gates is the single most important detail in this entire topic.
3. Know the GRU-versus-LSTM trade-off as parameter count and expressiveness, not as one architecture being strictly better than the other.

For the rest of the concepts this track covers, see our neural network interview questions, which links practice problems across architecture, optimization, and regularization topics.