BERT vs GPT: The Difference and Interview Questions

Why BERT vs GPT Is Still a Core Interview Question

Even with generative models dominating the headlines, "how does BERT differ from GPT, and why would you choose one over the other" remains one of the most reliable ways interviewers check whether a candidate understands transformer architecture at a level deeper than "it's a big language model." The two architectures were built around different pretraining objectives for different jobs, and that single design choice ripples into tokenization, embeddings, and which real-world tasks each one is actually good at.

This guide covers the three concepts that come up together most often: the BERT-versus-GPT pretraining objectives themselves, how contextual embeddings differ from older static embeddings like word2vec, and how the two dominant subword tokenization schemes — SentencePiece and WordPiece — differ in practice.

BERT vs GPT Pretraining: Bidirectional Understanding vs Autoregressive Generation

"Both BERT and GPT are transformers. What's the actual architectural and training difference between them, and why does it matter?"

BERT is an encoder-only transformer pretrained with masked language modeling: roughly 15% of input tokens are replaced with a mask token (or a random token, or left unchanged, per the original recipe), and the model has to predict the original tokens using context from both directions — everything before and after the masked position. This bidirectional context is exactly what makes BERT strong at understanding tasks: classification, named entity recognition, and producing embeddings that represent a full piece of text, since the model was trained to use the entire surrounding context to fill in gaps. The original BERT paper also included a next-sentence-prediction objective, but later work, notably RoBERTa, showed this contributed little and dropped it without hurting downstream performance.

GPT is a decoder-only transformer pretrained with causal (autoregressive) language modeling: predict the next token given only the tokens that came before it, never the tokens that come after. This requires a causal attention mask that prevents each position from attending to future positions during training, which is exactly what BERT's bidirectional objective does not have. Because GPT is trained to always predict what comes next from left to right, it can generate text one token at a time at inference by repeatedly feeding its own output back in — something BERT was never trained to do.

The framing interviewers want you to land on explicitly: this isn't "GPT is newer and better," it's that the two objectives optimize for different jobs. Bidirectional masked prediction produces excellent representations of text you already have; unidirectional next-token prediction produces a model that's naturally suited to generating new text. Production systems often use both — an encoder-style model for retrieval and classification, a decoder-style model for generation — rather than treating one as a strict upgrade of the other. Practice: BERT vs GPT Pre-training.

Word2Vec vs Contextual Embeddings: Fixing the Polysemy Problem

"Word2vec gives every word one fixed vector. What breaks because of that, and how do modern embeddings fix it?"

Word2vec learns a single, static embedding vector per word by training a shallow model to predict a word from its neighbors (or vice versa) across a large corpus, and that same vector is used every time the word appears, regardless of context. This runs into an obvious limit with polysemy: the word "bank" gets one vector whether it appears next to "river" or next to "loan," even though the two meanings have essentially nothing in common.

Contextual embeddings, produced by passing a full sentence through a transformer like BERT, generate a different vector for the same word depending on its surrounding context, because the representation at each position is computed by attending over the whole input rather than looked up from a fixed table. "Bank" next to "river" and "bank" next to "loan" end up with meaningfully different vectors, because attention lets the surrounding words directly shape each token's representation.

The follow-up interviewers often ask: does this mean word2vec is obsolete? Not entirely — static embeddings are far cheaper to compute and store (a lookup table versus a full forward pass through a transformer), and they're still reasonable for lightweight applications with tight latency or resource budgets where the cost of ignoring polysemy is acceptable. But for anything where the same word carrying different meanings in different contexts actually matters — search, question answering, most modern NLP pipelines — contextual embeddings from a transformer-based model are the default choice today.

It also helps to know how word2vec itself is trained, since interviewers sometimes probe one level deeper: the skip-gram variant learns embeddings by training a shallow network to predict surrounding context words given a center word, while the continuous-bag-of-words (CBOW) variant runs the prediction the other way, predicting a center word from its surrounding context. Both approaches rely on the distributional hypothesis — words that appear in similar contexts tend to have similar meanings — and neither one ever looks at the specific sentence a word appears in once training is finished, which is exactly the limitation contextual embeddings were built to remove. Practice: Word2Vec vs Contextual Embeddings.

SentencePiece vs WordPiece: Two Ways to Build a Subword Vocabulary

"BERT uses WordPiece tokenization and many newer models use SentencePiece. What's actually different between them?"

WordPiece, the tokenizer BERT uses, builds a subword vocabulary similarly to byte-pair encoding — starting from a base vocabulary and iteratively merging pieces — but instead of merging purely by raw frequency, it merges the pair that most increases the likelihood of the training data under the resulting vocabulary. In practice, WordPiece typically operates on text that has already been split on whitespace into words, then breaks unfamiliar or rare words into known subword pieces within that pre-split structure.

SentencePiece is better understood as a tokenizer framework than a single algorithm — it can implement BPE-style or a probabilistic unigram-language-model-style subword segmentation — but its defining feature is that it treats the input as a raw stream of Unicode characters, including whitespace itself (typically represented with a special symbol), rather than requiring a separate whitespace-based pre-tokenization step first. That single design choice matters more than it sounds: it makes SentencePiece straightforwardly applicable to languages like Japanese or Chinese that don't use whitespace to separate words, without needing a language-specific pre-tokenizer, and it makes tokenization fully reversible, since spaces are encoded as ordinary symbols rather than discarded during pre-tokenization and guessed at during detokenization.

Interviewers use this question to check whether a candidate has actually looked past "they're both subword tokenizers" — the useful distinction is the whitespace-handling and language-agnosticism SentencePiece buys you, not which one produces a marginally smaller vocabulary. Knowing that model families like T5 and Llama use SentencePiece-style tokenization while BERT uses WordPiece, and being able to say why that fits each model's design goals, is a strong signal of depth. Practice: SentencePiece vs WordPiece Tokenization.

How to Prepare

1. Lead with the objective, not the label: "masked, bidirectional, good at understanding" versus "causal, unidirectional, good at generating" answers most BERT-vs-GPT questions in one breath.
2. For embeddings, have the polysemy example ready — one word, two meanings, one vector versus two — it's the fastest way to demonstrate you understand why contextual embeddings matter.
3. For tokenization, remember that the interesting difference is how each method handles raw text and whitespace, not just the merge rule each one uses internally.

For the broader tokenization, attention, and fine-tuning questions that build on these fundamentals, see the full LLM interview questions hub, which links practice problems across every LLM interview topic.