RAG and LLM Evaluation Interview Questions

Why RAG and Evaluation Get Paired Together

RAG system design questions and LLM evaluation questions show up together constantly, and for good reason: once you've built a system that retrieves and generates, the natural next question is how you know it's working. This guide covers the RAG pattern itself and the three evaluation concepts interviewers lean on most — BLEU, ROUGE, and perplexity — plus how they fit (and don't fit) into evaluating a RAG system specifically.

This combination shows up so often because it tests two different skills at once: whether you can design a retrieval pipeline, and whether you have the judgment to know that shipping a demo and shipping something measurably reliable are different problems. A candidate who can describe RAG but has no answer for "how do you know it's actually working" hasn't fully closed the loop.

RAG: Retrieval-Augmented Generation

"How would you build a system that answers questions using a knowledge base the model was never trained on, without retraining the model every time the knowledge base changes?"

Retrieval-augmented generation answers this with a three-step pipeline: chunk the source documents and embed each chunk into a vector representation, store those embeddings in a vector store; at query time, embed the incoming question with the same embedding model and retrieve the most similar chunks; then insert the retrieved chunks into the prompt so the model generates its answer grounded in that retrieved context instead of relying purely on what it memorized during training.

The reason RAG usually beats fine-tuning for this kind of problem is worth stating explicitly in an interview: the knowledge base can be updated by re-indexing documents, with no retraining required, answers can cite which chunks they came from, and access control can be enforced at the retrieval layer — a user only retrieves from documents they're allowed to see — rather than being baked irreversibly into model weights.

Interviewers commonly probe the failure modes next: what happens when the retriever returns chunks that are topically similar but don't actually answer the question, and how do you prevent the model from confidently answering anyway using its own memorized knowledge instead of the retrieved context. Naming these failure modes unprompted — and connecting them to the evaluation approach in the closing section below — is what separates a candidate who's read about RAG from one who's debugged it. Practice: What Is RAG?.

BLEU and ROUGE: The Classic Overlap Metrics

"You've generated text and you have a human reference to compare it against. How do you score the similarity automatically?"

BLEU and ROUGE are both n-gram overlap metrics, meaning they compare chunks of consecutive words between the generated text and one or more reference texts, but they were built for different tasks and lean in different directions. BLEU, designed for machine translation, is precision-oriented: it checks what fraction of the n-grams in the generated text also appear in the reference, with a brevity penalty added so a system can't game the score by generating something too short. ROUGE, designed for summarization, is recall-oriented: it checks what fraction of the n-grams in the reference text are captured by the generated output, which fits summarization because missing important reference content is the failure mode that matters most there.

The follow-up interviewers almost always ask: what are the limitations? Both metrics compare surface-level word overlap, so a paraphrase that captures the same meaning with different words can score poorly, and neither metric understands semantic correctness — a fluent, well-overlapping answer that is factually wrong can still score well. That's why modern evaluation increasingly pairs these metrics with semantic or LLM-based judges rather than relying on n-gram overlap alone.

It also helps to know roughly how each is computed, not just what they're used for. BLEU typically combines precision across several n-gram lengths (unigrams through 4-grams are standard) into a single geometric-mean score, then applies the brevity penalty. ROUGE has several variants — ROUGE-N for n-gram recall, and ROUGE-L, which is based on the longest common subsequence between the generated and reference text rather than fixed-length n-grams, which makes it more tolerant of minor word reordering. Knowing that ROUGE isn't a single formula, but a family of related metrics with different variants used for different purposes, is a detail that tends to come up when interviewers dig one level deeper. Practice: BLEU and ROUGE Metrics.

Perplexity: What It Measures and What It Doesn't

"A model has low perplexity on a benchmark dataset. What does that actually tell you, and what doesn't it tell you?"

Perplexity measures how well a language model predicts a held-out sequence of text, computed as an exponentiated form of the average negative log-likelihood the model assigns to each token in that sequence. Lower perplexity means the model assigned higher probability, on average, to the tokens that actually appeared — in other words, the text was less "surprising" to the model. It's a natural fit for comparing raw language modeling quality, such as when deciding between checkpoints during pretraining.

What it doesn't tell you is whether any specific generated response is factually correct, helpful, or safe. A model can be very good at predicting fluent, natural-sounding text — low perplexity — while still confidently generating an answer that is wrong. This is exactly why perplexity is used to track language modeling ability during training and research, but production evaluation of an actual assistant or RAG system leans on task-specific metrics and human or LLM-based judgment instead.

There's a subtlety worth mentioning if pushed: perplexity is only directly comparable between models that share the same tokenizer, since the metric is computed per token and different tokenizers split the same text into different numbers of tokens. Comparing raw perplexity scores across models trained with different vocabularies without accounting for that is a common mistake that interviewers sometimes plant deliberately to see if you catch it. Practice: Perplexity as an Evaluation Metric.

Putting It Together: Evaluating a RAG System Specifically

The strongest answer to "how do you evaluate your RAG system" separates the two halves of the pipeline instead of treating it as one black box. Retrieval quality is checked with precision and recall over whether the chunks that should have been retrieved actually were. Generation quality is checked with a mix of overlap metrics like ROUGE against reference answers where available, and — increasingly common in 2026 — an LLM-based judge or human review that specifically checks whether the answer is grounded in the retrieved context, since a fluent answer that contradicts or ignores the retrieved chunks is a failure mode overlap metrics won't catch. A system can fail because retrieval missed the right document even when the language model itself is strong, so evaluating only the final output hides where the actual problem is.

How to Prepare

1. Practice the RAG pipeline out loud as three concrete steps — chunk and embed, retrieve, augment the prompt — and be ready to explain why it beats fine-tuning for a knowledge base that changes.
2. For BLEU and ROUGE, know which one is precision-oriented and which is recall-oriented, and be ready to name the shared weakness: surface overlap, not meaning.
3. Never present perplexity as a stand-alone quality metric — pair it with what it can't measure (factuality, helpfulness) to show you understand its limits, not just its formula.

For the architecture, tokenization, and inference questions that come up alongside evaluation in these loops, see the full LLM engineer interview questions hub.