LLM Inference Interview Questions: KV Cache to Speculative Decoding

Why Inference Gets Its Own Interview Category

Training gets most of the attention in textbooks, but most engineers working with LLMs day to day spend more time on inference: making a model respond fast enough and cheaply enough to ship. Interviewers, especially at companies serving models in production, test whether you understand the handful of techniques that make autoregressive generation practical at all. This guide covers four of them.

These questions tend to be asked as a connected story rather than isolated trivia: an interviewer might start with "why is generation slow?", let you land on the sequential, one-token-at-a-time nature of autoregressive decoding, and then walk you through each of the fixes below as the natural next question. Treating them as a connected narrative — the problem, then each technique's specific fix — tends to land better than presenting four disconnected facts.

The KV Cache: Why Generation Doesn't Redo All Its Work

"An LLM generates one token at a time. Why isn't generating the 500th token 500 times more expensive than generating the first?"

Without caching, generating each new token would require recomputing the key and value vectors for every previous token in the sequence from scratch — because self-attention needs those vectors for every token to compute the next one. The KV cache avoids this by storing each token's key and value vectors the first time they're computed, so generating a new token only requires a forward pass for that one new token, reusing the cached keys and values for everything before it.

The cost side of this trade-off is what interviewers push on next: the cache grows linearly with sequence length and with batch size, and at long context lengths or high concurrency it can become the dominant consumer of GPU memory during serving — which is exactly the problem techniques like grouped-query attention and cache eviction strategies exist to address.

A sharp follow-up: "why does the cache take so much memory in the first place?" Every layer of the model needs its own cached keys and values for every token, so the total cache size scales with the number of layers, the number of attention heads, the head dimension, the sequence length, and the batch size — multiplied together. That's why techniques like multi-query and grouped-query attention, which share key and value projections across multiple query heads instead of giving every head its own, exist specifically to shrink the cache without changing the model's overall behavior much. Being able to explain the cache's memory footprint as a product of those factors, rather than a vague "it uses a lot of memory," is what a strong answer sounds like.

Quantization: Trading Precision for Memory and Speed

"Your model doesn't fit on the GPU you have available. What's the first lever you reach for, before buying more hardware?"

Quantization reduces the numeric precision used to represent a model's weights — and sometimes its activations — from the 16-bit floating point typically used after training down to 8-bit or 4-bit integers. Lower precision means each parameter takes less memory and, on hardware with fast low-precision arithmetic, can also mean faster computation.

The honest trade-off interviewers want stated plainly: quantization introduces approximation error relative to the full-precision model, and how much that error matters depends on the technique and the task — well-implemented post-training quantization methods keep most tasks close to full-precision quality, but it isn't free, and very aggressive quantization can measurably hurt performance on tasks that need precise reasoning.

It's also worth distinguishing quantizing weights only from quantizing both weights and activations. Weight-only quantization is the more conservative, more common choice for serving, since weights are static and can be quantized carefully offline with calibration data, while activations vary per input and are more sensitive to precision loss. Interviewers in inference-heavy roles sometimes ask specifically about this distinction to check whether you've actually deployed a quantized model or only downloaded one someone else already quantized.

Flash Attention at Inference Time

"You already use Flash Attention to speed up training. Does it help inference too, and why?"

Flash Attention's core idea — computing exact attention while minimizing how much data moves between GPU high-bandwidth memory and fast on-chip memory — applies at inference just as it does at training, particularly for the initial prompt-processing step (often called the "prefill" phase) where attention still has to be computed over the full input sequence. The memory savings also matter directly for serving: less memory spent moving intermediate attention data leaves more headroom for the KV cache described above, which is often the real bottleneck when serving many concurrent requests.

This is a good place to connect the two techniques explicitly if asked to compare them: the KV cache avoids recomputing work across generation steps, while Flash Attention makes the work that does have to happen — especially the full-sequence attention pass during prefill — faster and lighter on memory. They solve different parts of the same overall latency problem, and production serving stacks use both together rather than choosing one over the other. Practice: Flash Attention.

Speculative Decoding: Generating Faster Without Changing the Output

"Token-by-token generation is inherently sequential and slow. How would you speed it up without changing what the model actually outputs?"

Speculative decoding pairs the large target model with a much smaller, faster "draft" model. The draft model proposes several tokens ahead cheaply; the target model then verifies all of those proposed tokens in a single parallel forward pass, which is much more efficient per token than generating them one at a time sequentially. Wherever the target model agrees with the draft model's guess, that token is accepted; the first point of disagreement is corrected by resampling from the target model's own distribution, and generation continues from there.

The detail that makes this a strong answer rather than a hand-wavy one: because every accepted token is checked against — and every rejected token is resampled from — the target model's actual distribution, the final output has exactly the same distribution as running the target model alone. The speedup comes from parallelizing verification, not from approximating the output.

A natural follow-up: "what happens if the draft model is bad at predicting the target model?" The technique still produces the correct output distribution regardless of how good the draft model is — that guarantee doesn't depend on draft quality — but a draft model that rarely agrees with the target model gives you little to no speedup, since most proposed tokens get rejected and corrected one at a time, which is no better than standard decoding. The practical art of speculative decoding is picking (or training) a draft model that's cheap to run and agrees with the target model often enough to be worth the overhead of running both. Practice: Speculative Decoding.

How to Prepare

1. Be able to explain why the KV cache exists before discussing anything that optimizes it — cache eviction, quantized caches, and grouped-query attention all only make sense once you know what problem the cache solves.
2. For quantization, always pair the benefit (memory, speed) with the honest cost (approximation error) — interviewers notice when a candidate only presents the upside.
3. For speculative decoding, lead with "same output distribution, different amount of parallel work" — that's the one sentence that shows you understand it rather than memorized it.

For architecture, fine-tuning, and evaluation questions that round out this track, see the full LLM engineer interview questions hub.