Fine-Tuning Interview Questions: LoRA, PEFT and RLHF

Why Fine-Tuning Questions Got More Specific

A few years ago, "how do you fine-tune a model?" had one answer: update all the weights on your task-specific data. In 2026, interviewers expect you to know why that's usually not how it's done anymore, and to be fluent in the alternatives — parameter-efficient fine-tuning in general, LoRA specifically, and the two dominant approaches to alignment, RLHF and DPO. This guide covers all four, with the depth interviewers actually probe for.

These questions tend to come from people who fine-tune models regularly, so vague, textbook-level answers get follow-up pressure fast. The good news is that the underlying ideas are genuinely learnable in an afternoon — the sections below build from the most basic trade-off (update everything or update a little) up to the two competing ways of teaching a model to follow instructions well.

Full Fine-Tuning vs Parameter-Efficient Fine-Tuning

"Why wouldn't you just fine-tune the whole model on your task-specific data?"

Full fine-tuning updates every parameter in the model, which sounds straightforward but is expensive in a way that surprises people who haven't done it: the optimizer (Adam, in most cases) needs to store multiple state values per parameter, so training memory can be several times the size of the model itself. It also produces a complete new copy of the model for every task, which doesn't scale if you need to support many customers or use cases from one base model.

Parameter-efficient fine-tuning (PEFT) is the umbrella term for methods that freeze the pretrained weights and train only a small number of additional parameters — a few percent of the original count or less. The frozen base model stays shared across tasks, only the small trained add-ons differ, and both training memory and storage per task drop dramatically. The trade-off interviewers want you to name honestly: PEFT methods sometimes trail full fine-tuning slightly on the hardest tasks, but for the vast majority of real-world use cases the gap is small enough that the efficiency win dominates the decision.

It also helps to name the operational reason PEFT won out beyond raw memory savings: serving many fine-tuned variants of the same base model is far more practical when each variant is a small set of add-on weights rather than a full model copy, since you can keep one copy of the base model resident and swap small adapters in and out per request. Full fine-tuning doesn't give you that option — every task-specific model is independent, full-sized, and has to be hosted or loaded separately.

LoRA: The Default PEFT Method

"Explain how LoRA reduces the number of trainable parameters without hurting the model's capacity too much."

LoRA (Low-Rank Adaptation) freezes the original weight matrices entirely and, next to selected layers — typically the query and value projection matrices in attention — adds two small matrices whose product approximates the update that full fine-tuning would have made. If the original weight matrix is large, LoRA represents its update as the product of a much smaller pair of matrices with an inner rank r, a hyperparameter you choose. Only those two small matrices are trained; the original weights never move.

Two details interviewers listen for: first, the rank r controls a real trade-off — higher rank captures more task-specific adaptation but adds more trainable parameters, and in practice fairly small ranks work well for most tasks. Second, at inference time the low-rank update can be mathematically merged back into the original weight matrix, so a LoRA-adapted model runs at exactly the same speed as the base model — there's no extra forward-pass cost, unlike some other adapter methods that add layers you can't merge away.

A good follow-up to be ready for: "which weight matrices do you actually apply LoRA to?" You don't have to adapt every weight matrix in the model to get most of the benefit — applying it to the attention projection matrices alone captures a large share of what full fine-tuning would achieve, which is part of why LoRA is cheap: not only is each adapted matrix's update low-rank, but the set of matrices being adapted is also a deliberate subset of the whole model. Candidates who've actually fine-tuned a model with LoRA usually mention that choosing which matrices to adapt is itself a tunable decision, not a fixed rule. Practice: LoRA Fine-Tuning.

RLHF: Aligning a Model to Human Preferences

"You have a model that completes text well but doesn't follow instructions helpfully. How do you fix that?"

This is the RLHF question, even when it isn't phrased with the acronym. Reinforcement learning from human feedback works in stages: first, human annotators rank multiple model outputs for the same prompt from best to worst; those comparisons train a separate reward model to predict which output a human would prefer. Then the language model itself is optimized, historically with a reinforcement learning algorithm like PPO, to produce outputs that score highly under the reward model — with a penalty term that keeps the updated model's outputs close to its starting distribution, so it doesn't drift into degenerate outputs that game the reward model.

This is the process behind the instruction-following behavior of models like InstructGPT and its successors, and it's worth being able to name the three stages — preference data, reward model, RL optimization against that reward model — without hand-waving through the middle one.

Interviewers frequently probe why the KL penalty term is necessary rather than just optimizing the reward directly. Without it, the RL process tends to find degenerate outputs that score well under the reward model's imperfect judgment but don't actually reflect good behavior — a failure mode generally called reward hacking. The penalty keeps the policy anchored close to the distribution of its starting model, trading off some reward-maximization for staying within the space of outputs the reward model was actually trained to judge reliably.

DPO: Alignment Without the Reward Model or RL Loop

"RLHF requires training a reward model and running reinforcement learning, both of which are notoriously fiddly to tune. Is there a simpler way to get the same result?"

Direct Preference Optimization (DPO), introduced in 2023, answers yes. It uses the same kind of human preference data as RLHF — pairs of outputs with one marked as preferred — but instead of training a separate reward model and then doing reinforcement learning, it derives a loss function directly from the preference data that can be optimized on the language model with ordinary supervised-style training. The key theoretical result behind it is that the RLHF objective has a closed-form optimal policy in terms of the reward model, and substituting that relationship back in lets you skip the reward model and RL loop entirely while targeting the same underlying goal.

The practical answer interviewers want: DPO is simpler to implement, has fewer moving parts to tune, and has become a common default for alignment work, while RLHF with an explicit reward model is still used, particularly when the reward signal needs to combine multiple objectives.

Be ready for the "what do you lose" side of that trade too. Because DPO trains directly on a fixed batch of preference pairs rather than sampling fresh generations from an evolving policy during an RL loop, it's less naturally suited to settings where the reward signal needs to be sourced from something other than static preference data — for example, a reward model that also scores factual correctness against an external tool call. RLHF's explicit reward model can, in principle, be swapped out or combined with other signals more flexibly, even though that flexibility comes at the cost of the extra training complexity DPO was designed to remove.

How to Prepare

1. Be able to state, in one sentence each, what full fine-tuning, PEFT, and LoRA specifically trade off against each other.
2. Practice sketching the RLHF pipeline as three labeled stages — preference data, reward model, RL optimization — since that structure is what most follow-up questions probe.
3. Know why DPO removes a step from RLHF, not just that it does; "same objective, fewer moving parts" is the answer interviewers are fishing for.

For inference and evaluation questions that come up alongside fine-tuning in these loops, see the full LLM engineer interview questions hub.