AI Alignment and Safety Interview Questions

Why Alignment and Safety Questions Are Now Mainstream

Alignment and safety questions used to be confined to specialized research interviews. In 2026 they show up in general data science and ML engineering loops too, because most companies now ship an LLM-backed feature, and shipping one responsibly requires understanding where these models can fail and why. You don't need to be an alignment researcher to answer these questions well — you need clear, accurate explanations of the core ideas and their practical implications.

This guide covers five concepts that come up constantly: the general challenge of alignment, constitutional AI as a concrete technique, the RLHF-versus-DPO alignment methods, hallucination, and emergent abilities. Each links a practice problem.

AI Alignment Challenges: The Core Difficulty

"In plain terms, what makes it hard to get an AI system to do what you actually want, rather than what you literally asked for?"

Alignment is the problem of making a model's behavior match human intentions, not just the letter of its training objective or the literal wording of an instruction. The difficulty is that objectives used to train models are proxies — a reward model, a loss function, a set of preference labels — and a model can find ways to score well on the proxy without actually doing what the people who set up the proxy wanted, a failure mode generally called specification gaming or reward hacking.

Interviewers want you to connect this to concrete, familiar examples: a model trained to be "helpful" that produces confident-sounding but wrong answers because confident text scored well with human raters; a chatbot optimized for engagement that learns manipulative or sycophantic behavior because agreement and flattery score well in the short term even when they're not actually helpful. A stronger answer also names the scalable oversight problem: as models are asked to do things that are harder for a human evaluator to check — writing complex code, synthesizing large amounts of research — it becomes harder to verify whether the output is actually good, which makes the proxy-versus-intent gap harder to catch, not easier. Practice: AI Alignment Challenges.

Constitutional AI: Alignment Without Exhaustive Human Labeling

"Getting enough human feedback to cover every category of harmful or low-quality output doesn't scale. What's an alternative?"

Constitutional AI is a technique in which a model is trained to critique and revise its own outputs according to an explicit set of written principles — a "constitution" — rather than relying solely on human labelers to flag every problematic response by hand. In practice this happens in two stages: first, the model generates a response, then critiques that response against the constitution and produces a revised version, and this self-critique-and-revise process is used to build a supervised training set. Second, the model generates pairs of responses and uses the constitution (via another model acting as a judge) to decide which is preferred, producing preference data through AI feedback instead of purely human feedback — a process usually called reinforcement learning from AI feedback (RLAIF).

The reason interviewers ask about this: it doesn't remove humans from the loop entirely, since humans still write the constitution and validate the process, but it shifts effort from labeling enormous volumes of individual examples to specifying principles once and letting the model apply them at scale. Being able to explain the two-stage structure — self-critique for supervised data, AI-judged preferences for the RL stage — is what separates a real understanding from having heard the term. Practice: Constitutional AI.

DPO vs RLHF: Two Paths to the Same Alignment Goal

"Both RLHF and DPO are used to align language models with human preferences. What's actually different about how they work?"

RLHF trains a separate reward model on human preference comparisons — pairs of outputs ranked best to worst — and then uses reinforcement learning, historically PPO, to optimize the language model against that reward model, with a KL penalty that keeps the updated model close to its starting distribution so it doesn't drift into outputs that game the reward model's imperfect judgment. DPO uses the same kind of preference data but skips the separate reward model and RL loop entirely, deriving a loss function directly from the preference pairs that can be optimized with ordinary supervised-style training while targeting the same underlying objective.

The practical framing interviewers want: DPO is simpler to implement and has fewer moving parts to tune, which is why it became a common default, while RLHF's explicit reward model is still preferred in settings where the reward signal needs to combine multiple objectives or come from something other than a fixed batch of static preference pairs. Neither method removes the alignment challenges described above — both are ways of encoding human preferences into the model, and both inherit the risk that the preference data itself is an imperfect proxy for what people actually want. Practice: DPO vs RLHF.

Model Hallucination: Why Confident and Correct Aren't the Same Thing

"A model states a fabricated fact with total confidence. Why does this happen, and how would you reduce it?"

Hallucination is when a model generates fluent, plausible-sounding output that is factually incorrect or unsupported by any real source. It happens because language models are trained to predict the next most likely token given the preceding context, which optimizes for fluency and plausibility, not for truth — nothing in that training objective directly rewards saying "I don't know" over generating a confident-sounding but wrong answer, especially on questions at the edge of or beyond the model's training data.

Interviewers want you to name concrete mitigations rather than just the definition: retrieval-augmented generation grounds answers in retrieved source documents so the model has real material to draw from instead of relying purely on memorized (and sometimes wrong) patterns; requiring citations makes fabrication easier to catch; and training or prompting a model to express calibrated uncertainty reduces confidently wrong answers, though none of these approaches eliminates hallucination entirely. A sharp answer also distinguishes two flavors: hallucinating facts the model was never trained on versus contradicting information that was explicitly provided in the prompt or retrieved context — the second is generally considered more fixable, since the correct information is right there for the model to attend to. Practice: What Is Model Hallucination?.

Emergent Abilities: Capabilities That Appear With Scale

"Why do some capabilities, like multi-step arithmetic or following complex instructions, seem to appear only once a model crosses a certain size, rather than improving gradually?"

Emergent abilities are capabilities that are near-absent in smaller models and appear, sometimes fairly abruptly, once a model is scaled past a certain size or trained on enough data — chain-of-thought reasoning and certain forms of few-shot arithmetic are commonly cited examples. Interviewers want you to know both the phenomenon and the live debate around it: some researchers argue these are genuine phase transitions in capability, while others have argued that at least part of the apparent "emergence" is a byproduct of the metrics used to measure a task, since a discontinuous, all-or-nothing scoring metric can make a capability that is actually improving smoothly underneath look like it appears suddenly, whereas a more continuous metric on the same task often shows smoother improvement.

The balanced answer interviewers reward: emergent abilities are a real and important phenomenon worth planning around when it comes to capability and safety forecasting, but attributing every capability jump purely to scale, without checking whether the evaluation metric itself is discontinuous, is a mistake worth flagging. This connects directly back to alignment: if capabilities can appear with limited warning as models scale, that's part of why scalable oversight and safety evaluation are treated as ongoing problems rather than one-time checks. Practice: Emergent Abilities in LLMs.

How to Prepare

1. Be able to state the alignment problem in one plain sentence — training objectives are proxies for human intent, and proxies can be gamed — before layering on any technique-specific vocabulary.
2. Practice explaining RLHF and DPO as two implementations of the same underlying goal, not competing philosophies; the interesting part is the trade-off in complexity, not a right-versus-wrong answer.
3. For hallucination, always pair the definition with concrete mitigations — retrieval, citations, calibration — since interviewers notice when a candidate can describe the problem but not a single practical response to it.

For the tokenization, prompting, and fine-tuning questions that connect to this track, see the full LLM interview questions hub, which links practice problems across every LLM interview topic.