MLOps and Model Deployment Interview Questions

Why MLOps and Deployment Questions Are Now Core Interview Material

Building a model that performs well offline is only part of the job. Getting that model into production reliably, cheaply, and fast enough to be usable is a separate skill set, and interviewers increasingly test it directly rather than assuming it's implied by strong modeling ability. This guide covers five concepts that come up constantly in these loops: model serving, the ONNX interchange format, distributed training with DDP, model distillation, and quantization for inference.

These questions tend to be asked as a connected story: how do you train a model across multiple machines efficiently, how do you package it so it isn't locked to one framework, and how do you shrink and serve it so it's fast and cheap enough to actually run in production. Treating them as a pipeline — train, export, compress, serve — tends to land better than treating them as five disconnected facts.

Model Serving: Turning a Trained Model Into a Usable Endpoint

"You've trained a model and it performs well in a notebook. What has to happen before it can handle real production traffic?"

Model serving is the layer that wraps a trained model in an API so applications can send it input and get predictions back, and it introduces a set of concerns that don't exist during training: request batching (grouping multiple incoming requests together to use the GPU efficiently instead of running one at a time), autoscaling (adding or removing serving instances as traffic changes), latency budgets (a prediction that's accurate but takes ten seconds may be useless for an interactive product), and versioning (safely rolling out a new model without breaking clients depending on the old one's behavior). Dedicated serving frameworks such as TorchServe and TensorFlow Serving exist specifically to handle these concerns instead of every team reinventing them.

Interviewers want to hear the trade-off between throughput and latency stated explicitly: larger batches use the GPU more efficiently and increase overall throughput, but waiting to accumulate a larger batch adds latency to any individual request, so production serving systems typically use dynamic batching with a maximum wait time, trading a small amount of latency for a large gain in throughput under load. A strong answer also distinguishes online serving (low-latency, request-by-request, often the interactive product surface) from batch or offline inference (running a model over a large stored dataset on a schedule, where latency per item barely matters and throughput is what counts). Practice: Model Serving: TorchServe and TF Serving.

ONNX: Decoupling the Training Framework From the Serving Stack

"You trained a model in PyTorch, but the team that owns your serving infrastructure standardized on a different runtime. What's the alternative to a full rewrite?"

ONNX (Open Neural Network Exchange) is an open, framework-agnostic format for representing a trained model as a standardized computation graph — the operations, their connections, and the learned weights — independent of the framework used to train it. A model trained in PyTorch can be exported to ONNX and then run through an optimized runtime such as ONNX Runtime on a variety of hardware and deployment targets, without needing the original training framework installed or maintained in production.

The practical reason this matters in interviews: it decouples two decisions that used to be tightly coupled — which framework is best for research and training, and which runtime is fastest or best supported in production. Interviewers listen for you to name the trade-off honestly too: not every custom operation or model architecture exports cleanly to ONNX, and highly novel research architectures sometimes need extra work (or custom operator implementations) to convert, so ONNX export is usually verified and benchmarked against the original model's outputs before being trusted in production, not assumed to be a drop-in replacement automatically. Practice: ONNX Format.

Distributed Training With DDP: Scaling Across Multiple GPUs

"Your model and dataset are too large to train efficiently on a single GPU. What's the standard way to scale training across multiple GPUs?"

PyTorch's DistributedDataParallel (DDP) is the standard data-parallel approach: the full model is replicated on every GPU, each GPU processes a different shard of the training data in parallel, and after each backward pass the gradients computed on every GPU are synchronized (via an all-reduce operation) so that every replica ends up with the same, averaged gradient before the optimizer step. Because every GPU applies the same update, all replicas stay identical throughout training, even though each one only ever sees a fraction of the data in any given step.

Interviewers want the distinction between data parallelism and model parallelism stated clearly: DDP is data-parallel — the model fits on one GPU and is copied, while the data is split — whereas model parallelism instead splits the model itself across devices, which becomes necessary when a single model is too large to fit on one GPU's memory even before considering the data. The two approaches are complementary, not competing, and large-scale training frequently combines both: the model is split across a group of GPUs (model or tensor parallelism), and that group is then replicated data-parallel style across many such groups. A good follow-up to be ready for: what does the all-reduce synchronization cost? It requires communication between GPUs after every backward pass, which is why fast interconnects between GPUs (and nodes) matter enormously for distributed training throughput — a poorly networked cluster can bottleneck on gradient synchronization even with plenty of raw compute available. Practice: Distributed Training: DDP.

Model Distillation: Compressing a Large Model Into a Smaller One

"You have a large, accurate model, but it's too slow and expensive to serve at your traffic volume. How do you get a faster model without starting from scratch?"

Model distillation trains a smaller "student" model to reproduce the behavior of a larger, already-trained "teacher" model, rather than training the student from scratch on raw labels alone. Instead of (or in addition to) the ground-truth labels, the student is trained on the teacher's output distribution — the full set of predicted probabilities across classes, often called "soft labels" — which carries more information than a single hard label, since it reflects how confident the teacher was and which incorrect classes it considered plausible.

The framing interviewers want you to land on: distillation is a knowledge-transfer technique, not a shortcut that produces something for nothing. The student model is meaningfully smaller and faster, which is exactly the point for serving at scale, and for many tasks the accuracy gap between student and teacher is small enough that the latency and cost savings are clearly worth it — DistilBERT is a commonly cited example, retaining most of BERT's performance at a fraction of the size and inference cost. Distillation is also frequently combined with the other techniques in this guide: a distilled model is often quantized on top, and the smaller footprint makes it easier to serve efficiently through a standard serving stack. Practice: Model Distillation.

Quantization for LLM Inference: Trading Precision for Memory and Speed

"A large language model doesn't fit in the GPU memory you have available for serving. Before provisioning more hardware, what's the first thing you'd try?"

Quantization reduces the numeric precision used to represent a model's weights, and sometimes its activations, from the 16-bit floating point typically used after training down to 8-bit or 4-bit integers. Lower precision means each parameter takes less memory, which directly increases how large a model fits on a given GPU, and on hardware with fast low-precision arithmetic it can also mean faster inference.

The honest trade-off interviewers want stated plainly: quantization introduces approximation error relative to the full-precision model, and how much that error costs in output quality depends heavily on the technique and how aggressively precision is reduced. Well-implemented post-training quantization keeps most tasks close to full-precision quality, particularly at 8-bit, while very aggressive settings like 4-bit can measurably hurt performance on tasks that need precise reasoning if not done carefully with proper calibration data. It's also worth distinguishing weight-only quantization, the more common and conservative choice for serving since weights are static and can be quantized carefully offline, from quantizing activations too, which is more sensitive since activation values vary per input at inference time. Practice: Quantization for LLM Inference.

How to Prepare

1. Practice describing the full pipeline out loud — train (possibly distributed), export (possibly to ONNX), compress (distill and/or quantize), serve — rather than treating each technique as an isolated fact.
2. For serving, always pair a technique with its trade-off: batching trades latency for throughput, quantization trades some accuracy for memory and speed, distillation trades some accuracy for size and cost.
3. If you've actually deployed a model to production, lead with that experience — a specific memory constraint you hit, or a latency budget you had to hit, demonstrates more than reciting definitions.

For the tokenization, attention, and fine-tuning questions that connect to this deployment track, see the full LLM interview questions hub, which links practice problems across every LLM interview topic.