Big Data Interview Questions: Spark, Partitioning, and Scale

When "It Doesn't Fit in Memory" Is the Question

Every data scientist eventually hits the wall where pandas stops working. Big data interview questions test what you do next — and at companies like Netflix, Uber, and Airbnb, they're asked in nearly every senior DS loop. You don't need to administer a cluster; you need the mental model of distributed computation.

MapReduce: The Foundation Everyone Skips

"Explain MapReduce like I'm a new grad."

Split the data across machines. Map: each machine transforms its chunk into key-value pairs. Shuffle: pairs with the same key are routed to the same machine. Reduce: each machine aggregates its keys. Word count is the canonical example — map emits (word, 1), reduce sums.

The insight interviewers want: the shuffle is the expensive part, because it moves data across the network. Practice: MapReduce Paradigm.

Spark vs Hadoop: The Classic Comparison

Spark won because it keeps intermediate results in memory, while Hadoop MapReduce writes to disk between every stage — a 10-100x difference for iterative work like ML training, where the same data is scanned repeatedly. Add lazy evaluation (Spark builds a DAG of transformations and optimizes it before executing) and a far friendlier API, and you have the modern default. Practice: Spark vs Hadoop MapReduce.

Batch vs Stream Processing

"The product team wants fraud detection. Batch or streaming?"

• Batch processes accumulated data on a schedule — simpler, cheaper, fine for daily reports and model retraining.
• Streaming processes events as they arrive — required when the value of the answer decays in seconds: fraud, alerting, live personalization.

The strong answer names the trade-off (latency versus complexity/cost) and picks based on how fresh the decision needs to be. Practice: Batch vs Stream Processing.

Data Lakes vs Data Warehouses

The interview angle is usually "where would you put X?" — clickstream logs and images belong in the lake; curated revenue tables belong in the warehouse. Practice: Data Lakes vs Data Warehouses.

Partitioning: The Senior-Level Differentiator

"Your Spark job is slow. Walk me through your debugging."

Partitioning answers separate seniors from juniors. Good partitioning means each machine works on a similar-sized chunk and queries can skip irrelevant data (e.g., partition by date). The failure mode is skew — one hot key (one huge customer, one busy day) lands 90% of the data on one executor and the whole job waits for it. Fixes: salting hot keys, repartitioning, or broadcast joins for small tables. Practice: Data Partitioning Strategies.

CAP Theorem (Quick, But Asked)

A distributed store can't be simultaneously Consistent, Available, and Partition-tolerant — and since network partitions will happen, the real choice is C vs A during a partition. Banking picks consistency; a social feed picks availability. Practice: CAP Theorem.

Practice

All 10 big data questions come with the follow-up traps interviewers use — skew, shuffles, and exactly-once semantics included.