How to Answer "Tell Me About a Time You Found a Discrepancy Reproducing an Analysis"

Why This Question Matters

"Tell me about a time you found a discrepancy reproducing an analysis" comes up at companies like Peloton, Roku, and other subscription and media businesses where quarterly reporting depends on analyses being reproducible months or quarters after they were first run. Interviewers ask it because reproducing someone else's work, or your own past work, and catching a mismatch is one of the clearest tests of methodological rigor a candidate can demonstrate.

This question also reveals how you handle a specific kind of uncomfortable moment: finding that a number leadership has already seen, and possibly acted on, does not hold up under closer inspection. Interviewers want to know whether you investigate that discrepancy fully and report it clearly, or whether you quietly patch it and move on without understanding what actually happened.

There is a process dimension too. A single discrepancy often reveals something broader, such as an undocumented filter, a join that silently duplicates rows, or a metric definition that drifted over time without anyone noticing. The strongest candidates use one discrepancy to fix a systemic weakness, not just the one number in front of them.

The STAR Method for Data Questions

STAR fits this story naturally, since it splits cleanly into finding the mismatch and then tracing it to a root cause.

• Situation: What analysis were you reproducing, and why did it need to be redone?
• Task: What was your responsibility in reproducing it accurately?
• Action: How did you find the discrepancy, trace its cause, and resolve it?
• Result: What was the actual root cause, and what changed as a result?

Spend real time in Action tracing the root cause. Interviewers care most about the diagnostic process, not simply that a number did not match.

What Interviewers Are Really Looking For

1. Attention to Detail

Did you notice the discrepancy at all, or would it have gone unnoticed without a careful, deliberate comparison against the original result?

2. Systematic Root-Cause Tracing

Did you methodically narrow down the cause, such as comparing intermediate outputs step by step, rather than guessing or re-running the whole pipeline repeatedly hoping for a match?

3. Comfort Surfacing Uncomfortable Findings

Were you willing to tell stakeholders that a previously reported number was wrong, even if it had already been used in a decision or presented externally?

4. Durable Fixes, Not Just Patches

Did you fix the underlying cause and document or safeguard against it recurring, rather than just manually correcting the one number in question?

Example Answer Structure

Situation: "At a streaming media company, I was asked to reproduce the prior quarter's subscriber-churn-driver ranking ahead of a board meeting, since the original analyst had left the company and nobody could verify the methodology firsthand."

Task: "My job was to rerun the analysis on the same underlying data and confirm the ranking of churn drivers still held, since the board deck referencing it was due in eight days."

Action: "My reproduction showed a noticeably different ranking, with 'price sensitivity' dropping from the top driver to third place, which was a meaningful enough change that I could not simply report it and move on. I compared intermediate outputs step by step against notes left in the original analyst's project folder and found the original analysis had defined the cohort window using calendar quarters, while I had defaulted to rolling ninety-day windows, which shifted a large batch of holiday-season cancellations between categories. I reran the analysis with the original calendar-quarter definition to confirm it reconciled exactly, then separately tested both window definitions going forward to see which was more stable over time."

Result: "The calendar-quarter version matched the original ranking exactly, confirming the original result was correct and giving the board an accurate, reconciled number in time for the meeting. Because the ambiguity came from an undocumented methodological choice, I wrote a short definitions document specifying the standard cohort window for churn analysis going forward, which the team adopted, and which prevented at least two similar reconciliation scrambles in later quarters that colleagues told me they would have otherwise hit."

Common Mistakes to Avoid

Assuming Your New Number Is Automatically Right

Treating a discrepancy as proof the original analysis was wrong, without first checking your own methodology, is a common and costly mistake.

Patching the Number Without Finding the Cause

Manually adjusting a result to match the original, without understanding why they diverged, leaves the underlying risk in place for the next person who runs the analysis.

Hiding or Softening the Discrepancy

Quietly using whichever number is more convenient, rather than surfacing the discrepancy and its cause to stakeholders, undermines the trust the analysis depends on.

No Documentation Afterward

If your story ends at the fix with no mention of documenting the ambiguity that caused it, the same discrepancy is likely to resurface with the next person who reproduces the work.

Overstating Certainty Before Fully Diagnosing

Reporting a root cause before you have actually confirmed it through a step-by-step comparison risks presenting a guess as a fact.

Preparing Your Stories

Think of a specific time you reproduced an analysis, your own or someone else's, and found a genuine mismatch. Write down the size of the discrepancy, the specific step-by-step process you used to trace its cause, what the actual root cause turned out to be, and what you did to prevent it from recurring.

If you have not encountered this directly, describe a case where you proactively re-validated a recurring report before handing it off to someone else, and explain what safeguards you built in.

Tailoring Your Answer to the Company

At a company with heavy reporting cadences, such as a public company preparing board or investor materials, emphasize the stakes of catching the discrepancy before it reached a wider audience. At an earlier-stage company without much reporting infrastructure yet, emphasize the documentation or process change you introduced to prevent the same ambiguity going forward.

Look at the job description for mentions of data governance, metric definitions, or a semantic layer, since a team using that language wants to hear that you already think about this kind of reproducibility problem systemically.

Handling Follow-Up Questions

Interviewers often probe with:

• "How confident were you in your root cause before reporting it?"
• "What would you have done if you couldn't fully reconcile the discrepancy?"
• "How did you communicate the finding to people who had already seen the original number?"
• "What changed in your process afterward?"

If you were not able to fully resolve a discrepancy in a past situation, say so honestly rather than implying every mystery you have encountered was neatly solved.

Key Takeaways

Finding a discrepancy while reproducing an analysis is a genuine test of rigor, honesty, and process thinking. Trace the cause methodically, surface the finding clearly even when it is uncomfortable, and fix the underlying ambiguity rather than just the one number, and this question becomes a strong demonstration of the reliability that makes data work trustworthy at scale.