theater-ziefen-website/.agents/skills/content-experimentation-best-practices/references/common-pitfalls.md

# Common Experimentation Pitfalls

Avoid these mistakes that invalidate results or lead to wrong conclusions.

## Statistical Mistakes

### 1. Stopping Early (Peeking)

**The problem:** Checking results daily and stopping when you see significance.

**Why it's wrong:** Statistical significance fluctuates. At any point during a test, you might see "significance" that disappears with more data. This is called the "peeking problem" or "repeated significance testing."

**The fix:**
- Pre-calculate required sample size
- Commit to running until you reach it
- If you must peek, use sequential testing methods that account for multiple looks

### 2. Underpowered Tests

**The problem:** Running tests without enough traffic to detect realistic effect sizes.

**Why it's wrong:** You'll conclude "no difference" when there actually is one—you just couldn't detect it.

**The fix:**
- Calculate required sample size before starting
- Be realistic about minimum detectable effect (can you act on a 0.5% improvement?)
- If traffic is low, test bigger changes

### 3. Multiple Comparisons

**The problem:** Testing many variants or metrics and celebrating any that reach significance.

**Why it's wrong:** With 20 metrics, you expect 1 false positive at 95% confidence—by chance alone.

**The fix:**
- Define ONE primary metric before starting
- Use Bonferroni correction or similar for multiple comparisons
- Treat secondary metrics as directional, not conclusive

### 4. Ignoring Segments

**The problem:** Only looking at aggregate results.

**Why it's wrong:** Simpson's Paradox—overall winner might be loser for your key segments.

**The fix:**
- Always segment by device, traffic source, user type
- Check if results are consistent across segments
- If segments differ dramatically, investigate why

## Design Mistakes

### 5. Testing Too Many Things

**The problem:** Changing headline, image, CTA, and layout simultaneously.

**Why it's wrong:** You won't know which change caused the result. And each variable multiplies required sample size.

**The fix:**
- Test one variable at a time (A/B testing)
- If testing multiple, use proper multivariate testing with adequate sample size
- Prioritize highest-impact changes first

### 6. Vague Hypothesis

**The problem:** "Let's see if this new design is better."

**Why it's wrong:** Without a hypothesis, you can't learn WHY something worked (or didn't).

**The fix:**
- State: "We believe [change] will [impact metric] because [reasoning]"
- Even if you're wrong, you learn something

### 7. No Control

**The problem:** Changing the control during the test, or not having one.

**Why it's wrong:** You need a stable baseline to compare against.

**The fix:**
- Never modify the control mid-test
- If you must change it, start a new test
- Document exactly what the control is

## Execution Mistakes

### 8. External Contamination

**The problem:** Running a test during a sale, holiday, or major event.

**Why it's wrong:** External factors affect both variants differently, contaminating results.

**The fix:**
- Avoid tests during unusual periods
- If unavoidable, note it and extend the test past the event
- Compare to the same period historically

### 9. Selection Bias

**The problem:** Testing on a non-representative sample (e.g., only logged-in users).

**Why it's wrong:** Results won't generalize to your full audience.

**The fix:**
- Test on representative traffic
- Be explicit about who's included/excluded
- Note limitations when reporting results

### 10. Implementation Bugs

**The problem:** Variants don't render correctly, tracking fires incorrectly, assignment is biased.

**Why it's wrong:** You're not testing what you think you're testing.

**The fix:**
- QA both variants thoroughly before launch
- Verify tracking events fire correctly
- Check assignment distribution matches weights

## Interpretation Mistakes

### 11. Celebrating Trivial Wins

**The problem:** Implementing a change because it was "statistically significant" even though the effect was tiny.

**Why it's wrong:** Statistical significance ≠ practical significance. A 0.01% improvement isn't worth the complexity.

**The fix:**
- Define minimum meaningful effect before starting
- Consider implementation cost vs. benefit
- Don't over-optimize

### 12. Ignoring Confidence Intervals

**The problem:** Only reporting point estimates ("5% improvement!").

**Why it's wrong:** The true effect could be anywhere in the confidence interval.

**The fix:**
- Report confidence intervals: "5% improvement (95% CI: 2%-8%)"
- Base decisions on the lower bound for conservative estimates
- Wider intervals = more uncertainty

### 13. Not Documenting Learnings

**The problem:** Running tests but not recording what you learned.

**Why it's wrong:** You'll repeat mistakes, forget context, lose institutional knowledge.

**The fix:**
- Document every test: hypothesis, results, learnings
- Include what surprised you
- Build a searchable knowledge base

## Organizational Mistakes

### 14. HiPPO (Highest Paid Person's Opinion)

**The problem:** Running experiments but ignoring results when leadership disagrees.

**Why it's wrong:** Defeats the purpose of data-driven decision making.

**The fix:**
- Get buy-in before testing that results will be honored
- Present data clearly to stakeholders
- Frame as "learning" not "winning/losing"

### 15. Testing Everything

**The problem:** Running experiments on trivial changes that don't matter.

**Why it's wrong:** Wastes resources, creates testing fatigue, delays important experiments.

**The fix:**
- Prioritize tests by potential impact
- Not everything needs a test—use judgment for low-risk changes
- Focus experimentation resources on high-value decisions

### 16. Sample Ratio Mismatch (SRM)

**The problem:** The actual traffic split doesn't match the intended split (e.g., you expect 50/50 but observe 52/48).

**Why it's wrong:** SRM is a strong signal of an implementation bug — broken randomization, bot contamination, or redirect issues. Results from experiments with SRM cannot be trusted.

**The fix:**
- Check the actual split ratio against expected before analyzing results
- Use a chi-squared test to detect statistically significant mismatches
- If SRM is detected, investigate the root cause before drawing any conclusions
- Common causes: bot traffic, browser redirects dropping users, bucketing bugs

### 17. Novelty and Primacy Effects

**The problem:** Users react differently to new designs initially, and the effect fades over time.

**Why it's wrong:** Short experiments may show inflated effects that don't persist. Returning users may click more simply because something looks new.

**The fix:**
- Run experiments for at least 2 full business cycles
- Segment results by new vs. returning users
- If possible, check whether the effect holds in the second week vs. the first