add skills

2026-05-18 08:39:42 +02:00
parent 92a80e8759
commit 57af0b8386
53 changed files with 10190 additions and 0 deletions
@@ -0,0 +1,109 @@
+# Experiment Design Principles
+
+Well-designed experiments produce actionable insights. Poorly designed ones waste time and can mislead.
+
+## The Experiment Framework
+
+### 1. Hypothesis
+State what you believe and why.
+
+**Bad:** "Let's test a new headline"
+**Good:** "We believe a benefit-focused headline will increase signup rate by 10% because users are currently confused about our value proposition"
+
+Structure: "We believe [change] will [impact metric] because [reasoning]"
+
+### 2. Success Metric
+Define primary and guardrail metrics.
+
+**Primary metric:** The main thing you're trying to improve (conversion rate, engagement time)
+**Guardrail metrics:** Things that shouldn't get worse (bounce rate, page load time)
+
+### 3. Sample Size
+Calculate required sample size before starting.
+
+Factors:
+- Baseline conversion rate
+- Minimum detectable effect (MDE)
+- Statistical significance level (usually 95%)
+- Statistical power (usually 80%)
+
+Use calculators like [Evan Miller's](https://www.evanmiller.org/ab-testing/sample-size.html).
+
+### 4. Duration
+Run tests for full business cycles.
+
+- Minimum: 1-2 weeks (capture weekly patterns)
+- Include weekends
+- Avoid holidays and major events
+- Don't stop early when you see "winning" results
+
+## What to Test
+
+### High-Impact Areas
+- Headlines and value propositions
+- Call-to-action text and placement
+- Form length and fields
+- Pricing presentation
+- Social proof placement
+
+### Lower-Impact (Usually)
+- Button colors
+- Minor copy tweaks
+- Image variations (unless hero)
+- Footer changes
+
+### Test Priority Matrix
+
+| Impact | Effort | Priority |
+|--------|--------|----------|
+| High | Low | Do first |
+| High | High | Plan carefully |
+| Low | Low | Quick wins |
+| Low | High | Avoid |
+
+## Sanity Integration Pattern
+
+```typescript
+// Experiment variant schema
+defineType({
+  name: 'experimentVariant',
+  type: 'object',
+  fields: [
+    defineField({ name: 'name', type: 'string' }),
+    defineField({ name: 'weight', type: 'number', description: 'Traffic allocation (0-100)' }),
+    defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
+  ]
+})
+
+// Experiment document
+defineType({
+  name: 'experiment',
+  type: 'document',
+  fields: [
+    defineField({ name: 'name', type: 'string' }),
+    defineField({ name: 'hypothesis', type: 'text' }),
+    defineField({ name: 'status', type: 'string', options: { 
+      list: ['draft', 'running', 'concluded'] 
+    }}),
+    defineField({ name: 'variants', type: 'array', of: [{ type: 'experimentVariant' }] }),
+    defineField({ name: 'startDate', type: 'datetime' }),
+    defineField({ name: 'endDate', type: 'datetime' }),
+    defineField({ name: 'winner', type: 'string' }),
+    defineField({ name: 'learnings', type: 'text' }),
+  ]
+})
+```
+
+## Avoiding Common Mistakes
+
+### Don't peek and stop early
+Statistical significance can fluctuate. Commit to your sample size.
+
+### Don't test too many things at once
+Each variable multiplies required sample size.
+
+### Don't ignore segmentation
+Winners may differ by device, traffic source, or user type.
+
+### Document everything
+Future you (and your team) will thank you.