add skills

.agents/skills/content-experimentation-best-practices/SKILL.md

@@ -0,0 +1,39 @@
+---
+name: content-experimentation-best-practices
+description: Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls. Use this skill when planning experiments, setting up variants, choosing success metrics, interpreting statistical results, or building experimentation workflows in a CMS or frontend stack.
+---
+
+# Content Experimentation Best Practices
+
+Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.
+
+## When to Apply
+
+Reference these guidelines when:
+- Setting up A/B or multivariate testing infrastructure
+- Designing experiments for content changes
+- Analyzing and interpreting test results
+- Building CMS integrations for experimentation
+- Deciding what to test and how
+
+## Core Concepts
+
+### A/B Testing
+Comparing two variants (A vs B) to determine which performs better.
+
+### Multivariate Testing
+Testing multiple variables simultaneously to find optimal combinations.
+
+### Statistical Significance
+The confidence level that results aren't due to random chance.
+
+### Experimentation Culture
+Making decisions based on data rather than opinions (HiPPO avoidance).
+
+## References
+
+Start with the reference that matches the current problem, such as design, statistics, CMS integration, or pitfalls. See `references/` for detailed guidance:
+- `references/experiment-design.md` — Hypothesis framework, metrics, sample size, and what to test
+- `references/statistical-foundations.md` — p-values, confidence intervals, power analysis, Bayesian methods
+- `references/cms-integration.md` — CMS-managed variants, field-level variants, external platforms
+- `references/common-pitfalls.md` — 17 common mistakes across statistics, design, execution, and interpretation

.agents/skills/content-experimentation-best-practices/references/cms-integration.md

@@ -0,0 +1,223 @@
+# CMS Integration Patterns
+
+Integrating experimentation with your CMS enables content teams to run tests without developer intervention.
+
+## Architecture Options
+
+### 1. CMS-Managed Variants
+Store experiment variants as content in the CMS.
+
+**Pros:** Content team autonomy, version controlled
+**Cons:** More complex queries, potential publish coordination
+
+```typescript
+// Experiment document
+defineType({
+  name: 'experiment',
+  type: 'document',
+  fields: [
+    defineField({ name: 'name', type: 'string' }),
+    defineField({ name: 'status', type: 'string', options: { 
+      list: ['draft', 'running', 'paused', 'concluded'] 
+    }}),
+    defineField({ 
+      name: 'variants', 
+      type: 'array',
+      of: [{
+        type: 'object',
+        fields: [
+          defineField({ name: 'name', type: 'string' }),
+          defineField({ name: 'weight', type: 'number' }),
+          defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
+        ]
+      }]
+    }),
+    defineField({ name: 'startDate', type: 'datetime' }),
+    defineField({ name: 'endDate', type: 'datetime' }),
+  ]
+})
+```
+
+### 2. Field-Level Variants
+Store variants as fields on the content document.
+
+**Pros:** Simpler queries, content stays together
+**Cons:** Less flexible, schema complexity
+
+```typescript
+defineType({
+  name: 'landingPage',
+  fields: [
+    defineField({ name: 'headline', type: 'string' }),
+    defineField({ 
+      name: 'headlineVariantB', 
+      type: 'string',
+      description: 'A/B test variant (leave empty if not testing)'
+    }),
+    defineField({ name: 'activeExperiment', type: 'string' }),
+  ]
+})
+```
+
+### 3. External Experimentation Platform
+Use dedicated tools (Optimizely, LaunchDarkly, VWO) with CMS content.
+
+**Pros:** Robust analytics, proven platforms
+**Cons:** Additional cost, integration complexity
+
+```typescript
+// CMS stores experiment IDs, platform handles assignment
+defineField({
+  name: 'experimentId',
+  type: 'string',
+  description: 'Optimizely experiment ID'
+})
+```
+
+## Implementation Pattern (CMS-Managed)
+
+### 1. Experiment Schema
+
+```typescript
+defineType({
+  name: 'experiment',
+  type: 'document',
+  fields: [
+    defineField({ name: 'name', type: 'string', validation: r => r.required() }),
+    defineField({ name: 'hypothesis', type: 'text' }),
+    defineField({ 
+      name: 'status', 
+      type: 'string', 
+      options: { list: ['draft', 'running', 'concluded'] },
+      initialValue: 'draft'
+    }),
+    defineField({
+      name: 'variants',
+      type: 'array',
+      of: [{
+        type: 'object',
+        name: 'variant',
+        fields: [
+          defineField({ name: 'id', type: 'string' }),
+          defineField({ name: 'name', type: 'string' }),
+          defineField({ name: 'weight', type: 'number', initialValue: 50 }),
+        ]
+      }],
+      validation: r => r.min(2).error('Need at least 2 variants')
+    }),
+    defineField({ name: 'targetPage', type: 'reference', to: [{ type: 'page' }] }),
+    defineField({ name: 'targetField', type: 'string' }),
+  ]
+})
+```
+
+### 2. Variant Content
+
+```typescript
+// On the page being tested
+defineField({
+  name: 'experimentVariants',
+  type: 'array',
+  of: [{
+    type: 'object',
+    fields: [
+      defineField({ name: 'experimentId', type: 'reference', to: [{ type: 'experiment' }] }),
+      defineField({ name: 'variantId', type: 'string' }),
+      defineField({ name: 'headline', type: 'string' }),
+      // Other variant-specific fields
+    ]
+  }]
+})
+```
+
+### 3. Frontend Assignment
+
+```typescript
+// Middleware or server-side
+function assignVariant(experimentId: string, variants: Variant[]): string {
+  // Check for existing assignment in cookie
+  const cookieKey = `exp_${experimentId}`
+  const existing = getCookie(cookieKey)
+  if (existing) return existing
+  
+  // Random assignment based on weights
+  const rand = Math.random() * 100
+  let cumulative = 0
+  for (const variant of variants) {
+    cumulative += variant.weight
+    if (rand <= cumulative) {
+      setCookie(cookieKey, variant.id, { maxAge: 30 * 24 * 60 * 60 })
+      return variant.id
+    }
+  }
+  return variants[0].id
+}
+```
+
+### 4. Query with Variant
+
+```groq
+*[_type == "page" && slug.current == $slug][0]{
+  ...,
+  "experiment": experimentVariants[experimentId->status == "running"][0]{
+    experimentId->{name, _id},
+    variantId,
+    headline
+  }
+}
+```
+
+## Analytics Integration
+
+### Event Tracking
+
+```typescript
+// Track experiment exposure
+function trackExposure(experimentId: string, variantId: string) {
+  analytics.track('Experiment Viewed', {
+    experimentId,
+    variantId,
+    timestamp: new Date().toISOString()
+  })
+}
+
+// Track conversion
+function trackConversion(experimentId: string, variantId: string, metric: string) {
+  analytics.track('Experiment Conversion', {
+    experimentId,
+    variantId,
+    metric,
+    timestamp: new Date().toISOString()
+  })
+}
+```
+
+### Data Layer
+
+```typescript
+// Push to data layer for analytics tools
+window.dataLayer.push({
+  event: 'experiment_assignment',
+  experiment_id: experimentId,
+  variant_id: variantId
+})
+```
+
+## Best Practices
+
+### Content Team Workflow
+1. Create experiment document with hypothesis
+2. Create variant content
+3. Set status to "running"
+4. Monitor results
+5. Set status to "concluded" and record winner
+
+### Avoid Flicker
+- Assign variants server-side when possible
+- Use CSS to hide content until variant determined
+- Pre-render both variants, show based on assignment
+
+### Clean Up
+- Archive concluded experiments
+- Remove losing variant content
+- Implement winner as default

.agents/skills/content-experimentation-best-practices/references/common-pitfalls.md

@@ -0,0 +1,200 @@
+# Common Experimentation Pitfalls
+
+Avoid these mistakes that invalidate results or lead to wrong conclusions.
+
+## Statistical Mistakes
+
+### 1. Stopping Early (Peeking)
+
+**The problem:** Checking results daily and stopping when you see significance.
+
+**Why it's wrong:** Statistical significance fluctuates. At any point during a test, you might see "significance" that disappears with more data. This is called the "peeking problem" or "repeated significance testing."
+
+**The fix:**
+- Pre-calculate required sample size
+- Commit to running until you reach it
+- If you must peek, use sequential testing methods that account for multiple looks
+
+### 2. Underpowered Tests
+
+**The problem:** Running tests without enough traffic to detect realistic effect sizes.
+
+**Why it's wrong:** You'll conclude "no difference" when there actually is one—you just couldn't detect it.
+
+**The fix:**
+- Calculate required sample size before starting
+- Be realistic about minimum detectable effect (can you act on a 0.5% improvement?)
+- If traffic is low, test bigger changes
+
+### 3. Multiple Comparisons
+
+**The problem:** Testing many variants or metrics and celebrating any that reach significance.
+
+**Why it's wrong:** With 20 metrics, you expect 1 false positive at 95% confidence—by chance alone.
+
+**The fix:**
+- Define ONE primary metric before starting
+- Use Bonferroni correction or similar for multiple comparisons
+- Treat secondary metrics as directional, not conclusive
+
+### 4. Ignoring Segments
+
+**The problem:** Only looking at aggregate results.
+
+**Why it's wrong:** Simpson's Paradox—overall winner might be loser for your key segments.
+
+**The fix:**
+- Always segment by device, traffic source, user type
+- Check if results are consistent across segments
+- If segments differ dramatically, investigate why
+
+## Design Mistakes
+
+### 5. Testing Too Many Things
+
+**The problem:** Changing headline, image, CTA, and layout simultaneously.
+
+**Why it's wrong:** You won't know which change caused the result. And each variable multiplies required sample size.
+
+**The fix:**
+- Test one variable at a time (A/B testing)
+- If testing multiple, use proper multivariate testing with adequate sample size
+- Prioritize highest-impact changes first
+
+### 6. Vague Hypothesis
+
+**The problem:** "Let's see if this new design is better."
+
+**Why it's wrong:** Without a hypothesis, you can't learn WHY something worked (or didn't).
+
+**The fix:**
+- State: "We believe [change] will [impact metric] because [reasoning]"
+- Even if you're wrong, you learn something
+
+### 7. No Control
+
+**The problem:** Changing the control during the test, or not having one.
+
+**Why it's wrong:** You need a stable baseline to compare against.
+
+**The fix:**
+- Never modify the control mid-test
+- If you must change it, start a new test
+- Document exactly what the control is
+
+## Execution Mistakes
+
+### 8. External Contamination
+
+**The problem:** Running a test during a sale, holiday, or major event.
+
+**Why it's wrong:** External factors affect both variants differently, contaminating results.
+
+**The fix:**
+- Avoid tests during unusual periods
+- If unavoidable, note it and extend the test past the event
+- Compare to the same period historically
+
+### 9. Selection Bias
+
+**The problem:** Testing on a non-representative sample (e.g., only logged-in users).
+
+**Why it's wrong:** Results won't generalize to your full audience.
+
+**The fix:**
+- Test on representative traffic
+- Be explicit about who's included/excluded
+- Note limitations when reporting results
+
+### 10. Implementation Bugs
+
+**The problem:** Variants don't render correctly, tracking fires incorrectly, assignment is biased.
+
+**Why it's wrong:** You're not testing what you think you're testing.
+
+**The fix:**
+- QA both variants thoroughly before launch
+- Verify tracking events fire correctly
+- Check assignment distribution matches weights
+
+## Interpretation Mistakes
+
+### 11. Celebrating Trivial Wins
+
+**The problem:** Implementing a change because it was "statistically significant" even though the effect was tiny.
+
+**Why it's wrong:** Statistical significance ≠ practical significance. A 0.01% improvement isn't worth the complexity.
+
+**The fix:**
+- Define minimum meaningful effect before starting
+- Consider implementation cost vs. benefit
+- Don't over-optimize
+
+### 12. Ignoring Confidence Intervals
+
+**The problem:** Only reporting point estimates ("5% improvement!").
+
+**Why it's wrong:** The true effect could be anywhere in the confidence interval.
+
+**The fix:**
+- Report confidence intervals: "5% improvement (95% CI: 2%-8%)"
+- Base decisions on the lower bound for conservative estimates
+- Wider intervals = more uncertainty
+
+### 13. Not Documenting Learnings
+
+**The problem:** Running tests but not recording what you learned.
+
+**Why it's wrong:** You'll repeat mistakes, forget context, lose institutional knowledge.
+
+**The fix:**
+- Document every test: hypothesis, results, learnings
+- Include what surprised you
+- Build a searchable knowledge base
+
+## Organizational Mistakes
+
+### 14. HiPPO (Highest Paid Person's Opinion)
+
+**The problem:** Running experiments but ignoring results when leadership disagrees.
+
+**Why it's wrong:** Defeats the purpose of data-driven decision making.
+
+**The fix:**
+- Get buy-in before testing that results will be honored
+- Present data clearly to stakeholders
+- Frame as "learning" not "winning/losing"
+
+### 15. Testing Everything
+
+**The problem:** Running experiments on trivial changes that don't matter.
+
+**Why it's wrong:** Wastes resources, creates testing fatigue, delays important experiments.
+
+**The fix:**
+- Prioritize tests by potential impact
+- Not everything needs a test—use judgment for low-risk changes
+- Focus experimentation resources on high-value decisions
+
+### 16. Sample Ratio Mismatch (SRM)
+
+**The problem:** The actual traffic split doesn't match the intended split (e.g., you expect 50/50 but observe 52/48).
+
+**Why it's wrong:** SRM is a strong signal of an implementation bug — broken randomization, bot contamination, or redirect issues. Results from experiments with SRM cannot be trusted.
+
+**The fix:**
+- Check the actual split ratio against expected before analyzing results
+- Use a chi-squared test to detect statistically significant mismatches
+- If SRM is detected, investigate the root cause before drawing any conclusions
+- Common causes: bot traffic, browser redirects dropping users, bucketing bugs
+
+### 17. Novelty and Primacy Effects
+
+**The problem:** Users react differently to new designs initially, and the effect fades over time.
+
+**Why it's wrong:** Short experiments may show inflated effects that don't persist. Returning users may click more simply because something looks new.
+
+**The fix:**
+- Run experiments for at least 2 full business cycles
+- Segment results by new vs. returning users
+- If possible, check whether the effect holds in the second week vs. the first

.agents/skills/content-experimentation-best-practices/references/experiment-design.md

@@ -0,0 +1,109 @@
+# Experiment Design Principles
+
+Well-designed experiments produce actionable insights. Poorly designed ones waste time and can mislead.
+
+## The Experiment Framework
+
+### 1. Hypothesis
+State what you believe and why.
+
+**Bad:** "Let's test a new headline"
+**Good:** "We believe a benefit-focused headline will increase signup rate by 10% because users are currently confused about our value proposition"
+
+Structure: "We believe [change] will [impact metric] because [reasoning]"
+
+### 2. Success Metric
+Define primary and guardrail metrics.
+
+**Primary metric:** The main thing you're trying to improve (conversion rate, engagement time)
+**Guardrail metrics:** Things that shouldn't get worse (bounce rate, page load time)
+
+### 3. Sample Size
+Calculate required sample size before starting.
+
+Factors:
+- Baseline conversion rate
+- Minimum detectable effect (MDE)
+- Statistical significance level (usually 95%)
+- Statistical power (usually 80%)
+
+Use calculators like [Evan Miller's](https://www.evanmiller.org/ab-testing/sample-size.html).
+
+### 4. Duration
+Run tests for full business cycles.
+
+- Minimum: 1-2 weeks (capture weekly patterns)
+- Include weekends
+- Avoid holidays and major events
+- Don't stop early when you see "winning" results
+
+## What to Test
+
+### High-Impact Areas
+- Headlines and value propositions
+- Call-to-action text and placement
+- Form length and fields
+- Pricing presentation
+- Social proof placement
+
+### Lower-Impact (Usually)
+- Button colors
+- Minor copy tweaks
+- Image variations (unless hero)
+- Footer changes
+
+### Test Priority Matrix
+
+| Impact | Effort | Priority |
+|--------|--------|----------|
+| High | Low | Do first |
+| High | High | Plan carefully |
+| Low | Low | Quick wins |
+| Low | High | Avoid |
+
+## Sanity Integration Pattern
+
+```typescript
+// Experiment variant schema
+defineType({
+  name: 'experimentVariant',
+  type: 'object',
+  fields: [
+    defineField({ name: 'name', type: 'string' }),
+    defineField({ name: 'weight', type: 'number', description: 'Traffic allocation (0-100)' }),
+    defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
+  ]
+})
+
+// Experiment document
+defineType({
+  name: 'experiment',
+  type: 'document',
+  fields: [
+    defineField({ name: 'name', type: 'string' }),
+    defineField({ name: 'hypothesis', type: 'text' }),
+    defineField({ name: 'status', type: 'string', options: { 
+      list: ['draft', 'running', 'concluded'] 
+    }}),
+    defineField({ name: 'variants', type: 'array', of: [{ type: 'experimentVariant' }] }),
+    defineField({ name: 'startDate', type: 'datetime' }),
+    defineField({ name: 'endDate', type: 'datetime' }),
+    defineField({ name: 'winner', type: 'string' }),
+    defineField({ name: 'learnings', type: 'text' }),
+  ]
+})
+```
+
+## Avoiding Common Mistakes
+
+### Don't peek and stop early
+Statistical significance can fluctuate. Commit to your sample size.
+
+### Don't test too many things at once
+Each variable multiplies required sample size.
+
+### Don't ignore segmentation
+Winners may differ by device, traffic source, or user type.
+
+### Document everything
+Future you (and your team) will thank you.

.agents/skills/content-experimentation-best-practices/references/statistical-foundations.md

@@ -0,0 +1,150 @@
+# Statistical Foundations
+
+Understanding basic statistics prevents misinterpreting experiment results.
+
+## Table of Contents
+
+- Key concepts
+- Sample size calculation
+- Common statistical mistakes
+- Interpreting results
+- Alternative approaches
+- When to trust results
+
+## Key Concepts
+
+### Statistical Significance
+
+A measure of whether observed differences are likely real or due to chance.
+
+- **p-value < 0.05:** "Statistically significant" at 95% confidence
+- Means: If there were no real difference, there's less than a 5% chance of seeing results this extreme
+- Does NOT mean: The change is important or meaningful
+- **Common misconception:** The p-value is NOT "the probability the result is due to chance." It's the probability of observing data this extreme *assuming* the null hypothesis is true.
+
+### Confidence Interval
+
+A range of plausible values for the true effect.
+
+Example: "Conversion rate increased by 5% (95% CI: 2% to 8%)"
+- Best estimate: 5% improvement
+- Could be as low as 2% or as high as 8%
+- Narrower intervals = more certainty
+
+### Statistical Power
+
+The ability to detect a real effect when it exists.
+
+- Standard: 80% power
+- Higher power = larger sample size needed
+- Low power = might miss real improvements
+
+### Minimum Detectable Effect (MDE)
+
+The smallest improvement worth detecting.
+
+- Smaller MDE = larger sample size needed
+- Be realistic: Can you act on a 0.5% improvement?
+
+## Sample Size Calculation
+
+Before running a test, calculate required sample size:
+
+```
+Required per variant = 16 × σ² / MDE²
+
+Where:
+- σ² = variance (for conversion rate: p × (1-p))
+- MDE = minimum detectable effect (absolute)
+```
+
+For a 5% baseline conversion rate, detecting a 1% absolute lift (5% → 6%):
+- σ² = 0.05 × 0.95 = 0.0475
+- MDE² = 0.01² = 0.0001
+- n = 16 × 0.0475 / 0.0001 = **7,600 per variant**
+- Total: ~15,200 visitors minimum
+
+## Common Statistical Mistakes
+
+### Multiple Comparisons Problem
+
+Testing 10 variants increases false positive rate.
+
+**Solution:** Adjust significance threshold (Bonferroni correction) or use sequential testing methods.
+
+### Peeking Problem
+
+Checking results daily and stopping when significant.
+
+**Why it's wrong:** Significance fluctuates. Early "winners" often regress.
+
+**Solution:** Pre-commit to sample size and duration. Use sequential testing if you must peek.
+
+### Simpson's Paradox
+
+Overall results hide segmented truths.
+
+Example:
+- Overall: Variant B wins
+- Mobile users: Variant A wins
+- Desktop users: Variant A wins
+- How? Different traffic mix per variant
+
+**Solution:** Always segment by major factors (device, traffic source).
+
+### Survivorship Bias
+
+Only analyzing users who completed the funnel.
+
+**Solution:** Include all visitors, not just converters.
+
+## Interpreting Results
+
+### Significant + Meaningful
+Clear win. Implement the change.
+
+### Significant + Trivial
+Statistically different but tiny effect. Consider if worth the complexity.
+
+### Not Significant + Large Effect
+Might be real but underpowered. Extend the test or accept uncertainty.
+
+### Not Significant + Small Effect
+No detectable difference. Either no real effect or test was underpowered.
+
+## Alternative Approaches
+
+### Bayesian A/B Testing
+
+An alternative to traditional (frequentist) hypothesis testing. Bayesian methods provide:
+- **Direct probability statements:** "There's a 95% probability Variant B is better" (more intuitive than p-values)
+- **No peeking problem:** Continuous monitoring is built in — you can check results at any time
+- **Credible intervals:** Directly interpretable as "the true value falls in this range with X% probability"
+
+Bayesian methods are offered by platforms like VWO and are useful when you need to make decisions with limited traffic or want more intuitive reporting for stakeholders.
+
+### Multi-Armed Bandits
+
+Dynamically allocate more traffic to winning variants while still learning:
+- **Thompson Sampling:** Balances exploration (learning) with exploitation (serving the best variant)
+- **Best for:** Ongoing optimization where you want to minimize regret during the test
+- **Trade-off:** Faster convergence to the winner, but less statistical rigor than fixed-allocation A/B tests
+
+Consider bandits for content recommendations, personalization, or situations where the cost of showing a losing variant is high.
+
+### Sequential Testing
+
+For teams that need to monitor experiments continuously:
+- **Group sequential designs** (O'Brien-Fleming, Lan-DeMets) allow pre-planned interim analyses
+- **Always-valid p-values** let you check results at any time without inflating false positive rates
+- Use when you must balance the peeking problem with business pressure to act on results quickly
+
+## When to Trust Results
+
+Checklist before declaring a winner:
+- [ ] Reached pre-calculated sample size
+- [ ] Ran for full business cycle (1-2 weeks minimum)
+- [ ] p-value < 0.05 (or your chosen threshold)
+- [ ] Effect size is meaningful for business
+- [ ] Results consistent across major segments
+- [ ] No external factors contaminated results