analytics · Alpha — design partners only

A/B experiment runner

Designs, ships, monitors, and calls A/B experiments. Stops underperformers automatically.

What it does

End-to-end A/B test runner. Designs the variant brief, ships variants to your analytics or A/B platform (Statsig, Posthog, GrowthBook, Optimizely), monitors statistical significance daily, automatically stops the loser when significance threshold is hit, writes the experiment recap document.

Inputs

A/B platform API key. Hypothesis + variants. Significance threshold (default 95%). Min sample size + max runtime.

Outputs

Live experiment with monitored variants. Daily significance update. Auto-stop on result. Recap doc on completion.

Cost per run

~$25 per experiment (design + monitoring + recap).

Models used

Claude Opus for hypothesis writing + recap. GPT-4o for variant design diversification. Custom statistical engine for significance calc.

Best for

Brands with enough traffic to run meaningful experiments (~10K visitors/month minimum to a single page). Conversion-rate-driven businesses.

Not for

Low-traffic brands where experiments would take 6+ months to reach significance — focus on qualitative research instead.

Common questions

Does it require a paid A/B platform?

No — works with Posthog (free tier covers most) or GrowthBook (free + open source). We integrate with paid platforms (Statsig, Optimizely) too if you have them.