Select Page

Something to remember as you repeatedly hear “test test test!” and embark on your own split testing for various creatives via Google Analytics etc…

How Not To Run An A/B Test: “Although they seem powerful and convenient, dashboard views of ongoing A/B experiments invite misuse. Any time they are used in conjunction with a manual or automatic “stopping rule,” the resulting significance tests are simply invalid. Until sequential or Bayesian experiment designs are implemented in software, anyone running web experiments should only run experiments where the sample size has been fixed in advance, and stick to that sample size with near-religious discipline.”

Basically, don’t peek at your testing, don’t test for significance and have sample sizes in mind for your tests.

This is insanely important in more emerging areas for creatives like social media or mobile.

This looks like calculus but it’s a good reminder that the observer often influences the test.

Yay science.

via Hacker News