Something to remember as you repeatedly hear “test test test!” and embark on your own split testing for various creatives via Google Analytics etc…

How Not To Run An A/B Test: “Although they seem powerful and convenient, dashboard views of ongoing A/B experiments invite misuse. Any time they are used in conjunction with a manual or automatic “stopping rule,” the resulting significance tests are simply invalid. Until sequential or Bayesian experiment designs are implemented in software, anyone running web experiments should only run experiments where the sample size has been fixed in advance, and stick to that sample size with near-religious discipline.”

Basically, don’t peek at your testing, don’t test for significance and have sample sizes in mind for your tests.

This is insanely important in more emerging areas for creatives like social media or mobile.

This looks like calculus but it’s a good reminder that the observer often influences the test.

Yay science.

via Hacker News

About the Author Sam Harrelson

Digital Marketing and Technology Consultant and Podcaster at Thinking.FM

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s