Testing is great. Finding a better solution by gathering data is one of the reasons our species has developed and survived. The same is true for solid results on websites, and using tools such as “A/B testing” to determine the best outcome is a worthwhile endeavor for small businesses, churches, community groups, and non-profits with a website…
At many Web publishers, such decisions can lead to impassioned arguments, fruitless debates, even hurt feelings. But 1-800-Dentist doesn’t leave it to chance or opinion. Instead it runs an experiment. It launches two or more versions of a Web page, and then watches as users react. After thousands of people have visited, one version will have edged out the others with a statistically significant improvement in the number of sign-ups.
via A/B Testing for Websites Redraws the Web | MIT Technology Review.
However, there are downsides to relying solely on tools such as A/B testing and the resulting data when it comes to web design (also from the link above)…
In fact, intensive testing appears to be reshaping what the Web looks like. But the page designs that are succeeding won’t win any awards for art direction, just as listicles don’t win Pulitzers. Even proponents of optimization technology admit it can produce sites with simple, cookie-cutter looks.
But A/B testing is spreading because it’s become easy to do. Optimizely says it can pick a winning design after as few as 100 visits for sites that have never been optimized. In practice, running experiments is often much harder. At 1-800-Dentist, which is based in Los Angeles, Kharkats says he’s testing text and images for several slightly different landing pages and estimates that he will need 150,000 visitors to each in order to detect a difference. That could take months, he says.
So, do some testing (we always do). Let it help you guide your path towards a better website or search campaign or poster design.
But, use your gut and trust your instincts (or the instincts of the agency you hire to help you out). After all, we’re humans, not robots.