CS Analytics: Testing… Testing…

on January 12, 2011

A lot has been made of multivariate and A/B testing with e-commerce sites. Anyone who has spent any amount of time reflecting on branding and marketing knows intuitively those small differences in color, layout and wording have a tremendous impact on conversion rates. Every once in a while, however, someone gives us a great example of the real power of multivariate testing.

In 2007, Obama was trailing by double digits in the Democratic primary. Eric Siroker, Obama’s Director of Analytics, began to experiment with multivariate testing to find the right combination of images, words and banners to close the gap between Obama and his opponents. His goal was simple: get more people to sign up for the campaign’s email list.

Siroker set up a testing campaign that focused on changing the large banner at the top of Obama’s web page and the call to action “button” at the bottom of the page (the button submitted your email and zip code to be added to the mailing list).

According to Siroker:

“We tried four buttons and six different media (three images and three videos). We used Google Website Optimizer and ran this as a full-factorial multivariate test, which is just a fancy way of saying we tested all the combinations of buttons and media against each other at the same time. Since we had four buttons and six different media that meant we had 24 (4 x 6) total combinations to test. Every visitor to the splash page was randomly shown one of these combinations and we tracked whether they signed up or not.”

Using sign up/subscriber rates as the barometer for success, Siroker and his team found one banner button combination to generate a conversion rate that was 40.6% better than their original page. That was a difference of 11.6% of the people signing up for the email list vs. 8.26%. Keep in mind we are dealing with millions hits to this site.

In Siroker’s words:

“Roughly 10 million people signed up on the splash page during the campaign. If we hadn’t run this experiment and just stuck with the original page that number would be closer to 7,120,000 signups. That’s a difference of 2,880,000 email addresses.”

So what?

Each email address on average was responsible for a $21 donation. There were 2,888,000 additional email addresses captured as a result of the multivariate test. Therefore, the adjustments they made in response to their experiment picked them up an extra $60,000,000. No. That is not a typo. $60 MILLION. All because someone considered how images and simple wording affected their campaign and took the time to set up a simple Google Website Optimizer to measure how each combination impacted conversion rates.

Lesson: Test what your gut is telling you. You could be off base. Nobody on the Obama campaign chose the banner/button combination that put up all those numbers. In fact, the majority was in favor of the losing combo. Be willing to listen to your web traffic and let it teach you. A simple A/B or multivariate test might be able to give you all the information you need.