Picking Winners

Web applications allow us to quickly try out new features, presentations and approaches.  But people are terrible at predicting which changes are beneficial and which ones are neutral or even harmful.  That’s one reason why a systematic approach to the analysis and optimization of changes through controlled experimentation is important.

At it’s simplest, controlled experimentation is just trying different approaches to a problem (which can be as simple as the color used on a web page) and measuring how user’s respond to these changes.

The paper “Online Experimentation at Microsoft” (PDF) was presented at KDD 2009 and provides a great overview with many concrete examples of actual experiments run at Microsoft.  Here’s one example:

The MSN Real Estate site (http://realestate.msn.com) wanted to test different designs for their “Find a home” widget. Visitors to this widget were sent to Microsoft partner sites from which MSN Real estate earns a referral fee. Six different designs, including the incumbent, were tested.
treatmentsA “contest” was run by Zaaz, the company that built the creative  designs, prior to running an experiment with each person guessing  which variant will win.  Only three out of 21 people guessed the winner…

The winner, Treatment 5, increased revenues from referrals by almost 10% (due to increased clickthrough).

In general, the paper documents that even experienced experts can only pick the winners less than 1/3 of the time.  Meaning, the other 2/3 of the time they are recommending changes that are at best neutral or at worst actually harmful.

This is a non-technical paper that provides motivation for taking an experimental approach.  They also describe the many cultural barriers they encountered at Microsoft.  Overall, a very good read.  Highly recommended.

Of course, they recommend a very sophisticated approach.  But the same principles apply in a broad range of situations.  One common activity that falls in to this category is the optimization of landing pages for SEM and SEO efforts.  In these situations you are usually assisted by tools that make it easy to get the statistical analysis right.

The take home message is that successful companies are learning how to fail fast forward rather than getting stuck in endless rounds of paralysis and internal arguments.  Real-world experimentation can be the final arbiter.

via Greg Linden


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: