Archive for the 'Microsoft' Category

Picking Winners

Web applications allow us to quickly try out new features, presentations and approaches.  But people are terrible at predicting which changes are beneficial and which ones are neutral or even harmful.  That’s one reason why a systematic approach to the analysis and optimization of changes through controlled experimentation is important.

At it’s simplest, controlled experimentation is just trying different approaches to a problem (which can be as simple as the color used on a web page) and measuring how user’s respond to these changes.

The paper “Online Experimentation at Microsoft” (PDF) was presented at KDD 2009 and provides a great overview with many concrete examples of actual experiments run at Microsoft.  Here’s one example:

The MSN Real Estate site (http://realestate.msn.com) wanted to test different designs for their “Find a home” widget. Visitors to this widget were sent to Microsoft partner sites from which MSN Real estate earns a referral fee. Six different designs, including the incumbent, were tested.
treatmentsA “contest” was run by Zaaz, the company that built the creative  designs, prior to running an experiment with each person guessing  which variant will win.  Only three out of 21 people guessed the winner…

The winner, Treatment 5, increased revenues from referrals by almost 10% (due to increased clickthrough).

In general, the paper documents that even experienced experts can only pick the winners less than 1/3 of the time.  Meaning, the other 2/3 of the time they are recommending changes that are at best neutral or at worst actually harmful.

This is a non-technical paper that provides motivation for taking an experimental approach.  They also describe the many cultural barriers they encountered at Microsoft.  Overall, a very good read.  Highly recommended.

Of course, they recommend a very sophisticated approach.  But the same principles apply in a broad range of situations.  One common activity that falls in to this category is the optimization of landing pages for SEM and SEO efforts.  In these situations you are usually assisted by tools that make it easy to get the statistical analysis right.

The take home message is that successful companies are learning how to fail fast forward rather than getting stuck in endless rounds of paralysis and internal arguments.  Real-world experimentation can be the final arbiter.

via Greg Linden

Microsoft and Google Declare War

On Thursday, May 28, 2009 Microsoft and Google officially declared war.

Microsoft announced Bing their new search initiative competing directly with Google’s core business.

Google announced Wave: a new product, platform and protocol that re-imagine communication and collaboration in the cloud.  It has grand ambitions that includes a direct assault on Microsoft’s core business of office communication and collaboration.

Initial reviews of Bing by industry insiders suggest it is competitive with Google’s search and offers some interesting features.  Most also believe these features and function won’t be sufficient by themselves to overcome people’s entrenched familiarity with Google.  Microsoft has anticipated this challenge by also announcing a massive advertising campaign to get people to try their new offering.

Microsoft is betting that search has matured, even becoming something of a commodity.  As such, by offering a comparable product they are able to shift the battle-field to a marketing and branding effort.  This MO is consistent with Microsoft history.  They have never been a first mover or an innovator.  They are an exploiter – a very determined one with deep pockets and patience.  And arguably, Bing is their ‘3rd generation’ of search engine (MSN and Live being the prior 2) — and it took them three tries to ‘get it right’ with Windows.  It will be interesting to see if they can finally grow their market share in search.

Google is not standing still on search – they continue to announce new search offerings at a heady pace.  It’s clear they intend to seriously defend their core business of search.  This highly profitable business is what allows them to make all their other big bets.

And Wave feels like a big bet.  It redefines how people communicate and collaborate.  This directly challenge’s Microsofts traditional paradigm of a ‘computer on every desktop’ in which that desktop computer is the primary repository of one’s information.  In Google’s cloud based future the desktop computer becomes irrelevant.  It people shift to the cloud, it represents a huge threat to Microsoft’s ability to license software stacks running on each of those computers.

Google has made many previous guerrilla attacks with products like Gmail and Google Docs.  But these are really just cloud based implementations of traditional paradigms.  Wave on the other hand is a full frontal assault because it encompasses not only these traditional means of communication and collaboration but also extends to include blogging, micro-blogging (Twitter) and activities currently associated with social networks.

It’s unusual for Google to announce such a grand product in such a relatively immature state.  The timing seems chosen to steal some of Microsoft’s Bing noise.  But it is a grand enough vision that Google will need help from legions of developers to make it happen   It is those legions who are the foot-soldiers in this battle – and they are mercenaries who will go where they see the biggest opportunity.

This battle is going to be played out over many years.  But we’ll likely look back and see these two announcements as a significant milestone in the struggle.

Microsoft Researchers Increase CTR 670% Using Behavioral Targeting

Behavioral targeting has been around for a while in various commercial products.  But now, for the first time, researchers at Microsoft Research Asia have completed an important empirical study demonstrating that it works.  In fact, it works extremely well increasing the click-through rate (CTR) up to 670% with the potential to achieve improvements in excess of 1000%.  Wow.  And kudos to Microsoft for undertaking this as an academic research program and making the results available to all.

They cover a lot of important ground in an academically rigirous way.  There are three key conlusions.

The Basic Assumption Holds

The basic assumption of behavioral targeting is that all the people who click on an ad are more similar to each other than they are to all those people who clicked on other ads.  If you can’t prove this assumption, it’s back to the lab to whip up something new.

But, not to worry, the researchers found that the people who clicked on the same ad are up to 90 times more similar to each other than users who clicked on another add.  Whew!  I guess that’s good news for anyone who has been touting the merits of behavioral targeting.  It’s also intuitively satisfying.  Still, it’s great to have it proven by research.

CTR Can Be Increased by Up to 670%

They used click-through rates as their measure of performance (because it is a readily available measure).  They implemented behavioral targeting by segmenting the users with various strategies and then compared what the CTR would have been with and without the segmentation strategy:

Through studying ads CTR before and after user segmentation for ads delivery, we observe that ads CTR can be improved by as much as 670% over all the ads we collected.

and:

In addition, we notice that if we can further design more advanced BT strategies, such as novel user representation approaches and novel user segmentation algorithms, ads CTR can be further improved beyond 1,000%.

Short Term Search Behavior Gives the Best Results

Finally, the researchers examined several different approaches to implementing behavioral targeting:

Through comparing different user representation strategies for BT, we draw the conclusion that the user search behavior, i.e. user search queries, can perform several times better than user browsing behavior, i.e., user clicked pages. Moreover, only tracking the short term user behaviors are more effective than tracking the long term user behaviors, for targeted ads delivery.

What it Means: A Mobile, Local Perspective

This study was done using logs from users searching, browsing and clicking on the web.  Local and mobile bring additional nuances to the equation.

This report is exciting for us here at Predictabuy because it confirms a lot of our own research which is specifically aimed at understanding user behavior in a mobile, local context.  Our research shows:

  1. short term behavior is also a stronger predictor than long term behavior in a mobile, local context;
  2. situational factors such as location, time of day, day of week and weather are very useful in user segmentation; and
  3. advertising performance benefits from dividing users in to  more segments than the number used in the Microsoft study.

via Greg Linden.

Read the full paper for yourself:”How much can Behavioral Targeting Help Online Advertising?” (PDF)