Tuesday, March 14, 2006

Beyond the commons: Personalized web search

I probably started salvating when I bumped into this new Teevan et al. paper, "Beyond the Commons: Investigating the Value of Personalizing Web Search" (PDF).

Not only is it about personalized search, but also the authors include Susan Dumais and Eric Horvitz from Microsoft Research.

The paper describes a user study that showed wide variation for the same query in user intent and in user perception of the most relevant search result. That data is used to motivate personalized search as a way to better capture searcher intent.

Some excerpts:
Our analysis shows that rank and rating were not perfectly correlated.

While Web search engines do a good job of ranking results to maximize their users' global happiness, they do not do a very good job for specific individuals.

If everyone rated the same currently low-ranked documents as highly relevant, effort should be invested in improving the search engine’s algorithm to rank those results more highly, thus making everyone happier. However ... our study demonstrated a great deal of variation in their rating of results.

We found that people rated the same results differently because they had different information goals or intentions associated with the same queries.

Rather than improving the results to a particular query, we can obtain significant boosts by working to improve results to match the intentions behind it.
The paper then discusses providing users with explicit customization or more powerful search tools to explicitly specify their intent, but dismisses that approach as too much work, too difficult for searchers to do accurately, and too unlikely to be helpful:
One solution to ambiguity is to aid users in better specifying their interests and intents ... [Ask] users to build a profile of themselves ... [or] help users better express their informational goals through ... relevance feedback or query expansion.

While it appears people can learn to use these techniques ... they do not appear to improve overall success ... We agree with [Jakob] Nielsen, who cites the importance of not putting extra work on the users for personalization.

Even with additional work, it is not clear that users can be sufficiently expressive. Participants in our study had trouble fully expressing their intent even when asked ... In related work, people were found to prefer long search paths to expending the effort to fully specify their query.
The paper goes back at that point to looking toward implicit personalization to improve search, saying that inferring "users' information goals automatically" is the most promising way to disambiguate the "range of intentions that people associate with queries."

Great stuff. I am curious how much of this work at MSR is making it over to MSN.

See also my thoughts on the Teevan et al. SIGIR 2005 paper where the same authors describe a prototype of keyword-based approach to personalizing web search.

See also my previous post, "Attention and life hacking", discussing a NYT article on some of Eric Horvitz's work on attention and linking to some more of Eric's papers.

See also my earlier posts ([1] [2]) about Google's Personalized Search.

See also the announcement of Findory's alpha test of personalized web search.

See also my May 2004 post, "Why do personalized search?"

6 comments:

jeremy said...

(Before I launch into this again, I want to just ask you to please tell me to stop leaving comments if I'm going overboard here. I don't want to fill this space up if I'm not being useful.)

So.. thanks for the post. Good paper. And yeah, Sue Dumais. Venerable IR researcher. Her experience and expertise certainly carries a lot more weight than my own.

However, as I read through it, I realized that the only experimentally verified result in the paper centered around this idea: "For example, the Trailblazer example above did not clarify exactly what kind
of information (e.g., pricing or safety ratings) was sought. This suggests searchers either need help communicating their intent or that search systems should try to infer it.
"

This is essentially the point on which you and I agree as well: Personalization is needed. The issue, however, is how this personalization is done. Should the system help the user more clearly express their intent? (my approach) Or should the system automatically infer that intent, from past user behavior? (your approach)

Teevan/Dumais/Horvitz do not experimentally verify either of these approaches. They only express a preference for the 2nd method, and a preference against the 1st method. They do cite a number of papers to support their preference. But I would have to go and read those papers before I would be able to discuss this more.

However, there is also work from 2002-2003 that shows no real improvement using search histories/behavior for personalization. So what does that say about the history-inferred approach? Well, it's only one datapoint right now, so not much. But I think the jury is really out on the tools/explicit feedback approach, too. I hear all the time that users are too lazy to use tools. But I also read about how when Google introduced the spelling correction feedback tool (force users to explicitly click it, rather than try to "personalize" or correct it automatically), traffic literally doubled. And only after they put the tool in the "right" place, too. People use it like mad. And that seems to lend credence to the tools camp.

jeremy said...

One more comment on this: Ultimately, I don't think this is an either/or choice. Just because you do tools doesn't mean you can't have any automatic, history-based personalization inferences. And vice versa. I could easily imagine a system that does its first ranking based on my history-inferred preferences, and then gives me tools/choices to either (1) return to the global, unpersonalized list, or (2) further personalize the list by giving explicit feedback on my current intent. So these two approaches are not mutually exclusive. It is when a search engine only takes one approach, to the exclusion of the other, that I think it becomes problematic.

Check out this article by Danny Sullivan. The "New York hotel" example is telling. The issue is whether someone is looking for hotels in New York, or that are named New York (such as the one in Vegas). Let's suppose that I do a lot of business travel to New York, so my own personal behavior tends to favor the former interpretation. But now suppose I now travel to Las Vegas once, for a conference. The behavior-based personalized search is going to say "well, all the other times, he meant in New York, so that's what I'll give him again this time." And so the user's query will fail. And the user will have no way of modifying those results, except perhaps to turn off personalized search. And when that doesn't work either, because most hotels in the world are in New York and not named New York, the user will be at a dead end, and frustrated.

Oh, one more thing I just noticed: Look at Google blog search. They are now offering yet another tool/widget: Sort by relevance vs. Sort by date. This is an explicit feedback widget, something Google expects users to go out of their way to use. They're not doing the implicit thing, where they say "oh, this person mostly wants popular, or mostly timely, blog information", and making that decision for you. They are offering you a way of explicitly making your information need clear.

This is good. I hope we see more of this.

Anonymous said...

I agree with Jeremy on the point re: an either / or choice. I'm not sure Greg if you disgagree with this either.

One thing I did want to say is this: I'm finding Findory to have a real knack for finding posts that are right up my street. So whatever you're doing in the background is working. Thanks.

Do you fancy doing a podcast soon? If so, please ping me: alexbarn at thatbigsoftwarecompnayinredmond dot etc.

Alex.

Anonymous said...

Greg,

You may have seen this already, but Raul makes some cases against personalization here. While he is obviously biased, I think he has some valid points. The strongest being the one pointed out by jeremy already which is people are not static, and so personalization will fare poorly when it fails to catch up w/ my changing interests, or on one-off searches.

I do think personalization can be useful, but I would definitely also want an easy way to turn it off as well so I can see the unbiased global list. Also, would you build different profiles for different languages? (e.g., I often search in two different languages, some times more).

Great blog btw, I really enjoy your writings. Please keep up the good work.

yu-kuan

searchfind said...

It is not a trivial game to store a profile. Either you accept an explicit login (e.g. Google, Yahoo!) or you exploit desktop search information

jeremy said...

Just a quick update: Microsoft also now appears to be taking the Vivisimo route, and offering more of a tool-based approach.. eliciting explicit feedback rather than implicit personalization.

And, as a funny example of what happens when things go wrong, here is someone who typed in "endangered species africa" into Google, and gets back Google's automatic "we will organize the world's information for you" set of links to cookie recipes. Lol.

So these things sometimes go wrong. And when they do go wrong, I would have much rather had a tool where I could have given feedback, and said "no" to cookie recipes, rather than have it automatically foisted upon me.