Wednesday, December 07, 2005

Organizing chaos and information overload

In his recent post "Organizing Chaos", Peter Rip talks about the value of targeting content:
Targeting equates to value. Targeting specificity increases as volume increases, lifting the value of the entire inventory. It is a virtuous cycle .... More users generate and attract more content. Content expansion increases the value of targeting. Value is extracted by making the content more searchable, and ultimately, reusable.
This reminds me of what Bill Joy said about information overload:
Our lives are overwhelmed by all the information coming at us in a very disorganized way. We're going to hunger for something that will make sense of all the chaos--that will look at all the things happening in the world and filter and order them in a way that's personalized to us. That will be the next great revolution--that is something that doesn't take an index of the dead information on the Net, but the live information of things as they are occurring and as they are relevant to us.
Or what John Doerr said:
Maybe we'll get to 3 billion people on the web and say that what matters to all of us is information, and products, and more. Which is we live in time and we're assaulted by events. And, so, let's just say there's 3 billion events going on at any given time. And if you wanted to compute the cross product of the 3 billion people and the 3 billion events -- 'cause you need to filter very carefully the information that's going to get to this device -- I don't want to be assaulted by anything but the most relevant information ...
Or what Bill Gates said:
Workers are increasingly deluged with ... scads of information ... But finding just what they need when they need it is tough. "The software challenges that lie ahead are less about getting access to the information people need, and more about making sense of the information they have."
Or what John Battelle said:
Through the actions we take in the digital world, we leave traces of our intent, and the more those traces become trails, the more strongly an engine might infer our intent given any particular query ... I expect those trails ... to turn into relevance gold .... Clickstreams are the seeds that will grow into our culture's own memex -- a new ecology of potential knowledge -- and search will be the spade that turns the Internet's soil.
Or what I have said ([1] [2] [3]):
The urgent scaling problem for our users ... is scaling attention. Readers have limited time ... It will become harder and harder to find and discover the gems buried in all the noise. We need to help readers focus, filter, and prioritize.

There is tremendous potential in this flood of data, an opportunity to extract knowledge from the noise. ... There is wisdom in that crowd. All we need to do is find it.

Show me what matters. Help me find what I need .... Where before there was an undifferentiated glut of information, now there is focus. Where before there was noise, now there is knowledge.


Tony G. Thomas said...

An interesting quest, and it does seem to be needed looking forward. Your blog does capture some of the ideas well. Though, I'm not sure the current personalization approach that relies on user history is going to be very fruitful.

To win this battle, the need is to be armed with data on what the community is reading and buzzing about. This data should be available in a timely way, and usable by the reader application to rate/rank content.

The big sites will be able to leverage their user base and larger databases, and smaller sites will need to federate their data to have the same ability to rank content for users.

Just as Google search ranks the mass of content, so should readers, and still let users read every page if desired.

Dr. Chadblog said...

I don't know the answer to this but it seems to me these social software scenarios are usually shaking out to use either a predominantly human or a predominantly algorithmic solution. Its an open question what the right approach is to this issue at the moment for attention optimization, although memeorandum and its clones, which take an algorithmic approach seem to be the hottest new entrant to the field...of course the "algorithmic" approaches are just doing analysis of user intention anyway but its still a declarative versus imperative style difference between them and explicit recommendation systems like Digg.