The art of determining what’s relevant, and what’s not.

Question: Who is monitoring the search results?

Jimmy Wales:  Well, right now it’s a very small community, because we’ve just gotten started.  We’re trying to make it as easy to participate as possible.  So the idea would be just as with Wiki, anybody can click on edit and change something and save it, and some people choose to not participate, and some people choose to participate only in a small way.  They occasionally see a spelling error, or realize that they know something that’s not in there, and they’ll add something.  And then other people get really deeply involved, because they find it entertaining and interesting and a worthwhile thing to be doing.  So we expect the same kind of thing.  You know, certainly, one of the things, one of the very useful things that we think lots and lots of people will participate in is, you know, if you do a search and you see a search result that’s just clearly wrong, just clearly spam, it’s just one click to delete it, and then it’s gone for everyone else.  And that’s really pretty powerful.  There’s obviously a lot of interesting implications to that, a lot of social implications, because you know, even in something like Wikipedia where the goal is to write an encyclopedia, you still have to deal with problems with conflict of interest editing, people trying to put links to their own website and all those kinds of things.  In a search engine that problem is intensified even more.  In other words, what do you do to prevent people from going in and deleting all of their competitors links, or adding their own link in lots of inappropriate places.  So part of the design structure of this is to have a social network behind it that when you join, all of your activities like that, all of your editing activities are public activities.  So anyone can see what you’ve done.  And it’d be one click to simply revert everything that you’ve done.  So if you go in and you start deleting hundreds of URLs from your competitor, you’re gonna get reverted and banned very quickly.  We’re not sure yet how that’s gonna work out.  I mean, one of my big design philosophies around social communities and the web is to avoid excessive apriority thinking.  It’s really easy to sit and think of all the reasons why something won’t work.  And then think of all the bad things that people might do, and design everything around preventing the bad things.  But if you do that, I think far too often you end up locking everything down way too much.  You end up trying to prevent problems that would’ve turned out you never would’ve had anyway.  And I think that the better approach is just to say, “Okay, let’s actually go in, let’s keep everything as radically open as we possibly can, so particularly in Version 0.2, people are gonna see that wow, anyone can delete anything at any time.  And there’s really no controls over it whatsoever.  And when we start to think about well, how will we add controls?  In other words, how will we deal with this problem.  Well, we’ll see when it comes.  We’ll see what kinds of actual problems we’re having, and how we think we might best deal with it.  And it’s gonna be an empirical question based on what we see coming out as results.  So it’s an evolutionary process.  Not something that we can pre-design too much.

Question: How do you determine relevance?

Jimmy Wales:  Right, so, I mean, this is an interesting question around sort of in the case of assume that different parties are acting in good faith.  They’re people of good will, who are simply trying to think about what are the-- what should the results be like?  What is correct for a search result?  And I think there’s some key principles that we really haven’t flushed out in like really great detail yet, but when I think about this, I draw pretty heavily on my experience with the Wikipedia model in thinking about things like neutrality, for example.  If you think about what do you need for neutrality, you need-- in search results you need a reasonable mix.  You don’t want to have all results from just the company or something like that.  I mean, you can throw out any random search term and I think it’s gonna be contextually dependent on what it is that you’re searching for.  But if you think of something like General Motors, right?  You search General Motors, what should you find?  What is the correct answer?  Well, probably the official site’s a correct answer.  Probably the Wikipedia entry’s a correct answer.  Probably some reviews of their cars, or something like this, some criticism of them as a corporation, if there’s a good site for that.  There can be a variety of search results.  Now getting to a list of ten, right, might be a little tricky.  And figuring out how exactly do you negotiate amongst figuring out what is-- what should go-- rank where and things like that.  Those are really interesting questions.  What we’re looking at is things like well let’s look at different modes of feedback.  In other words if people-- one of the things you can do is rate URLs one through five.  How does this URL answer this search query?  And we expect that’s one of the most popular things that people will be doing is just simply saying, “This was a good result, or this was not a good result,” rather than deleting or adding.  Those can give some indication, right?  And those would give to the community some sense of where the general public of end users stands on the question.  Which could be overridden in some cases.  I mean, you just have to really take a very editorial approach and say, “We’re not really sure yet.”  I think a lot of people suggest that, well, neutrality isn’t the right principle, because search results could never be neutral, because well, gee, you’re picking ten things and you’re saying, these are the best ten things.  Well, yes, okay.  In a certain sense I can understand where people are coming from  when they say that, but I think we can still look at it and say, “Look, even though we’re constrained to come up with the best ten things, there’s still better and worse answers.  And the better answers it seems to me are the ones that strive for a healthy balance.  So that’s some of the things that we’ve been thinking about.

Date Recorded: 4/30/08