« Open Source Second Life | Main | James Larsson: Pimp My Garbage (hardware hacking) »
July 27, 2007
Open or Closed? The Future of Search
Listening to Jimmy Whales (founder of the Wikipedia) at OSCON 2007 talking about Wikia. Wikia is a completely new organization. Not a sister or subsidiary of Wikipedia.
Building a wedbsite today is so much different than it was before open source. Prior to open source there was a huge financial investment (millions of dollars). Since open source it is much easier to get up and running.
Before Wikipedia there was one way to get encyclopedic information, the printed encyclopedia. Now this encyclopedic information is readily available to everyone, in many lancuages. Wikipedia is much more comprehensive than the printed encyclopedia (Jimmy admits not always as accurate).
Like building the web and the encyclopedia, search needs a shift. Currently there is a handful of proprietary players (Google, Yahoo!, msn, Ask). When you search today, the engine makes an editorial judgement on what is important. The general public doesn't know what drives the results.
The idea with wikia is to change the balance of power. Create search tools, algorithms, and crawlers that are public.
Jeremie Miller (founder of Jabber) is the lead on the project.
The driving principles:
- Transparency - openness in the system and algorithm operation
- Collaboration - everyone can contribute in some way
- Quality - significantly improve the relevancy and accuracy of search results
- Privacy - be aware of what is happening with your personal data
There are things that humans do better and things that machines do better and right now there is too heavy reliance on computers.
Science progresses better when research is freely available. Currently students doing research in search don't have access to the latest technology. One of the reasons that proprietary search companies site for keeping search algorithms secret is that it prevents spammers. This security through obscurity isn't a great idea.
Wikia announced today that it has acquired Grub. It is a distributed crawler project. Something that can be done by vast numbers of end-user machine when there are free CPU cycles and bandwidth.
Jimmy gives a good example of the wikipedia/wikia way of thinking. The example is that you are desigining a restaurant. You know it will have knives for cooking, but people are bad when they have knives. They might stab something os someone. So with this in mind you design the restaurant so that people who eat there must sit in a cage. This makes a bad society. We shouldn't design things thinking about all of the bad things people will do.
Posted by mike at July 27, 2007 9:24 AM