What Is Research?

March 24, 2009

Concluding notes on the polymath project — and a challenge

Filed under: polymath,Wikis — vipulnaik @ 4:27 pm

In this previous blog post, I gave a quick summary of the polymath project, as of February 20, 2009. The project, which began around February 2, 2009, has now been declared successful by Gowers. While the original aim was to find a new proof of a special case of an already proved theorem, the project seems to have managed to find a new proof of the general case. There’s still discussion on how to clean up, prepare the draft of the final paper, and wrap up various loose ends.

In a subsequent blog post, Gowers gave his own summary of the project as well as what he thinks about the future potential of open collaborative mathematics. Michael Nielsen, who hosted the Polymath1 wiki where much of the collaboration occurred, also weighed in with this blog post.

In Gowers’ assessment, the project didn’t have the same kind of massive participation as he had hoped for. People like Gowers and Terence Tao participated quite a bit, and there were also a number of other people who made important contributions (my own estimate of this is around eight or nine, based on the comment threads, with an additional three or four who made a few crucial comments but did not participate extensively). But it still wasn’t “massive” the way that Gowers had envisaged it. Nielsen felt that, for a project just taking off, it did pretty well. He compared it to the early days of Wikipedia and the early days of Linux, and argued that the polymath project did pretty well compared to these other two projects, even though those projects probably had a lot larger appeal.

Good start, but does it scale?

Before the polymath project began (or rather, before I became aware of it), I wrote this blog post, where my main point was that while forums, blogs and “activity” sound a lot appealing, the greater value creation lies in having reliable online reference material that people can go to.

Does that make me a critic of polymath projects?

Well, yes and no. I had little idea at the time (just like everybody else) about whether the particular polymath project started by Gowers would be a success. Moreover, because Ramsey theory is pretty far from the kind of math I have a strong feel for, I had no idea how hard the problem would be. Nonetheless, a solution within a month for any nontrivial problem does seem very impressive. More important than the success in the project, what Gowers and the many others working on it should be congratulated for is the willingness to invest a huge amount of time into this somewhat new approach to doing math. Only through experimentation with new approaches can we get a feel for whether they work, and Gowers has possibly kickstarted a new mode of collaboration.

The “no” part, though, comes from my strong suspicion that this kind of thing doesn’t scale. (more…)

Advertisements

March 22, 2009

Making subtle points

Filed under: Teaching and learning: dissemination and assimilation — vipulnaik @ 4:05 pm

This year, as part of my duties, I am a College Fellow for the undergraduate Algebra (Hons) sequence at the University of Chicago. I conduct a weekly problem session for the students, which serves as an opportunity to review class material, go over tricky points in homework problems, and introduce students to more examples.

On occasion, I’ve come across the problem of wanting to make a subtle point. Often, my desire to make that particular subtle point seems to simply be an ego issue. I’m not referring to the kind of subtle points that there is universal agreement need to be made — the kind that every book introducing the topic makes explicitly. I’m referring, instead, to subtle points that may be considered pointless both by beginners and experienced people. The beginners consider them pointless because they don’t comprehend the point, while experienced people may consider them pointless because they understand it and don’t see what the fuss is about.

One example is the two definitions of group — one definition that defines a group as a set with one binary operation satisfying a number of conditions, and the other defining a group in terms of three operations (a binary operation, a unary operation, and a constant function) satisfying some universally quantified equations. The latter definition is termed the universal algebraic definition, and its importance lies in the fact that it shows that groups form a variety of algebras, which allows for a lot of general constructions. The former definition is important because it indicates that essentially just one group operation controls the entire group structure. This kind of differentiation may seem pointless, or “yeah, so what?” to a lot of mathematicians.

Another example, that actually occurred in a recent problem session, is that of differentiating between a polynomial and a function. if R is a ring, a polynomial in R[x] gives rise to a function from R to R. However, polynomials carry more information than simply the functions associated with them: in other words, different polynomials can give rise to the same functions (this happens, for instance, over finite rings, though it does not happen over infinite integral domains). I’d made this point a few times in earlier problem sessions, and there was even a homework problem that essentially did this stuff.

So, during a problem session review of the notion of characteristic polynomial, I decided to make this subtle point distinguishing the characteristic polynomial from its associated induced function. I made the point that in order to compute the characteristic polynomial, we need to do a formal computation of det(A - \lambda I_n) over a polynomial ring R[\lambda] rather than simply think of a function that sends \lambda to det(A - \lambda I_n). This is a rather subtle, and in many ways, an apparently useless point (in fact, I don’t know of too many places that make this distinction very carefully at an introductory stage). However, I wanted to make it in order to rub in the difference between a polynomial and its associated function.

I hovered over this point for quite some time, so I guess a reasonable fraction of the students did get it, but at the beginning, one girl simply didn’t see the distinction between the two things, and was honest enough to admit it. So it took a couple of minutes to spell the distinction out.

In this blog post, I want to explore some of the arguments for making subtle points, and some effective ways of doing so without stealing too much attention away from the not-so-subtle-but-more-important point. (more…)

March 17, 2009

The “fair copyright in research works” controversy

Filed under: Culture and society of research — vipulnaik @ 4:38 pm
Tags: , ,

There are competing interests, and there are competing interests. In recent years, there has been a growing Open Access (OA) movement, that advocates that scientific research be published for open access. So, not only should people be able to download scientific papers for free, they should also be able to reuse the results for their own experiments. Creative Commons, an organization that came up the the Creative Commons licenses that allow for creators of original works to give others rights that go beyond fair use in copyright law, has been among the organizations at the forefront of Open Access. They even have a separate branch, called Science Commons, that specifically deals with opening up scientific research.

Science Commons was understandably delighted at the NIH Public Access Policy. According to this policy, the National Institute of Health (a governmental organization of the United States) mandated that all research conducted with NIH grants be released to the public within a year of publication.

Recently, a bill was proposed by John Conyers, a Democratic House Representative from Michigan (a United States state) that would have the effect of scrapping the NIH public access policy, and effectively making it impossible for such policies to be instituted. The act was titled the Fair Copyright in Research Works Act”. Many in the blogosphere and scientific community were up in arms at this act. Consider, for instance, that 33 Nobel Laureates apparently opposed the bill. Or consider two posts by Lawrence Lessig and Michael Eisen in the Huffington Post, suggesting that Conyers’ decision to reintroduce the bill was influenced by the campaign contributions he received from commercial publishers. Conyers’ reply was severely trashed by Michael Eisen, Peter Subers, and Lawrence Lessig. Blog posts by Paul Courant, Molly Kleinman, and numerous others. Clearly, the bill hasn’t gone down well with a lot of open access advocates.

What precisely is the problem?

There are a number of arguments for making open access mandatory rather than voluntary, at least in certain contexts. One is the moral argument: taxpayers are paying for the research, so they have a right to access that research. This argument in and of itself isn’t extremely strong. After all, if we really want to be fair to taxpayers, shouldn’t taxpayers get to vote on how the NIH spends its money in the first place. Also, considering that only U.S. taxpayers are paying for the research, shouldn’t the open access policy make the materials openly available only in the United States, so that other countries do not free-ride off the money of U.S. taxpayers? Further, shouldn’t people who pay more taxes be given greater privilege in terms of the level of access they get? Further, considering that most taxpayers would not be interested in reading medical and health-related research, doesn’t this effectively subsidize researchers at the cost of publishers, while keeping the majority of taxpayers exactly where they are?

The taxpayer argument doesn’t seem to be a very strong argument logically, but I think it carries good moral and emotional overtones, which is why it is touted so much.

A more valid reason is the argument that opening up the findings of research creates very strong positive externalities for further research. Thus, a lot of other researchers can benefit from opened up research and thus benefit science, even though these researchers would not have found it personally worth it to subscribe to the journal. Further, science papers, once freely available, can be used by people who may not have tried them out in the past, such as high school students, people with health problems, and others. (more…)

Academic and journalistic support for Wikipedia

Filed under: Wikipedia — vipulnaik @ 3:24 pm

Seth Finkelstein kindly responds to my blog post Wikipedia criticism, and why it fails to matter. Seth agrees with my basic point — it is hard to influence people away from either reading or editing Wikipedia. However, he entertains the hope that criticism might affect what he calls the “public intellectual” perspective of Wikipedia.

This is probably more likely, but the chances of this are dismal too. The digging up I’ve been doing of references to Wikipedia in books and writings suggests a fairly formulaic description of Wikipedia even by academic intellectuals who should know a lot more.

Here are the typical elements of a description of the online encyclopedia:

  • History beginning: Jimbo Wales started out in 2000 with the idea of a free encyclopedia. They started a “top-down” “expert-led” project called Nupedia that produced few articles. Then, “somebody told Wales about wiki software” and they implemented it and the ordinary people started contributing. (The writers who research facts better usually highlight the co-founder controversy and the fact that the proposal for Wikipedia was originally made by Larry Sanger, many writers omit to mention this).
  • Surprise, surprise, it works: Here’s the paradox. An encyclopedia with nobody in charge, with nobody getting paid and with people supplying volunteer time, is as accurate as Britannica, the expert-written encyclopedia (quote a Nature study by Jim Giles). Surprise, surprise, surprise. Then, use this to prove the favorite theory the author is expounding (this could be “intellectual commons”, “creative commons”, “produsage”, “commons-based peer production”, or some variant of that). (The writers who do more research usually mention that the study was investigative journalism rather than scientific research, and also mention Britannica’s repudiation and Nature’s response to those repudiations.)
  • Wikipedia is not without its flaws (surprise, surprise, surprise). The most popular example here is the John Seigenthaler story. Some more serious researchers go so far as to mention other controversies such as the Essjay controversy.
  • But the fact that anybody can edit Wikipedia is its greatest strength — because what can be undone can be done. Quote this study by researchers at MIT and IBM to prove the point, or talk about some specific anecdotal example.

There are few books written about the phenomenon of Wikipedia on the whole. Most references to Wikipedia are as part of books on something such as Web 2.0, user-generated content, the greatness of the Internet, and the above kind of treatment fits in well with the points the author is trying to make.

Among the authors who have praised Wikipedia but done a more in-depth analysis than the above, I can think of Clay Shirky (in his book Here comes everybody), Axel Bruns (in his book Blogs, Wikipedia, Second Life and Beyond: From Production to Produsage) and Chris Andersen (in his book The Long Tail). Though I think the presentations of each of these authors has ample deficiencies, I believe they’ve made some original contribution each in their analysis of Wikipedia. But the same cannot be said of a lot of people who give this simplistic presentation of Wikipedia, often stretching it out over several pages.

Why so much academic support for Wikipedia?

Contrast the legions of academics who write gloriously about Wikipedia, following and adding to the outline I’ve discussed above: Clay Shirky, Axel Bruns, Chris Andersen, Jonathan Zittrain (The Future of the Internet: and how to stop it), Lawrence Lessig (Code v 2), Cory Doctorow (Content), Yochai Benkler (Wealth of Networks), Tapscott and Williams (Wikinomics), Tom Friedman (The World Is Flat) and many others. Contrast this with the much smaller number of authors who take a genuinely critical stand of Wikipedia: Andrew Keen (who describes his own book, The Cult of the Amateur, as a “polemic” as opposed to a serious and balanced presentation) and Nicholas Carr (The Big Switch). Other critics of Wikipedia, many of whose names I outlined in my earlier blog posts, typically content themselves with writing blog posts. It may also be noted that most of the critics aren’t academic intellectuals who work and teach at universities. There are some librarian critics, such as Karen Schneider, who wrote this piece among others critical of Wikipedia; there are also subject experts who write critically of Wikipedia’s treatment of their particular subjects. But this isn’t the same as academics who’re supposed to be experts on the Internet, collaboration, and emerging trends and on other things that Wikipedia is arguably about, offering cogent negative analyses of Wikipedia.

So why is there such an overwhelmingly strong academic support for Wikipedia?

I suspect this goes down to the way academics write books. A book by an academic, whether written for a select academic audience or for a borader audience, typically has a “point” or a “theme”, and most of the themes of books that include Wikipedia are supportive of some of the ideas for which Wikipedia is a poster child. For instance, if Yochai Benkler wants to write a book arguing for the power of commons-based peer production (CBPP), or Axel Bruns wants to describe the power of produsage, the best way to use Wikipedia is as a publicly visible and easy-to-appreciate example of this. This creates a selection bias for the author to pick those aspects of Wikipedia that further the point and ignore the aspects that do not. To add the appearance of a fair and balanced treatment, things like the Seigenthaler episode can be thrown in.

On the other hand, criticism of Wikipedia doesn’t generally add up to any big point or theme that is exciting to write a book about. At least, not yet. Perhaps, ten years down the line, somebody may write a book on how corporations and organization systemically exploit free labor to produce results, and once this kind of narrative starts gaining a foothold in academia, the standard Wikipedia tale will morph — instead of “Wikipedia is subject to vandalism; however, its openness to editing is its greatest strength” might become “although openness to editing is a strength, Wikipedia is subject to vandalism, edit wars, and a lot of unproductive disputes”. But in order for such a book to be written, the theme has to be big enough, or thick enough, to fill an academic book.

Another related factor is “academic herding” — the tendency of academics and intellectuals to herd together. Better wrong together than right alone, as the saying goes. We’ve often been told how financial herding (where different investors, brokers, and fund managers prefer to do the same things their peers are doing, for fear of standing out) precipitates market crises. I suspect that academics herd too. Currently, the herding tendency is towards singing the virtues of an uneasy amalgam of open source, free culture, user-generated content, participation, bottom-up, and a lot of buzzwords. I say “uneasy” because in principle, many of these are independent and a supporter of one may very well choose not to be a supporter of the other. In practice, they come in a bunch and self-appointed progressives like to bundle disparate things such as “the fight against restrictive copyright”, “enabling ordinary users to create content”, “the fight for opening up source code”, and the success of that specific thing called Wikipedia.

These things aren’t always unbundled. Seth Finkelstein and Jason Scott, among many others, while ardent critics of Wikipedia, have been supporters of many of the causes that usually come bundled with it, such as open source and using Creative Commons licenses. Nonetheless, I suspect that the bundling effects, along with herding effects, could be pretty strong.

Journalism on Wikipedia

Journalists such as Tom Friedman may be excused for giving a shallow treatment of Wikipedia in the four pages he devoted to it in his long book The World Is Flat. Nonetheless, some of the mistakes that Friedman makes are echoed sadly too often. One of these is to spend too much time interviewing the “person in charge” or the “people at the helm”. I’m guessing that journalists typically need to do this to get up to scratch, but interviewing people at the helm can be tricky for something like Wikipedia where nobody really is in charge.

Sincere and hardworking journalists, including Pulitzer Prize winners such as Friedman and New Yorker writer Stacy Schiff, make painstaking efforts to interview people in charge of the encyclopedia. For this New Yorker piece, Stacy Schiff did an amazing amount of work interviewing people that the Wikimedia Foundation directed her to. Only, it turned out later that one of these people whom Jimbo Wales vouched for turned out to have faked his identity (this person was Essjay, who claimed to have several academic degrees but later turned out to be a college drop-out). That said, Schiff’s piece was probably among the best in a mainstream publication that I’ve seen, along with this New York Times Magazine piece, and some pieces in the Chronicle.

Less accomplished journalists (as well as some academics) make the usual gaffe of interviewing Jimbo Wales and then saying something like “Wales has the following plans for Wikipedia …”, as if Wales is responsible for the success of Wikipedia the same way a corporate entrepreneur is responsible for the success of his or her own enterprise. The following lines are only a slight exaggeration: “The sky is the limit for Wales. Having created the world’s biggest encyclopedia for free, Wales is now working on a free dictionary, free news, and a free resource for books and source text in the public domain.”

Conclusion

The implicit support that Wikipedia enjoys from academics and journalists will last for some time, despite the excellent efforts by some journalists and academics to go beyond the surface. To get a new academic perspective on Wikipedia, what is needed is a coherent theme or theoretical framework in which a negative assessment of Wikipedia can fit — and such a theme needs to overcome the herding and bundling tendencies seen in academia. To get a new journalistic perspective that is reflected in more than just a handful of thoroughly researched articles, we need enough prominent academics and other people in a position that they are likely to get interviewed by journalists looking to write an article on Wikipedia. Neither prospect seems immediately forthcoming.

March 7, 2009

More on Wikipedia criticism

Filed under: Wikipedia — vipulnaik @ 1:54 am

My previous blog post on Wikipedia criticism generated quite a few comments. This was partly because it got covered in this forum post at The Wikipedia Review. There were several points that I made during the post that some of the commentators disagreed with, so I’ll try to elaborate the rationale behind those points a bit here, as well as what might be new insights.

Sausage: eating and making

The perspective from which I’m analysing Wikipedia is primarily an end-user perspective. The central question I explored last time was, “Does criticism of Wikipedia ultimately affect whether people read Wikipedia articles?” My rough conclusion was that there is unlikely to be any direct effect. This is particularly true of criticism that is aimed at Wikipedia’s process, because end-users care more about the end product, rather than the process.

One opposing viewpoint to this is that criticism of Wikipedia may make people less comfortable with using Wikipedia, because it might change the perception of whether the Wikipedia entry is reliable, accurate, or unbiased enough to be used. For instance, if potential web users are aware that Wikipedia entries can be edited by anybody, they may rely less on Wikipedia. I think this is plausible, but my personal experience suggests that if, even after all the possible unreliability is taken into the balance, Wikipedia is still the easiest to use, people will go to Wikipedia.

One argument that I mentioned, and that some commentators also mentioned, was that criticism of Wikipedia may have an indirect effect by discouraging people from contributing. After all, knowing the bad conditions in a sausage factory may not be that much of a disincentive for eating sausage, as long as the sausage is good. But it may discourage people from joining the sausage factory. If new contributors fail to arrive to replace the old ones who leave, then, the argument goes, Wikipedia entries will decay to the point where they get so visibly bad that even the end-users start noticing a quality decline. The argument doesn’t claim that end users care about process, but it does claim that contributors care about process.

In the last blog post, I pointed out one problem with this argument. Namely, if the most conscientious editors are the ones who are put off editing the most by criticism, the people who’re left may be the ones who are most likely to have agendas to peddle. This may result in a decline in quality — but not an obvious or visible one. That’s because the information-peddlers who are still left after some people get put off the sausage factory may also be the people who are most skilled at masking disinformation as information.

Here, I’ll try to elaborate on this, as well as give my understanding of how Wikipedia entries actually evolve.

Improve over time?

The naive belief among wiki-utopians is that Wikipedia entries keep getting better with time. The improvement may not be completely monotone, but it is largely so, with bulk of the edits in the positive direction, punctuated by occasional vandalistic edits and good-faith edits that go against the Wikipedia policy. For instance, consider this piece by Aaron Krowne, written way back in March 2005, as a critical response to this article by Robert McHenry. Another article in the Free Software Magazine gloats about the rapid growth of Wikipedia.

Wikinomics, a book by Tapscott and Williams, says that [Wikipedia] is built on the premise that collboration among users will improve content over time, and later continues on this theme:

Unlike a traditional hierarchical company where people work for managers and money, self-motivated volunteers like Elf are the reason why order prevails over chaos in what might otherwise be an impossibly messy editorial process. Wales calls it a Darwinian evolutionary process, where content improves as it goes through iterations of changes and edits. Each Wikipedia article has been edited an average of twenty times, and for newer entries that number is higher.

In his book The Long Tail, Chris Anderson writes:

What makes Wikipedia really extraordinary is that it improves over time, organically healing itself as if its huge and growing army of tenders were an immune system, ever vigilant and quick to respond to anything that threatens the organism. And like a biological system, it evolves, selecting for traits that help it stay one step ahead of the predators and pathogens in the ecosystem.

Not everybody believes that Wikipedia articles keep increasing in quality. In this philosophical essay, Larry Sanger (a co-founder of Wikipedia) makes the following interesting hypothesis: the quality of a Wikipedia entry does a random walk about the best possible value that the most difficult of the editors watching it can allow. This is merely a hypothesis, born out of Sanger’s experience, and Sanger makes no attempts to provide quantitative or even anecdotal verification of the hypothesis. Others have pointed out that many of the Wikipedia articles that receive featured article status (some of which even make it to the Wikipedia front page) later revert to being middling articles. Consider, for instance, this excerpt from an article by Jason Scott on the general failure of Wikipedia:

It is not hard, browsing over historical edits to majorly contended Wikipedia articles, to see the slow erosion of facts and ideas according to people trying to implement their own idea of neutrality or meaning on top of it. Even if the person who originally wrote a section was well-informed, any huckleberry can wander along and scrawl crayon on it. This does not eventually lead to an improved final entry.

My view: precarious equilibrium

There is exactly one Wikipedia article on every topic. Given Wikipedia’s dominance, this is the canonical source of information on the subject for hundreds of thousands (perhaps millions) of people. This canonicity is part of what makes Wikipedia so appealing to end-users, but it also means that even minor disagreements among potential editors of the article can become pretty significant when it comes to controlling that scarce and extremely valuable resource: the content of the Wikipedia article. After all, it feels like it pays off to put up a fight if fighting a little more can affect what thousands of people will learn about the topic.

A Wikipedia article on a controversial topic typically settles into a precarious equilibrium between different factions or interest groups that want to take the article in different directions (or prevent it from going in different directions). Consider, for instances, articles such as evolution, intelligent design, abortion, and alternative medicine topics ranging from well-known topics such as homeopathy to relatively lesser known topics such as Emotional Freedom Techniques.

The Emotional Freedom Techniques article is an example of such an equilibrium. There are roughly two camps: the proponents/believers, or people who for other reasons, feel that the article should contain more details about the subject. On the other side are the skeptics/disbelievers, or people who otherwise feel that putting too much information on an unproven therapeutic approach may in fact amount to an endorsement by Wikipedia. At some point long past, the article was an editing hotbed (in relative terms to its current status). It was much longer, with a lot of discussion on the talk page; sample, for instance, this revision dated 9 January, 2006.

For the next year, till around February 2007, the article remained in roughly the same state, with proponents adding in positive details and removing negative details and critics doing the opposite. On 30 January 2007, the article was nominated for deletion (here is an archived copy of the deletion discussion). A sequence of edits in the next three weeks gradually reduced the scope of the article to a considerably smaller one. The idea was that in order to “save” the article, it needed to be reduced in scope. The critics had managed to disturb the past equilibrium. By March 2007, a new equilibrium had been established, and modulo the addition and removal of a few references, this new equilibrium has been maintained for the past two years.

My experience suggests that for controversial topics, this is typical: there are two or more camps of editors, and depending on their relative strengths, the article enters a certain state around which it varies a bit but roughly remains the same. Once this equilibrium has been established, it is not easy to break. One way of breaking the equilibrium is using a “war of attrition”: keep making changes in your direction until the other person gets tired and walks away. Another one is to recruit other forces to help you, and a third approach is to threaten drastic measures, such as deletion.

Of course, the story of conflicting agendas plays out even in relatively non-controversial topics. Even editors who aren’t ideologically opposed to each other can find a lot of different things to quibble about, thus barring progress of a Wikipedia article. In the less controversial and more low-profile cases, it isn’t so much blood-curdling fights that create an impasse but simply a lack of common vision on how to take the article forward. Different editors come to Wikipedia with their own baggages and agendas — even simple agendas on how mathematics or physics should be written or what restaurants should be mentioned in the article on a local community. Typically, editors join feeling enthusiastic that they’ll be able to share their ideas and knowledge with the rest of the world, and also learn from how others are sharing. Once they realize that others holding opposing views are going to work in orthogonal or at times opposing directions, they either get put off, or they enter a wiki-war. In some cases, I think there is a clear situation of some people seeking to do constructive editing and others trying to obstruct. In most cases, it is a bunch of minor ideological mismatches that lead to people either getting put off Wikipedia or choosing to get aggressive to defend the articles.

Thus, there are roughly two kinds of articles: the controversial ones where there is a precarious equilibrium between different interest groups trying to pull it in their direction, and the relatively non-controversial ones where competing agendas and views on how things “should be” written lead, not so much to warring, as to a simple lack of activity. The former happens in cases where the stakes are more significant, and where people can feel good and hot about taking particular stands. The latter happens in more mundane things such as normal subgroup or T. Nagar where people simply couldn’t be bothered to fight.

The precarious equilibrium exists at levels higher than the level of the individual article. For instance, Wikipedia has a long history of a battle between inclusionists and deletionists. In the beginning, when the encyclopedia was small, deletionists hardly existed. As the encyclopedia became larger, deletionists started gaining the upper hand, as the need for keeping the encyclopedia free of garbage began to be appreciated. The deletionists had their heyday in 2005-2006, but inclusionists have started gaining ground again. In a recent Guardian column, Seth Finkelstein describes some of the battles and underlying agendas.

Stagnation is not the same as death

The precarious equilibrium for controversial topics and the relative stagnation for non-controversial topics may suggest that Wikipedia article quality could well get on the decline, to the point where users start noticing. However, this is probably not going to happen, for many reasons. First, encyclopedia articles on non-controversial topics are often the kind of thing that do not get outdated anyway. For instance, the definition of a normal subgroup is unlikely to ever change. Similarly, basic definitions such as friction in physics and historical articles such as Aristotle aren’t likely to get outdated either.

For controversial topics, it may well happen that a precarious equilibrium may inhibit development, but then again, the whole problem is that nobody can define “development” of an article in clear terms.

But the bigger problem, both for controversial and non-controversial topics, is that it is highly unlikely that Wikipedia article quality will actually decline significantly. So, a stagnation or stabilization in article quality can spell doom for Wikipedia only if some competing resource is trying to improve. Stagnation or stability equals death only in the presence of serious competition.

Okay, so can competition succeed?

The problem here is that the success of an encyclopedia effort requires a large amount of collation of people’s efforts, and this kind of collation doesn’t happen easily. What Wikipedia has managed to do is give enough people the impression that it can be a useful place to pool their efforts. I remain unconvinced of whether Wikipedia has actually achieved to solve the many inherent problems of large-scale collaboration, but the very fact that they can convince a lot of people, even if for a short period of time for each person, to spend some time on Wikipedia, is impressive.

People often join Wikipedia sold on the idea of working together with others, sharing their ideas, and learning from others. But not too many people are really willing to learn or change their worldviews, and the one-article paradigm of Wikipedia really forces a lot of conflicts out into the open. Thus, after trying a bit to make their voice heard amidst the din, a number of people leave.

Of course, most people who leave are bound to think of themselves as in the “right” and stubborn other Wikipedians as in the wrong, which creates a temporary bonhomie between disgruntled ex-Wikipedians. Such temporary good feelings towards one’s fellow wronged might result in a new idealistic commitment to create something better than Wikipedia. But beneath that is still the fact that many of the disgruntled ex-Wikipedians have agendas that compete with and are incompatible with each other, and new efforts that seek to do better than Wikipedia haven’t yet found a way to overcome this problem. (Fundamentally, I don’t think a way exists that the problems can be overcome). There are many examples of Wikipedia forks that have rapidly settled down into obscurity. An example is Veropedia, run by ex-Wikipedian Danny Wool and Cassiopedia, The True Encyclopedia (this already seems to have vanished and been replaced by some new wiki). Other encyclopedias that try to do Wikipedia right include, for instance, Conservapedia, intended as the replacement to Wikipedia for conservative Christians.

An example of a could-be-better-than-Wikipedia encyclopedia effort is Citizendium, founded by Wikipedia co-founder Larry Sanger. Sanger, and many of the others who work on Citizendium, seem bent on avoiding the many edit wars and other conflict situations that arise in Wikipedia. So far, so good — the Citizendium has survived and has been growing slowly for the last one and a half years. Yet, beyond the substantially greater civility and the substantially lesser activity, there is little to distinguish Citizendium from Wikipedia in terms of the competing agendas of its users. The main difference right now, as far as I can see, is that the Citizendium articles typically settle into an equilibrium of inactivity (which is similar to most Wikipedia articles on non-controversial topics) as opposed to a precarious equilibrium born of warring parties.

I personally do not think that there is room for another Wikipedia-like endeavor, at least in the near future. This does not mean that everything that seeks to do Wikipedia better will necessarily fail outright. It is probable that Citizendium will continue to grow over the next few years, and may at some stage become good enough as a general-purpose encyclopedia. Nonetheless, it is unlikely to become seriously competitive with Wikipedia in the near future.

The direction in which competition to Wikipedia could indeed be dangeorus is the direction of an increased number of more specialist sites, that help provide answers to people’s queries in somewhat more specialized topics. These specialized sites, of course, have their own conflict problems, but may be able to overcome these problems better simply because there is no single one of them. This allows people with competing agendas to work for competing specialist sites, rather than battle needlessly on the same turf. The best example of a somewhat specialized wiki-based site is Wikitravel, which is a great site for travel information. Since this is a relatively more narrow-focused site, it has clearer policy that reduces conflict over the structuring of articles. There are many others at varying ends of the spectrum between general and special. For instance, there’s WikiHow, which also seems to be doing pretty well for itself: a wiki-based how-to manual. This sacrifices some of the canonicity of the Wikipedia entry by allowing different how-to articles to be written by different people. There are a lot of substantially more specialized and narrow efforts, ranging from the hastily conceived to the well-planned. My own efforts at a group theory wiki, followed by an effort to generalize this to subject wikis in general, is one small example.

Yes, but how can diversified competition succeed?

If, as I believe, a challenge to Wikipedia can be presented only through a large number of specialist sites that compete healthily with each other and with Wikipedia, we have a bit of a paradox. The paradox is that the very reason people go first to Wikipedia is so that they do not need to navigate through or remember a bunch of different sites — Wikipedia is useful as a one-word solution to the problem of finding information.

The paradox isn’t all that big once we remember that for each specialized topic, there’s likely to be only one, or a few, places to go to. And the other important point is that within each resource, locating the article or piece of information that’s needed is pretty fast. Imagine, for instance, that people surfing casually for medical information, instead of going to the Wikipedia entry, were guided towards a collection of competing medical information websites. At first sight, a random surfer may just pick one website, and look up information there. If the surfer found that information well-presented and useful, the surfer may continue to visit that specialized site for medical information of that sort. Another surfer with somewhat different tastes may not like that first pick and may try a different medical information site. Since the medical information sites need to compete for users, they strive to provide better information that answers users’ needs more effectively.

There are two differences with Wikipedia: first, different people use different competing sites. Second, each site restricts itself to something it can specialize in, so the choices a person makes with regard to medical information sites can be independent of the choices the person makes with regard to sports information sites or glamour/fashion sites.

The idea that diverse competition can succeed against Wikipedia is also possibly an affront to people who view the collaborative principles behind Wikipedia as morally superior to the cut-throat competition that characterizes much of the messy market. Somehow, competition seems to be inherently more destructive and wasteful than the “working together” that Wikipedia engenders. Seem as it may, I think that a more diverse range of knowledge offerings can actually help reduce effort as people spend less effort fighting each other and canceling each other’s efforts, and more effort building whatever things they believe in.

Further, there are ways to ensure that a spirit of competition coexists with a spirit of sharing of ideas and knowledge. Academic research and software development often follow extremely open sharing principles, and yet can be fiercely competitive. The key thing here is that since a lot of independent entities are separately pursuing visions and borrowing ideas from each other, there is little destructive warring for the control of a single scarce resource. A spirit of sharing and openness can be backed by open content licenses such as the Creative Commons licenses.

What about search engine and link dominance?

Back at the beginning of the 21st century, when Google was nascent and Wikipedia non-existent, there were a number of books highlighting possibly disturbing tendencies that might develop on the Internet. Among these was Republic.com by Cass R. Sunstein (also co-author of Nudge, and now a member of the Obama administration). Sunstein warned of the dangers of group polarization on the Internet, with extremely personalized surfing, linking only to similar sites. This, Sunstein argued, could potentially lead to two bad outcomes: the absence of a public space, where issues of general interest could be addressed, and the total non-exposure to opposing or different thoughts and ideas.

The problem today seems to be of a somewhat different nature: the presence of a few sites that dominate much of Internet surfing. Wikipedia is increasingly becoming a destination for information-seekers, both as a direct destination and as a destination via search engines and links. As I explained earlier, I believe that the canonicity of Wikipedia as an information source is what makes it so attractive to edit and control.

That is why some people have suggested that Google and other search engines that place Wikipedia highly are largely responsible for Wikipedia’s success. Some research has shown that about half of the visits to Wikipedia still come through search engines. This suggests a “solution” to the problem of Wikipedia dominance: demote Wikipedia in the search engines.

I don’t think such a solution will either work or make sense.

Unlike Wikipedia, which faces no serious competition, search engines face tremendous competition. Google may be a market leader but it cannot afford to sit back and relax. Search engines also have a strong incentive to please their users. This means that if Google is placing Wikipedia high up in its entries, then it has a strong incentive to do so: that’s what its users want. Whether this incentive is due to conscious tweaking by Google employees or simply an unforeseen consequence of Google’s PageRank algorithms is unclear, but if it were something that displeased users, Google would fix it.

I suspect that a lot of the traffic that comes to Wikipedia through search engines actually comes through algorithms of the sort: “Do a search and pick a Wikipedia entry if it shows up in the top five, otherwise pick whatever seems relevant.” Here, search engines are being used as a “Wikipedia+”: Wikipedia, plus the rest of the web if Wikipedia failed. If the search engine failed to turn up Wikipedia in the top five, and the user later found a Wikipedia entry on the topic that he or she felt should have been up there, the user may start bypassing the search engine and go directly to Wikipedia.

Second, even if search engines stop favoring Wikipedia, the default-to-Wikipedia rule runs through many things other than search engines. Many smartphone applications and other Internet-connected devices such as Amazon’s Kindle give Wikipedia privileged status: for instance, the Kindle enables special access to Wikipedia for no extra charge. Some smartphone lookup services utilize Wikipedia articles as a knowledge base.

But even if all these big guns decided to disfavor Wikipedia, the link dominance of Wikipedia is too widespread: there are any number of blogs that provide links to Wikipedia articles on topics, many of them probably based not so much on the particular qualities of the Wikipedia entry as on the brand name.

So yes, I do believe that if all linkers, commentators, search engines, and smartphone applications suddenly revolted against Wikipedia, people, finding a lot fewer links to Wikipedia, may start forgetting Wikipedia, or at any rate, make it less of a default. I just don’t see any reason for such a collective epiphany to occur.

Blog at WordPress.com.