Since I recently blogged about Math Overflow website, I’ve been wondering what happened to various other math websites that once looked promising, and how they’re faring. Some of them seem to be going strong, but none of them seem to have been exploding in popularity.

Tricki

I blogged twice about Tricki, the Tricks Wiki, which went live in April 2009 (see the annoucement by Tim Gowers). Tricki held a lot of promise. Of late, the enthusiasm seems to have slowed down, though this might be a temporary phenomenon. The most recently created article and the most recent comments appear to be two weeks old as of today (October 27, 2009). According to Alexa data, the site has a rank of 1,200,000+ worldwide and about 550,000-600,000 in the United States. For comparison, subwiki.org, which I run, has Alexa data showing a site rank of 500,000-550,000 in the world and 150,000-200,000 in the United States, while Math Overflow has Alexa data showing a rank of 350,000 worldwide and about 60,000 in the United States (the numbers you see clicking on the links may be different if you don’t view this post within a few hours of my writing it).

Tricki also hasn’t been mentioned on Gowers’ blog since June 25, 2009 and on Terence Tao’s blog since August 2009.

Is the Tricki falling into disuse? Clearly, the initial spate of interst seems to have subsided, but it might well regain a slower and steadier momentum in some time.

Planetmath

I remember a time when Wikipedia had much less mathematical content than planetmath, which was one of the first places to check mathematics on the Internet. Planetmath appears to be going strong, though not as strong as before. While their message forum seems reasonably active, their latest addition was about a week ago, and they seem to be getting somewhere between 0 and 2 new articles in a day, and around the same number of revisions a day. Not exactly dead, but not bubbling with life. Their Alexa data indicates fairly steady performance with a traffic rank of around 130,000 over the last six months, but a decline over a longer timeframe — setting the drop-down parameter to “max” below the chart shows that their traffic rank and daily pageviews have been following over the longer run. Why? Decline in quality? Probably not — it’s more likely that people are increasingly using Wikipedia.

I recently came across a post by John Baez on the n-category cafe titled Cheaper Online Textbooks?. Baez’s post has a number of interesting links: a piece on “Affordable Higher Education” by CALPIRG, a piece on the legislation based on this report by Capital Campus News, an article in the Christan Science Monitor on the rising cost of textbooks, and a blog post in the Chronicle of Higher Education on an e-textbook program. So, reading about all these posts, I began to wonder: are textbooks getting too expensive? And should anything be “done” about it?

Are textbook prices soaring?

So I decided to look at the range of calculus books. The general impression from the things I read seemed to be that it would be hard to get a decent textbook for under $100. So, I went and typed calculus on Amazon, and looked at the first page of search results. Among these search results was Calculus for Dummies ($12.99), Forgotten Calculus ($11.53), Calculus Made Easy ($26.95), Schaum’s Outline of Calculus ($12.89), and The Complete Idiot’s Guide to Calculus ($12.89). Most of these books would seem reasonable for a low-level introductory semester or two quarters in calculus — admittedly, they may not be suitable for all calculus courses, but if price really is a primary consideration, it isn’t as if there are no options. There is also a wikibook on Calculus and an old public domain book on calculus. If you want somewhat more advanced stuff for free, you can try MIT’s OpenCourseWare course on single variable course, which includes video lectures, their course on calculus with applications, and their course on multivariable calculus.

Okay, so perhaps calculus is a bad example? Well, I decided to pick point set topology. The standard book for this is the second edition of Munkres’ book, which I think is one of the best, and it costs $107.73 on Amazon. But searching for topology on Amazon gives a number of other considerably cheaper books, such as Mendelson ($7.88), Gamelin and Greene ($10.17), Springer Undergraduate Math Series book by Crossley ($23.40), Schaum’s Outline of General Topology ($12.89), among many others. None of them seem as good as Munkres, but they all cover the basic material — and reasonably well, it seems.

Of course, I have picked on calculus and topology, both topics that are more than fifty years old, and where most of the material that should be included in an elementary textbook is widely known. In other words, the field for writing books is wide open. No publisher or author has significant scarcity power. When we are looking at exotic topics such as the theory of locally finite groups, then yes, you probably wouldn’t find cheap textbooks. But most undergraduate-level textbooks would likely be of the level of calculus or topology texts, and not exotic texts on locally finite groups.

Why do instructors choose expensive textbooks when cheaper alternatives exist?

Why do instructors choose $100+ calculus textbooks or Munkres’ topology textbook when there are so many cheaper books available on the market? One explanation, pointed out in a comment to Baez’s post, is the “moral hazard” explanation. This states that instructors do not need to bear the costs of buying the textbooks, so they just prescribe the “best” textbook based on their personal criteria rather than taking the price into consideration.

This is a reasonable-sounding argument, and is possibly true in many situations, but is it possible that even taking the cost into consideration, a slightly better book worth $100 might be better than a slightly worse book worth $10? For instance, at the University of Chicago, where I am pursuing my graduate studies (and now teaching calculus to first-year undergraduate students) the University charges students around $3000 per course per quarter. This means that students are paying something in the range of $100 per lecture. Even if you believe that only 25% of the money charged by the university as tuition is charged for the actual education, that still comes to $25 per lecture. For a calculus textbook that will be used for three quarters (and hence, about 75 lectures), is a price difference of $100 versus $20 worth worrying about? And even at universities where students pay smaller out-of-pocket fees due to more direct taxpayer subsidies, the cost of textbooks doesn’t come to a very large fraction of the total costs borne by students plus taxpayers, and it becomes even smaller considering all the other ways that students “pay” by working.

So, when a point set topology instructor chooses Munkres over a cheaper textbook, the instructor may well be making a smart choice! This isn’t to say that there aren’t situations where the better textbooks are the cheaper ones. But the point here is that the cost difference may simply not be worth worrying about in the grander scheme of things.

Is it evil to bring out new editions?

Another point raised in the material that Baez refers to is that textbook publishers are being evil by regularly bringing out new editions of their work. Certainly, one rationale for bringing out new editions of calculus textbooks is that people keep buying textbooks and don’t simply use the used books in circulation, but there are other reasons to bring out new editions, such as correcting things that were wrong, improving exposition, and so on. Even in a subject like calculus, terminology and best teaching practices change with time, student and teacher needs change with time, and new editions help keep pace with these changes. If new editions were completely useless, why couldn’t everybody just make use of calculus books old enough to be in the public domain, such as this one on Wikisource?

And if there is so much concern about bringing out new editions, all a college needs to do is buy enough copies of the current edition to keep in circulation for several years, offer students special deals whereby they can “rent” books for a year instead of buying them. If you buy enough books for three years’ worth of students and half the students in each year use the “rent” option, then a given stockpile of books can last five years. If such strategies seem ludicrous to contemplate, it suggests that the evil of new editions may not feel that threatening.

To be fair, there do exist book rental services such as Chegg (profiled in the New York Times PING column) and Campus Book Rentals. Also, publishers are offering to rent books to students.

Should legislators put price controls? And should textbook innovation be subsidized?

The Capital Campus News piece quoted by Baez says that a law passed in California state to control textbook prices lacks “teeth”. The piece sounds almost rueful about the fact that publishers cannot be stopped from bringing out new editions of their textbooks, since that would violate the First Amendment. While they don’t explicitly say so, they’re almost hinting as though this is a bad thing — an inconvenience. CALPIRG, which claims to “speak truth to power”, says that “Congress should require publishers to curb practices that drive up the cost of a college education.”

Let’s pass up here the issue that, in an issue of bargaining between two parties (students versus publishers) each with its own vested interests, one party is asking legislators to step in and tilt the tables in its favor. That kind of thing happens all the time in democracies. Let’s go instead to the question of whether price controls may be a good idea.

As Tyler Cowen says at the “Marginal Revolution” blog: “We know that textbook innovation saves lives and has a very high benefit to cost ratio. Thus, price controls or other restrictions that reduce prices are almost certainly a bad idea.”

What does this mean? The general idea is that price controls reduce the incentives for new entrants to enter the market. Those who are already writing and selling textbooks can handle the price controls — their profits may reduce, but they can still stay, but those who are thinking of whether to enter may choose not to stay put. Similarly, innovation into new kinds of product mixes might be drastically curtailed by price controls. (see also a similar argument against price controls on pharmaceuticals by Alex Tabarrok of Marginal Revolution, and the video titled “Price Gouging:Myth or Fact” on John Stossel’s videos on ABC).

Some people are skeptical that price-based incentives can result in better, or cheaper, calculus instruction. For instance, this comment on Baez’s post seems to suggest that if calculus textbooks became free, people would be motivated to write calculus for the love of the subject rather than in order to make money. The comment doesn’t say what this will do to the quality of textbooks.
Also, some mathematicians seem enamored with the idea of free instruction, in the sense of copylefted material that students can freely use and recycle. As Baez says:

One way to tackle this problem is to develop free online textbooks. I think a wiki-based approach could be good. People are trying it. Will it ever catch on?

It might also make sense for the NSF, or other funding agencies, to pay for scholars to write free online textbooks — or improve existing ones.

He then optimistically points out to a Chronicle of Higher Education blog post on a program at the Universoty of Wisconsin at Oshkosh using a $300,000 grant from the Fund for the Improvement of Postsecondary Education.

There’s definitely a possibility that such pilot projects, based on grants from the government or from private foundations, that aim to explicitly create freely available copylefted or remixable projects, will succeed. MIT OpenCourseWare has been pretty successful, though it doesn’t seem to have obviated the need for textbooks. Yale has also published open courses, and so have many other universities (see OCW consortium for more details). Independently, there have been various projects that aim to write “modules” — small units that can be fitted together into larger books and have Creative Commons licenses — the most prominent example of this being Connexions.

But while these are promising directions of investment, they’re probably not the only possible ones, and at this stage, it would be hard to predict what will catch on and to what extent new resources of this kind could replace textbooks. Besides, there are plenty of alternatives that may well be motivated by the lure of high prices. Some of these may start out even more expensive than textbooks, but may offer other features that textbooks don’t, and hence offer more value-for-money. Price caps could significantly deter investment into such tools.

For instance, Thinkwell, which was started over ten years ago, offers video textbooks. Their college calculus book includes 50+ hours of high-quality video instruction, with lecture transcripts, lecture notes, and practice exercises, and costs $140 + tax. This isn’t just video instruction in the sense of a person at the board — it is designed in a way that includes a lecturer writing stuff, side notes on what the important points covered so far are, and a useful presentation. It seems unlikely to me that this kind of investment into a “video textbook” would have occurred without the incentive to make money through selling online subscription — even though I’m sure there are non-financial motivators for them too.

The upshot? And broadsides

The upshot is that I don’t see any immediate and pressing reasons that the college textbook cost is a major issue or that politicians need to take action about it. Yes, there is a legitimate “moral hazard” argument, but the cost of college textbooks should probably be kept in perspective, and that perspective is the cost of college, both in terms of direct financial cost and the opportunity costs of attending college. Perhaps the appropriate standard against which to compare the rise in textbook costs is the rise in college costs, and not general inflation. It is rather ironical for colleges, that have been shamelessly raising fees well above inflation, and for faculty, who often make more money for every lecture they teach than the cost of a calculus textbook that lasts an entire year, to wax eloquent about the problems that students face because of rising college textbook costs. (For more on college costs, see this College Board report and this New York Times article on the report).

It’s always better for consumers to have things cheaper than to have them more expensive, so I’m all for cheaper books of the same quality or better books at the same price. But getting these is less about one group of vested interests trying to get support for their barganing position and more about encouraging innovation. And one way of at least not discouraging innovation is to not impose price controls and not try to determine the final shape or direction of things. And some of the innovations could drive average prices up, because they may be high-price-high-quality points on the price-quality spectrum.

Timothy Gowers and Michael Nielsen have written an article for Nature magazine about the polymath project (I blogged about this here and here).

In the meantime, Terence Tao started a polymath blog here, where he initiated four discussion threads (1, 2, 3 and 4) on deterministic ways to find primes (I’m not quite sure how that’s proceeding — the last post was on August 28, 2009). (UPDATE: A new post (thread 5) was put up shortly after I published my blog post).

Around the same time, Gil Kalai started a polymath on the polynomial Hirsch conjecture (1, 2, 3, 4 and 5).

Also, some general discussion posts on polymath projects: by Tim Gowers and by Terence Tao.

It remains to be seen whether any of these projects are able to reach successful conclusions or make substantial inroads into the problem. If there is another success for a polymath project, then that would be a major booster to the idea of polymath projects. Otherwise, it might raise the question of whether the unexpected degree of success of the first polymath project led by Gowers (which aimed for, and got, an elementary proof of the density Hales-Jewett theorem) was an anomaly.

In recent times, the Math Overflow website has been getting a lot of “press”, which is to say, it has been mentioned in some highly prominent math blogs. It was reviewed in Secret Blogging Seminar by Scott Morrison, who is also involved with Math Overflow, and it was mentioned by quomodocumque, Timothy Gowers, Terence Tao, the n-category cafe and others.

Math Overflow is a website where people can ask math-related questions (the questions should be of interest to people at the level of Ph.D. student or higher), answer the questions, and rate the answers. It uses the Stack Exchange software, which is used for many other websites, such as Stack Overflow. Funding for the website is being provided by Ravi Vakil of Stanford University, and it has a bunch of moderators — but anybody who earns enough points through participation can rise to the status of moderator. For more information, see the Math Overflow FAQ.

Participation on the website has been increasing rapidly since the first post (September 28). Here’s the Alexa data, which seem to indicate that usage has been growing (Alexa is not very reliable for low-volume sites, since it uses a small sample of users and most Math Overflow users may not be using Alexa’s toolbar).

The software and site layout seem well-designed to encourage participation. The long-term performance seems unclear, since a lot depends on how effectively the site is able to allow users to fruitfully explore past questions and answers and discover things similar to what interests them. But, as of now, it has a bunch of interesting questions, and seems to have reached the ears of a lot of people who’re interested in asking good questions and giving good answers.

UPDATE: See the polymath project backgrounder for the latest information.

It’s been some time since I last wrote about the polymath project (see this, this for past coverage), and an even longer time since I wrote an extremely lengthy blog post about Michael Nielsen’s ideas about collaborative science.

The first polymath project, polymath1, was about the density Hales-Jewett theorem. This was declared a success, since the original problem was solved within about a month, though the writing up of the paper is still proceeding. The problem for the project was proposed by Timothy Gowers.

Terry Tao (WordPress blog) has now started a polymath blog discussing possible open problems for the polymath project, strategies for how to organize the problem-selection and problem-solving process, and other issues related to writing up final solutions and sharing of credit.

In this blog post, Jon Udell reflects on how the introduction of LaTeX typesetting into wordpress was a positive factor in getting talented mathematicians like Terence Tao and Timothy Gowers into the blogosphere, and leading to innovative user projects such as the polymath project. Udell notes that introducing existing typesetting solutions into new contexts such as Internet blogging software can have profound positive effects.

Christina Pikas offers her take. It seems to me that her post is a summary plus some cautious optimism and historical context.

Flip side?

This post on the backreaction blog considers the negative side of open science. Influenced by James Surowiecki’s book The Wisdom of Crowds, this talks about the pitfalls of too much openness leading to science developing only in certain directions that attract more immediate attention, compromising on independence during the decision-making process of what direction to pursue research in. The author says that it may in fact be the case that there is already too much collaboration in science leading to the underdevelopment of lines of research that have long-run promise but will not attract collaborators in the short term because others may not think that highly of them.

The backreaction blog post is a response, not specifically to the polymath project (in fact, it does not even mention polymath and is not specific to mathematics). Rather, it is a reaction to this post on the Open Science blog.

I think there are two somewhat differing and confused aspects here. The first is whether scientific findings should be open and widely available. The second is whether science should be done more collaboratively, with early sharing of findings and solicitation of inputs from others, rapid publishing, more “polymath”-style research. These are somewhat different issues.

Massive collaboration (of the kind that Nielsen talks about) is admittedly possible only when scientific findings are openly available. But the converse is not necessarily true. Greater open availability, accessibility, and ease of interaction with scientific content do not form a sufficient prerequisite for massive collaboration. Thus, it is possible to believe that openness and accessibility of findings is a good thing with few pitfalls while massive collaboration is a mixed blessing.

The key point is that open availability of research that is in progress can be achieved without making such research prominent to other researchers, except those who specifically express their interest in it. At the same time, well-established information should be made available in a manner that makes it very easy to find and use for all, as I’ve argued before. My paradoxical view is that the maximum openness can be achieved with the minimum downside if we concentrate on making sure that the material that is already well-known and clearly areed upon is made very widely available and “in-your-face”.

What about new and counterintuitive ideas?

For newer research and speculative findings, we need modes of availability that minimize the dangers of groupthink. One way of doing this would be to have incomplete information, betting, challenges, and competition.

For instance, in sharp contrast to polymath-style projects, one thing that could be done is that one person announces that he/she is going to conduct a certain experiment with a certain methodology and also explains how different outcomes for the experiment will result in different conclusions. This is put up openly and inconspicuously. The person then withdraws into his or her inner shell (or research lab) to do the experiment, and then puts up the results. In the meantime, the formulation of the experiment attracts some people (though not a lot) and they post their varying ideas on whether there could be problems with the methodology.

In this case, the people evaluating the experiment (who would be the select few with interest in that area) are not biased by the actual outcomes of the experiment when questioning the potential merits or demerits of the methodology. This could in principle reduce the possibility of herding.

This “betting” kind of approach could provide greater incentives to work on less apparently useful things. Suppose person A has an alternative explanation of some hypothesis, and designs an experiment to test it. But person A is uncertain about getting collaborators because nobody believes her hypothesis.

So she designs an experiment and puts up the methodology online (without actually carrying out the experiment). She challenges detractors to come up with possible flaws in the methodology. Detractors come up with such potential flaws. The methodology is accordingly fixed. Once the methodology is fixed to the comfort of the detractors, both she and the detractor are convinced that the experiment will yield opposing outcomes. This is an ideal time to place a bet. (This brings us to ideas of futures markets for ideas, which is an intriguing topic).

Here, openness contributes not so much by encouraging cooperative collaboration but by encouraging competitive collaboration.

Such a system could be adapted in many cases to mathematics, even though mathematics has fewer of these “experiments”.

Again, however, this runs the risk of getting people too much into wanting to find “counterintuitive” things rather than doing the good old solid work of progressing on the existing foundations by proving that epsilon more (too much cooperative collaboration probably leads to the opposite problem). Further, even with this, there may be some ideas where the problem isn’t that they’re counterintuitive but rather that they seem totally unimportant because people aren’t used to thinking in terms of them at all.

Here, a “futures market” type of idea may be useful. A person who is convinced that an apparently unimportant idea could be very important when developed further, can make such a claim. The exchange (or community, or market) then asks for something (the stake of the person’s reputation) in exchange for a large reward if that person turns out to be successful in developing the idea to a great level.

In recent years, there has been a growing trend towards “open access” among librarians and academics. For instance, the University of Michigan recently held an Open Access Week, where they describe open access as:

free, permanent, full-text, online access to peer-reviewed scientific and scholarly material.

In an earlier blog post, I discussed some issues related to open access. Here, I attempt to look at the matter more comprehensively.

Rationales for open access

There are many rationales for open access. The simplest rationale is that open access means reduced cost of access to information, which allows more people to use the same research. Since the marginal cost of making research available via the Internet to more people is near zero, it makes sense from the point of view of efficiency to price access by yet another person to the research at zero.

Another rationale is a more romantic one: making scientific and scholarly publishing available openly allowsfor a free flow of ideas and a grander “conversation”. Support for this rationale also indicates that open access should be more than just free (in the sense of zero cost) access to materials, but also a license that permits liberal reuse of research materials in new contexts. Academia already has strong traditions of quoting from, linking to, and building upon, past work, but this form of open access seeks to provide a legal framework that explicitly specifies reuse rights that go beyond the traditional copyright framework of countries such as the United States. An example of such permissive licensing is the Creative Commons licenses.

As shorthand for these two rationales, I shall use cost rationale and conversation rationale.

Open access policies/mandates

One of the major problems the open access movement has faced so far is getting people to publish papers in open access journals. As long as the best papers continue to be published in closed-access journals, academics who want to read these journals will pressure their university libraries to subscribe to these journals, even when the journals overcharge. Thus, librarians are unable to push open-access terms on publishers.

To overcome this problem requires collective action by major academic institutions. The most typical action is an “open access policy” undertaken at a department-wide or institution-wide level, that says that all papers published by people in that department or university have to be available in an open access repository. Harvard University has three open access policies. As Stuart Shieber explains in this blog post, the Harvard policy should not be viewed as a mandate (though it has been described in those terms) because Harvard has no effective enforcement mechanism. Rather, it should be seen as a powerful default rule that encourages faculty who may otherwise be indifferent to move towards open access (read here Richard Poynter’s blog post parsing and picking issue with this).

Open access has been accepted by many other universities, including an institution-wide policy by MIT.

At another level, the National Institutes of Health instituted a public access policy a little while ago, that requires all papers written with funding using NIH grants to be submitted to an open access repository a year after publication. The compliance rate for the NIH policy, when I last checked, was around 50%. This means that about half of all NIH-funded papers published a year or more ago are now freely available online. If other funding agencies such as the national Science Foundation and the National Security Agency introduce similar public access policies, the effect on open access is potentially huge.

Is this lowering the cost and increasing the availability of information?

Are the cost rationale and conversation rationale for open access justified? Do open access policies reduce the overall cost of accessing information?

Some of you may be saying at this moment that it has, by definition, reduced the cost of access. I don’t think it is completely obvious. The question here is whether there were people in the past who knew they wanted to access some content and were unable to do so because of the cost, but are now able to do so. People who have a demand but whose reservation price is less than the cost of the journal (and who are not in institutes that subscribe to the journal) would be the people for whom most value was created. The real question then is: to what extent are the open access resources being accessed? Is there a sudden, explosive increase in demand when something becomes freely accessible? Chris Anderson, in his book Free, argues that there is an explosion in demand when the price drops to the magical number of zero.

I am not so convinced that there will be an explosion of demand. If there isn’t, then I don’t think the conversation rationale is supported, and the best that can be said about the cost rationale is that it distributes income away from journal publishers towards academia and libraries, while possibly expanding access to others (even if very few actually use it). While an open access policy could still be supportable on these diminished grounds, the question then becomes one of whether publishers or libraries are most likely to “do good” with the greater money that they have.

Unfortunately, I do not have any idea of whether demand has indeed exploded thanks to the open access policies, or whether new demand segments have been created. What I do know is that in much of the discussion of open access, success is measured not in terms of the effect on increased demand but rather in terms of the compliance rate with the open access policy. This contrasts with initiatives to make things free outside academia, where we regularly hear about authors and artists putting up their work for free and getting millions of downloads and a boost in sales of their books and albums.

Better strategies

An extraordinary focus on “open access” from a principled perspective might obscure what I consider the larger goals: the cost rationale and the conversation rationale. Reducing costs and increasing availability are unlikely to be achieved by a few mandates without a change in the fundamental nature of incentives. My admittedly limited reading of academic papers published in journals makes it clear that these papers are often written for small audiences. For these audiences, they are often fairly user-friendly, but for others outside the cliche, they could be quite user-unfriendly. The nature and structure of the academic paper itself is a much greater issue when it comes to ensuring access to more people than the problem of cost.

Further, the fact that open access policies need to be made across-the-board in order to ensure compliance suggests that for many academics, their incentives make them largely indifferent to open access in practice, even while they may support it in principle. For instance, in this blog post, open access advocate and University of Michigan librarian Molly Kleinman writes:

Most of the time, I only hear from faculty seeking copyright advice after they have a problem. Until they have a problem, author rights and open access are simply not on their radar. I can send postcards and emails and speak at department meetings until I’m blue in the face, but it’s going to take an outside force to convince busy academics that this is something they should be paying attention to.

What then would be a better strategy? I don’t claim to have any concrete answers, but I will here outline some possibilities. First, something has to be done to make accessibility of materials more than just a matter of anybody freely being able to download one’s work. Rather, it should be a matter of understanding what kinds of other people might benefit from looking at one’s work, including segments within one’s profession and outside it. Then, it involves being able to reach out to these diverse audiences and increasing their awareness of, and hence their ability to benefit from, such work. Of course, this is already done in academia, but my limited experience suggests that it could be done a lot more.

Is the problem one of incentives? Possibly. More importantly, it is a problem of “thinking small”. Lecturers try to teach good courses, and some of them put up course notes online, that other lecturers can benefit from. But these course notes are heavily context-based, and their usefulness as a general resource for others is severely limited. It takes a whole new degree of effort to create something like MIT OpenCourseware. OpenCourseWare was possible because of thinking big, and in particular thinking deeply about the needs of people outside MIT to access course materials in a comprehensive and useful ways.

The localized incentive to teach a course well (often at a huge cost of preparation time) without a comparable incentive to work towards the design of a high-quality, widely available course is similar to the localized incentive to write a good paper to solve a problem in the field versus thinking about the larger, broader implication of making one’s research work and the many directions one has explored widely available and accessible to a larger audience including people working in totally different areas.

Some might argue that thinking “big” takes too much time and effort and it is best if such big thinking is restricted to a few individuals who specialize in such work while most others concentrate on tackling smaller problems. This is plausible. Even if this is the case, we need to have explicit roles for people who specialize in thinking big and who also have enough of the respect and admiration of their colleagues to be able to spread the best practices emerging from their work.

To some extent, the move towards open access may simply be a reflection of the bigger thinking that has emerged in a world where people have more ready access to a lot more information and ideas. In so far as it reflects part of a broader trend of this sort, the future looks promising for both the cost rationale and the conversation rationale. In so far as open access is treated as an end in itself, I think it has little promise to revolutionize research, teaching or academia, though it might definitely ease the budget burden on libraries.

In a previous post mainly about the Tricki, I conjectured that in order to make information easiest to access, the Tricki should seriously consider the use of topic pages, that serve as central hubs from which users of the Tricki can get an idea of the Tricki’s coverage on a particular topic. I argued that, even in the presence of a website such as Wikipedia that has a high-quality article on the topic (though this is a moot point according to me) such a page serves a useful local function. I also pointed to the example of New York Times Topic pages. See, for instance, the Times Topic pages on Wikipedia, Twitter, and The Amazon Kindle.

Apparently, as I learned from this Register article, Google, in a testimony by its employee Marissa Mayer (PDF), said that:

Consider instead how the authoritativeness of news articles might grow if an evolving story were published under a permanent, single URL as a living, changing, updating entity. We see this practice today in Wikipedia’s entries and in the topic pages at NYTimes.com. The result is a single authoritative page with a consistent reference point that gains clout and a following of users over time.

To paraphrase, I think Mayer’s point is that individual news stories come and go, they are hard to search, find, and catalogue, but a single evolving story can gain clout over time as people refer to it and know it’ll be there. Apparently, another blogger saw a connection between the issue of topic pages for journalists and my suggestion to have topic pages on the Tricki.

Two kinds of users

I have no knowledge about building newspaper websites; in fact, I have extremely limited knowledge about building websites. So what I’m saying here should be treated largely as amateur speculation.

I broadly see two reasons why people visit sites on the Internet. One is to get the new, the latest, information, wherein they don’t know exactly what they’re looking for but want new information. For instance, when I type in a newspaper or periodical website into my browser URL, I’m not looking for specific information, but rather, looking for news. At the other extreme are situations where I am looking for specific information. For instance, I want to know whether there exists an infinite group where every element has finite order.

Of course, much web use lies somewhere in between. For instance, when I follow a subject-specific blog, such as this or this or this, then what I’m looking for is “new” information but restricted to something specific rather than the general wide swath of things that could happen to the world. Something similar may be said for social networking, in so far as the information-gathering there is related to updates and new information but limited to friends and others in one’s network.

On the Internet, these two forms of exploration serve each other. As I read a news article in the New York Times or an editorial in Foreign Policy, there are terms, concepts, ideas, historical facts, etc. that I encounter about which I don’t know enough. I am curious about other coverage of these. In this case, an initial desire for news in general gives rise to specific questions. Conversely, sometimes I get stuck at specific questions, so I do a more general exploration, looking for news that might shed light on one of the many specific questions I have unresolved.

An astute reader might point out that if the main value of “new” information is to provide stimulus for curiosity, the “new” feature isn’t crucial at all. Reading the New York Times of ten years ago would probably stimulate my curiosity just as much as reading that of today. In some ways, it may do so even more. I agree with this, but I also suspect that we humans have a somewhat irrational bias towards favoring the “new” information. (Personally, I enjoy reading archives and past news coverage, as well as books written in a different era, though I don’t do much of this. Reading their predictions and comparing them with what actually happened is both amusing and sobering).

The Internet has changed the dynamics of how we work by meeting both these kinds of needs, and allowing us to jump from one to the other. On the one hand, newspapers and periodicals seem to be in a suicidal race to put up more and more of their content on the Internet, creating a very very huge stream of new information. On the other hand, basic reference material on the Internet is also growing rapidly. The most high-profile example of this is Wikipedia, the free online encyclopedia, that I’ve talked about in the past. But there are other, more low-profile examples, too.

For instance, the online bookseller Amazon is just one among many freely available online book catalogues. Many other retailers as well as libraries now maintain extensive online catalogues. Sites such as How stuff works and WikiHow have a lot of basic useful to-do information. More importantly, many organizations, including corporations and non-profits, are putting their data and advice online in a fairly well-organized form. Consider, for instance, societies devoted to medical conditions in the United States, such as Alzheimer’s Association. Or consider the many governmental websites (such as in the United States, but now also increasingly in less developed countries such as India) containing forms and help information that one would earlier need to go to a government office to obtain. Or consider the large amount of mapping information available that allows people to plan trips and find locations.

Linking the two forms of exploration better

Note: I use the term “newspaper” here as a generic term that may refer to a newspaper, periodical, or an online-only news/news commentary service.

Newspapers fulfill our need for “new” information, for information about things around us that we didn’t know we were looking for. For many of us, most of the time, this itch for new information is enough; we like to read and then forget the information and not think more about it. But it often happens that a particular news story makes one curious about further information on the topic. For instance, reading a story about recent progress in reducing infant mortality in an African country led me to wonder: what has been the general progress on infant mortality over the past century? Which countries have led and which countries have been behind?

To meet this kind of need, the newspaper needs to do something more than put an online version of a print article — it needs to link to pages that give an overall, better flavor of the subject, which may include past newspaper coverage on the topic, links to (in this case) sites on health/medicine, sites on secular demographic trends, etc. For instance, the New York Times has a topic page on infant mortality, from which I could see all the past coverage the New York Times had on the subject, as well as a multimedia chart comparing infant mortality across different countries and over time. Ideally, there would be a lot more information on the topic, neatly organized in the topic page, but this is a start. Unfortunately, many of the articles that the New York Times lists on this page (such as this one) lack backward links to infant mortality. (Incidentally, according to this blog post, the New York Times plans to release, in usable form, the linked data cloud that it uses for topic tags.)

Unfortunately, building this kind of topic page structure is a daunting challenge for smaller news and news commentary organizations than the New York Times. Many, such as Newsweek, have settled on a somewhat intermediate solution — they have topic pages, but the explanation part at the top is simply pulled from Wikipedia (with attribution) and all pages referencing the article are listed in a sortable table. (See, for instance, the Newsweek topic page on health care costs). Other organizations, such as the The Huffington Post, have tried a different, more sophisticated approach. See, for instance, the Huffington Post pages on John McCain and on wikipedia. However, even the Huffington Post does not advertise this extensive use of tags as a way of searching. Many smaller news organizations simply rely on search rather than any (externally visible) tagging system, probably because their scale and scope makes investment in tagging systems not worthwhile.

One of the solutions for the problem of lack of scale (i.e., it is hard for small news organizations to create separate articles or a separate taxonomy) may be for multiple news organizations to pool together their taxonomies, and even to share topic pages and results. The Washington Post/Newsweek company, to take one example, has such publications as the Washington Post, Newsweek, Slate, Foreign Policy, and The Root. These already link to each other, but by having pooled-together topic pages, they can reap more benefits. Small news organizations that do not want to invest in taxonomies can use public taxonomies, or those contributed by larger news organizations (for instance, that of the NYT).

Topic pages versus search

Some people have argued that powerful search features have largely made things such as tagging, categorization, and other forms of arrangement largely redundant. There is some truth to this argument. To quite an extent, the utility of categorization was to speed up extremely slow and cumbersome search processes, and the existence of quick computer-enabled search reduces that need.

But a close look would reveal that tagging today is a lot more powerful and useful than it could have been in a pre-digital era. Now, because of the lack of constraints of physical placement, the same topic can receive numerous tags. These tags allow a much larger number of cross-connections to be discovered. So, while the need for unsophisticated users to use tags/categories may have considerably reduced, the scope for their use by even marginally interested and sophisticated people has expanded considerably.

But the more important point is that a central hub, such as a topic page, is more than a mere tag. It is an intelligent starting point because of the way it collates together diverse kinds of information related to a topic. For instance, consider the New York Times topic page on Iran. This is more than just a list of pages referencing Iran, and its value lies in the way it gives a clear idea of the way the newspaper covers Iran.

Genuine advantage

News organizations in the United States are currently (As of 2009) experiencing severe financial problems, and have often been criticized of not innovating and adapting to the Internet culture. But even a little sampling of newspaper websites such as the New York Times, Time, Newsweek, the Wall Street Journal, and the Chronicle of Higher Education shows that news organization websites have been very innovative in adapting to the Internet to improve the quality of presentation of news even as they have to make painful cuts to their news staff.

In the past, newspapers were a one-stop shop for all kinds of information such as weather information (now, weather services across the world have websites and can send alerts to mobile phones), stock price information (again, stock exchange movements are now publicly available on their own and many other websites), and real estate and matrimonial information (that got them a lot of their ad revenues). In practically all these respects, there will probably never be a single one-stop shop.

There are two things news organizations should continue to be good at, and should lead in almost by definition: gathering news on-the-ground (thus meeting people’s instant-news-need) and providing a framework linking current news to past coverage (thus providing a vast reference corpus for answering specific questions people have, particularly those raised by current events). It is in the second respect that the shift to online news really helps improve news quality. But for this, there needs to be an effective mechanism to link the dots. I assume that part of a journalist’s training involves linking the dots and placing the news being gathered with their knowledge of history and past events. By making these links more public and sharing them with readers, news organizations can significantly improve the news value of their content.

As more and more amateur news gatherers come on to the scene and break a lot of news, even the ability of news organizations to manage to always be the first with new news may be in question (though newspapers are currently far ahead). Their large corpus of well-organized internal news archives can, however, give these organizations one advantage that will be harder to eliminate: the ability to put new stories in a broader perspective and grow topic pages and central hubs to provide readers with a quick overview of practically any topic.

Addenda

Concerns have been raised by many that the New York Times does not link to source content even when such content exists on the web, and that a large fraction of its links are internal links to Times Topic pages. Some suspect that it is a strategy to maximize PageRank juice. This is definitely a valid concern, though I think that of late, the Times has started to link more to outside sources as well, and it definitely does freely link for blog entries, and at least it links to websites when talking about content specific to those websites. In any case, whether the Times is suppressing outward links at a cost to readers is largely tangential to the question of the utility of Times Topic pages. Here is a book chapter about linking practices.

It is said that a picture is worth a thousand words. A six-minute video is probably a thousand pictures.

There is a tradition in the mathematics department of the University of Chicago whereby, at the end of each academic year, the second-year class organizes an evening of skits called “Beer Skits” (“beer” because the skits are accompanied by huge servings of beer). As part of the tradition, we decided this year to create a mocumentary (mock documentary) of life in the first year at Chicago. I had the original concept and fleshed out the main scenes, but a lot of the editing work before, during, and after the shooting was done by many of my batchmates, who all added their own insights and removed some of my original ones that would have been too abstruse.

Our six-minute video is a little funny, but it is also quite realistic, as those who have gone through a similar experience can attest. Okay, agreed, some of the scenes towards the end stretch the boundaries of realism, but almost everything we have is a slight adaptation of something that actually happened during our first year.

Below is an embedded Youtube video (you can make it full-screen for better viewing).

Download a high-resolution 3mbps DVD quality version in Windows Media Video format (104 MB).

Some friends as well as other regular readers of this blog may remember that I mentioned the extensible automorphisms problem: if an automorphism of a group can be extended to any group containing it, must that automorphism be inner?

I had managed to prove this for finite groups, and had drafted a short paper with that and some other related results. My advisor George Glauberman then shared the results of this paper with some other mathematicians, and one of these people, Avinoam Mann, recalled that these problems had been the subject of papers in 1987 and 1990. You can see the extensible implies inner page on the groupprops wiki for more details.

Since I came up with the question myself, and also came up with the final answer myself (of course, with help and guidance from many others), I actually feel vindicated rather than distressed that this question was considered and solved long ago — it shows that the questions I ask have been asked by other people before, and been considered worth publishing.

In mathematics, one may be tempted to feel that it is getting harder and harder to get original results because more and more of the simple stuff has been taken. At the same time, as we acquire more and more knowledge, new researchers can stand on the shoulders of their predecessors and therefore ask and answer deeper questions. I feel that the benefits of being able to look farther and deeper outweigh the costs of having all the more basic results “already taken”.

I still do have a few additional results that I believe to be original, such as this one and its corollary, this one. I also have some partial results on the NPC conjecture, that I believe are new, such as this and others discussed on the NPC conjecture page. I hope to put these together with other results that I am yet to discover and hobble together something publishable in the not-too-distant future.

A short while ago, I blogged about the Tricki, mentioning what I considered an unnecessary subservience to Wikipedia. At the time, Tricki was still pre-live. Recently, Tim Gowers announced that the Tricki is “fully live”, which means that anybody can create a login and add and edit entries.

The Tricki already has a reasonable number of decent articles — an interesting article that I located in group theory is: first pretend that a normal subgroup is trivial. Here’s a link to the list of group theory-related tricki articles. Here is a list of tags that can be used to navigate the tricki.

The concept of the tricki is interesting, and I suspect that a lot of interesting stuff will go on there in the next few months. On the other hand, the main problem I see with the Tricki is that, while it has a lot of useful tricks, there is no easy standard reference it can use for looking up more details of the terms and facts used. The current consensus seems to be to link to Wikipedia, but using only one resource, particularly a resource of dubious value, seems restrictive, particularly since the specific Wikipedia article pointed to may not be particularly good. Consider, for instance, this page where, in the comments, Gowers notes that he “wouldn’t mind being reminded” what a characteristic subgroup is.

Here is what I suggest. For each of the concepts, Tricki should have a short page giving the definition and useful links, and the links from the tricki article pages should point to this short page. This way, article writers can concentrate on writing their articles rather than on defining all the auxiliary concepts involved (particularly in cases where these auxiliary concepts are reasonably standard and known to many of the readers of the article).

The concept page could serve three roles:

• A concise definition.

• A list of related tricki articles (this could be automatically generated by backlinks, or manually created and organized).

• A list of references/weblinks for more details. This list should be carefully curated, and people making such links should actually read the reference being linked to! Wikipedia could, of course, be one of the references typically given, but it need not be the only one.

Why do I think that separate concept pages are important? To take a totally different example, consider newspaper and magazine websites. Many of these websites have their own reference pages on a number of important topics — these reference pages provide a short summary, external links, as well as details of past coverage of the topic. For instance, The New York Times has Times Topic pages on almost all the things it reports on regularly, and its news articles have internal links to these Times Topic pages. While this is partly a tactic to maintain link juice, it also helps provide entry points for people seeking to get information on topics and get an overview of how the NYT has covered the topic. See, for instance, the Times Topic pages on Wikipedia, Twitter, and The Amazon Kindle. It seems to me that despite the presence of Wikipedia pages that may carry a lot more information, these Times Topic pages serve a useful function to readers who are interested in newspaper coverage of the material.

Also, there is something — a lot, in fact — to be said for keeping links largely internal, particularly when building up an extensively cross-referenced body of knowledge, information or insight. Part of Wikipedia’s success can be attributed to its policy of strong internal linking — extensive linking to other Wikipedia pages from within Wikipedia articles.

UPDATE: Gowers has started a forum discussion on some of the issues raised in this blog post.