Nature Shuts Down Open Wikipedia-Like Peer Review; If You Build A Wiki, It Doesn't Mean Anyone Will Edit It

Overhype

from the empty-room-problem dept

One of the bad things about the tremendous success of Wikipedia is that it’s created this widespread belief that all you need to do is set up a wiki or a wiki-like tool, and people will flock to it and fill it with content. For all the hype about “user generated media” and such, it’s a lot of hard work getting the system to provide enough value to get people to participate. It’s also quite difficult to build the critical mass necessary to get any kind of network effects. However, with so much hype, many seem to feel that a “just build it and the masses will come” plan will work. It’s not so simple. A few months back we talked about an effort to get better peer reviews in scientific journals by having a “wiki-like” system, where stories shortlisted for publication would be posted online first, and anyone could critique the paper prior to publication (and yes, the use of “wiki” here isn’t particularly accurate — but it didn’t stop everyone from describing it that way). The hope was that it would be a more open process that would help avoid some of the recent scandals over peer-reviewed, published research papers that later turned out to be bogus. Instead, though, it looks like Nature is killing off the experiment after just a few months. It appears that most authors had absolutely zero interest in pre-publishing their works for the rabble to critique — and, not surprisingly, of the ones that were published, there was very little activity in terms of peer review.

Honestly, none of this should have come as a surprise to Nature or anyone involved in the project. While we were hopeful that it would at least get some traction as an alternative method for peer review, it was pretty much doomed from the start. With no additional incentives on either side, there were few reasons for anyone to take part. There are benefits to being a part of the traditional peer review process, but there were no additional benefits to taking part in this effort. Also, it assumed that people would automatically be interested in following what stories were put up for review, when most experts in these fields have pretty narrowly focused specialties and wouldn’t be much help for the vast majority of the content placed up for review. That isn’t to say that the peer review process can’t be improved — and the internet and various other new technologies can certainly help. However, it’s going to take a lot more thought that just throwing up an open system and claiming “it’s just like Wikipedia.”