Would Online Open Review Lead to an Academic Popularity Contest?

Double blind peer-review in science and other fields has been the norm for decades. Now some scholars, as featured at the NY Times this week, are arguing that peer-review needs to move online, eliciting unblinded judgments from a greater number and diversity of experts. But would removing the anonymity of peer-review and widening the scope of participation play into the hands of several common human biases?

Following the New York Times story, I discussed the advantages of open-review and open-access publishing, noting though some of the challenges in moving academic fields and universities in this direction.  In reaction, my colleague at American University Wendy Melillo Farrill offered several important insights, referencing in part the arguments made in Nicholas Carr’s The Shallows: What the Internet is Doing to Our Brains. I asked Wendy to share her insights with readers in a guest post.–Matthew Nisbet

I would like to add one more piece to Matt’s suggestion about peer review in the digital age. I just finished reading Nicholas Carr’s The Shallows: What the Internet is Doing to Our Brains.

At the end of the book, Carr refers to a study by James Evans, a sociologist at the University of Chicago. Evans analyzed 34 million scholarly articles published in academic journals from 1945 through 2005. He looked at the citations in the articles to determine if there were changes in the patterns of citation and research since academic journals shifted from being published in print to being published online.

The assumption was that that making journals available on the Web would “significantly broaden the scope of scholarly research, leading to a much more diverse set of citations,” as Carr put it in his book. But that’s not what happened in the study Evans did.

Since journals moved online, scholars cited fewer journals than they had previously. Scholars also cited more recent articles with increasing frequency. The broadening of information led to a “narrowing of science and scholarship,” Evans found. When Evans discussed his findings in a 2008 article in Science, he explained his counter-intuitive results by suggesting that search engines, which emphasize popular search findings, quickly establish and reinforce a consensus about what information is important and what isn’t. The ease of clicking on hyperlinks leads online researchers to “bypass many of the marginally related articles that print researchers” would routinely skim as they browsed through journal articles and books in a library.

One question that should be addressed in this discussion about the movement described in The Times article should be: Would establishing an online, more open peer review system mimic some of the patterns Evans found? Would the most popular academic submissions get the most votes from the reviewers simply because they were popular? Does all of this actually lead to more narrow scholarship or to better scholarship?

–Wendy Melillo Farrill

—