School Information System
Newsletter Sign Up |

Subscribe to this site via RSS: | Newsletter signup | Send us your ideas

March 9, 2009

The Black Box of Peer Review

Scott Jaschik:

Countless decisions in academe are based on the quest for excellence. Which professors to hire and promote. Which grants to fund. Which projects to pursue. Everyone wants to promote excellence. But what if academe actually doesn't know what excellence is?

Michèle Lamont decided to explore excellence by studying one of the primary mechanisms used by higher education to -- in theory -- reward excellence: scholarly peer review. Applying sociological and other disciplinary approaches to her study, Lamont won the right to observe peer review panels that are normally closed to all outsiders. And she was able to interview peer review panelists before and after their meetings, examine notes of reviewers before and after decision-making meetings, and gain access to information on the outcomes of these decisions.

The result is How Professors Think: Inside the Curious World of Academic Judgment (Harvard University Press), which aims to expose what goes on behind the closed doors where funds are allocated and careers can be made. For those who have always wondered why they missed out on that grant or fellowship, the book may or may not provide comfort. Lamont describes processes in which most peer reviewers take their responsibilities seriously, and devote considerable time and attention to getting it right.

She also finds plenty of flaws -- professors whose judgment on proposals is clouded by their own personal interests, deal making among panelists to make sure decisions are made in time for panelists to catch their planes, and an uneven and somewhat unpredictable efforts by panelists to reward personal drive and determination over qualities that a grant program says are the actual criteria.

On diversity, Lamont's research finds that peer reviewers do factor it in (although the extent to which they do so varies by discipline). But peer reviewers are much more likely to care about diversity of research topic or institution than gender or race, she finds.

Posted by Jim Zellmer at March 9, 2009 3:03 AM
Subscribe to this site via RSS/Atom: Newsletter signup | Send us your ideas