Peer review reviewed by its peers (Interdisciplinary Conversation IV)

I wouldn’t be the first person to say there were a few things wrong with peer review as a process, if I said that. I mean, for example, I’ve before now linked to two bitterly humorous takes on how to respond to peer review, and I tend to be able to guess who’s reviewed my work because three of the four or five English-writing academics who know anything about early medieval Catalonia have very distinctive styles. At a broader level, editors don’t ask people they don’t know to review things, by and large, do they, and so peer review is also as with so much else in academic life mediated through personal contacts and who-you-know. But it has to be, because the process relies on trust in others’ expertise, and it’s like Churchill’s democracy, a least worst way to do what needs doing.

Because of this tension, I was very interested in this recent study in The Chronicle of Higher Education, which reports on a study of what other academics think of the peer review process and themselves, or rather, each other, as reviewers. This is not just because the position of the author, Michèle Lamont, closely matches my own: “I don’t think any other process for assessing quality – for example, quantitative rankings on questionnaires – works as well. But we could do it better.” It’s more because of the rather unscientific comparisons she encouraged her respondents to make about other kindred disciplines and their idea of excellence. So, for example, this seems to speak directly to a dispute I’ve been part of before:

In history there is a high degree of consensus among scholars about what is good. But it is not based so much on a common theory, or method, or whether people think the discipline is part of the humanities or social sciences. It’s a shared sense of craftsmanship. People care about whether the work is careful. They believe they can identify careful work. And that they can convince others about it. The degree of consensus has varied over the years. In the 1960s, for example, the discipline was polarized politically. But it has found consensus in the practice of scholarship. Historians believe that contrasts sharply with English literature.

Then from the other side of the divide that historians (and I include myself whole-heartedly here) are perhaps over-zealous in constructing:

Panelists who are in English literature perceive that their discipline has a “legitimization crisis.” Perhaps because of the influence of poststructuralism in the discipline, literary scholars are particularly aware that the standards of evaluation are intersubjective, resulting from the interaction of panelists. They’re ambivalent about how successful a peer-review panel can be. Asked whether “the cream rises to the top,” they emphasize that doesn’t necessarily happen. Some are unsure whether “quality” exists.

There are also summaries for philosophy, anthropology and economics, but the overall conclusions I get from this are, firstly, that those who shout loudest on the Internet are not necessarily a majority, and second, that even the belief that one has empirical standards helps with the strength of the discipline generally. Contrariwise, even if it be true that reality is subjective, there are times when this perspective mostly serves to undermine the ability to make necessary judgements. How far can we after all afford to question our own reality? I think the characterisation of history certainly fits within the discipline as I experience it: there is certainly a sense that caution is a sine qua non, but also a sense that there is such a thing as quality. I wonder if people reading from other disciplines think they get a fair press from this study?

About these ads

One response to “Peer review reviewed by its peers (Interdisciplinary Conversation IV)

  1. Pingback: On peer review and an alternative « A Corner of Tenth-Century Europe

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s