I wouldn’t be the first person to say there were a few things wrong with peer review as a process, if I said that. I mean, for example, I’ve before now linked to two bitterly humorous takes on how to respond to peer review, and I tend to be able to guess who’s reviewed my work because three of the four or five English-writing academics who know anything about early medieval Catalonia have very distinctive styles. At a broader level, editors don’t ask people they don’t know to review things, by and large, do they, and so peer review is also as with so much else in academic life mediated through personal contacts and who-you-know. But it has to be, because the process relies on trust in others’ expertise, and it’s like Churchill’s democracy, a least worst way to do what needs doing.
Because of this tension, I was very interested in this recent study in The Chronicle of Higher Education, which reports on a study of what other academics think of the peer review process and themselves, or rather, each other, as reviewers. This is not just because the position of the author, Michèle Lamont, closely matches my own: “I don’t think any other process for assessing quality – for example, quantitative rankings on questionnaires – works as well. But we could do it better.” It’s more because of the rather unscientific comparisons she encouraged her respondents to make about other kindred disciplines and their idea of excellence. So, for example, this seems to speak directly to a dispute I’ve been part of before:
In history there is a high degree of consensus among scholars about what is good. But it is not based so much on a common theory, or method, or whether people think the discipline is part of the humanities or social sciences. It’s a shared sense of craftsmanship. People care about whether the work is careful. They believe they can identify careful work. And that they can convince others about it. The degree of consensus has varied over the years. In the 1960s, for example, the discipline was polarized politically. But it has found consensus in the practice of scholarship. Historians believe that contrasts sharply with English literature.
Then from the other side of the divide that historians (and I include myself whole-heartedly here) are perhaps over-zealous in constructing:
Panelists who are in English literature perceive that their discipline has a “legitimization crisis.” Perhaps because of the influence of poststructuralism in the discipline, literary scholars are particularly aware that the standards of evaluation are intersubjective, resulting from the interaction of panelists. They’re ambivalent about how successful a peer-review panel can be. Asked whether “the cream rises to the top,” they emphasize that doesn’t necessarily happen. Some are unsure whether “quality” exists.
There are also summaries for philosophy, anthropology and economics, but the overall conclusions I get from this are, firstly, that those who shout loudest on the Internet are not necessarily a majority, and second, that even the belief that one has empirical standards helps with the strength of the discipline generally. Contrariwise, even if it be true that reality is subjective, there are times when this perspective mostly serves to undermine the ability to make necessary judgements. How far can we after all afford to question our own reality? I think the characterisation of history certainly fits within the discipline as I experience it: there is certainly a sense that caution is a sine qua non, but also a sense that there is such a thing as quality. I wonder if people reading from other disciplines think they get a fair press from this study?
A Corner of Tenth-Century Europe 
Pingback: On peer review and an alternative « A Corner of Tenth-Century Europe
Pingback: In praise of Marc Bloch’s Feudal Society | A Corner of Tenth-Century Europe
Over the years I’ve attended seminars by physicists, chemists, biologists, engineers, metallurgists, mathematicians, medics, economists, geographers, and historians: they have all seemed to me to be in the same sort of game. I have also attended seminars in Eng Lit and Sociology: not so much a different game as a different planet.
Imagine my curiosity on first attending a seminar by a scholar of historical linguistics. Would he be from Planet Fluff? Nope: Planet Firm.
This is close to Snow’s ‘Two Cultures’, isn’t it? My sense is that history can split either way over that line, in fact. There’s definitely some history which considers itself, if not scientific, at least proceeding scientifically with an aim to reconstruct as far as possible the past, even while admitting that that may not in fact be very possible. But there’s also certainly history that regards any attempt at the recovery of what we might call facts misguided and doubts whether such things even exist, so fluid and constructed is human knowledge. And slung uncomfortably across the middle are a small third group who think we are indeed involved in the pursuit of the past but that almost everything we think we can know about it is unfounded…
One can be firm even as one explains gaps in knowledge, gaps in understanding, gaps in imagination.
“This is close to Snow’s ‘Two Cultures’, isn’t it?” Good point, but my experience is that the gap isn’t between science and the humanities, it occurs somewhere within the humanities/social science fields.
Yes, I could probably agree with that; I’ve heard other scientists speak of some of the same realisation. If you wanted harder evidence, this article gives both some reason to believe it and some to think otherwise, though whether journals is the best way to test this (and indeed how they did that) I don’t know…
Pingback: Stand back, all! Something takes shape within the swirling mist! | A Corner of Tenth-Century Europe
Pingback: Universities Are Not… Like Businesses | A Corner of Tenth-Century Europe
Pingback: Why not syndicate with ACI? | A Corner of Tenth-Century Europe