The kind of maths we should not do

A lot of the problems any historian of the early Middle Ages faces are about how typical any given piece of evidence is. When so little survives, can we generalise from the few fragments we have across the great spaces where we simply know nothing? I came up against this while writing the post some time back about widow warlords, where as you may remember I wound up trying to argue for a level of social occurrence that could be common enough to be frequent while still being statistically unusual. The question remained then: how unusual? And this led me to thinking about the best evidence I have for female presence in local society, the good old Vall de Sant Joan hearing, and then the temptation stole upon me to do some very bad maths.

Aerial view of the Vall de Sant Joan de les Abadesses

Aerial view of the Vall de Sant Joan de les Abadesses. I know I use this picture a lot but I find it really hard to get tired of. However, I can no longer find where I got it from, so if you happen to own it do let me know…

Y’see, the Vall de Sant Joan hearing seems to be really good evidence for population size, at least by our starvling early medieval standards. We do not know the whole population of the area, but we think we know how many households there were in it, and we know what size it was: 269, by my count, and about 7 km2.1 Now, we could just multiply up, because the Vall de Sant Joan is in some sense a jurisdictional term and we know how many of those there were in the tenth-century county of Osona, give or take a few for changes, and it’s thirty-seven. If each contained this many households, tenth-century Osona would have been a county of nearly ten thousand households.

The town of Sant Joan de les Abadesses, viewed from across the Pont Vell

There probably aren’t that many households in the Vall de Sant Joan now, for a start, though I wouldn’t mind going back again to look (albeit this time with a car). Image by Espencat (Own work) [Public domain], via Wikimedia Commons

In fact, that is really unlikely to be true, because this was a frontier county and we’re counting its very inward corner, where we can document, more or less by the fact that we can document anything but also by the nature of the actual documents, that an ambitious lordship interest was moving people in here and encouraging settlement that is quite unusually dense.2 Such a figure is likely to be a massive over-estimate. So what should we do instead? Now, here the words of my old colleague Ted Buttrey come back to me with force:

“What should we do? We should do nothing. Nothing can be done. There is no solution to this problem, beyond inventing new data to push the inquiry into the realm of the fanciful. This is uncomfortable but it is true. If we allow ourselves, in our frustration, to confect the missing data, we will to that extent have destroyed our own purpose. To create quantitative studies built of imaginary data, to force an answer by assuring ourselves and others that we know what we do not, and cannot, is to compromise everything that we hold important. Each of us builds, and others build upon us: when we dress up guesses as data we do permanent damage to our scholarship, and to the scholarship of others.”3

He is right, of course, I know he’s right. He is also right that bad guesses get out there and get used even when they are explicitly qualified as such.4 So I must not, I must not attempt to correct the above error by breaking the data down, down to the level of households per villa (which would be 12·2 NO STOP IT), and then multiplying up by the number of villae in Osona. I should not do that not least because we don’t know with any certainty how many villae there were in Osona around the year 913, which is when this data would be comparable, probably not even in total for the tenth century which would add many more than there then were and would fail in any way to counter for the factor of population change over that century; I should not do that because, again, villae in the Vall de Sant Joan were probably over-many and over-stuffed compared to other areas and though those two errors might tend in opposite directions, we cannot know that they would cancel each other out; I should not do it because any operation involving multiplying up a small number to obtain a large one necessarily multiplies the error in that number just-as-many-fold; and I should not do it for many other good solid reasons of mathematical rigour. And in fact I will not. But it is sorely tempting, just because it’s hard to rid myself of the idea that if I could allow for enough factors, this would actually be a better basis for early medieval population figures than we currently have anywhere else.5 But every one of those corrections would be another piece of fiction, an error to be multiplied up. Ted again has the correct admonitions:

“When we enter on these kinds of calculation, we can be confident of two things. First, the answer will be wrong. Whatever it is, it will be wrong, since it cannot be right—once you are guessing, the number of possible permutations is gigantic. Worse, where the errors lies, and how serious they are, cannot be determined… Secondly, we can be confident of something else: when we publish this sort of thing, no matter that it be all set about with caveats and qualifications, the very fact that we thought it worth publishing will give it credibility.”6

And that is of course exactly the pain of it; there are figures that are thought credible abroad already that I feel must be wrong, because the person who put them together on the evidence we don’t have made his own set of assumptions about how the lack of evidence should be countered, and now I prefer my assumptions to his and would like to put into circulation alternative figures that are no more verifiable but feel more likely to me. But this will not make things better. Ted can have the last word, albeit he gives it to someone else:

“We should take to heart the dictum of a character in Umberto Eco’s novel, Foucault’s Pendulum, who explains, ‘For every complex problem there is a simple solution; and it is wrong.'”7

1. The reason we assume that the document, which is a vast parchment recording the names of people who swore that Abbess Emma of Sant Joan de Ripoll had been given the valley to settle by her father Count guifré after he expelled the Saracens from it, records households is because about half of its signatories are female, and mostly appear with a male partner. This looks like an attempt to implicate all the conjugal pairs of the valley in what was in fact a political fiction (see Jonathan Jarrett, “Power over Past and Future: Abbess Emma and the nunnery of Sant Joan de les Abadesses” in Early Medieval Europe Vol. 12 (Oxford 2005 for 2003), pp. 229-258, DOI: 10.1111/j.0963-9462.2004.00128.x), but since there are others who aren’t in pairs, it must also be more than that. Hence, households seems likely. The argument is made most thoroughly in Gaspar Feliu i Montfort, “Sant Joan de les Abadesses: algunes precisions sobre l’acta judicial del 913 i el poblament de la vall” in S. Claramunt and M. T. Ferrer i Mallol (edd.), Homenatge a la memòria del Prof. Dr. Emilio Sáez: aplecs d’estudis dels seus deixebles i collaboradors (Barcelona 1989), pp. 421-434. The count of these households I just redid from a spreadsheet I constructed when writing the thesis that lies behind Jonathan Jarrett, Rulers and Ruled in Tenth-Century Catalonia: pathways of power (Woodbridge 2010), where you can find more detail at pp. 35-51. The area I estimate from the map in Jordi Bolòs & Victor Hurtado, Atles del Comtat d’Osona (798-993) (Barcelona 2001), pp. 94-95 at p. 94. Thus my doubtless inaccurate estimation is already one basic source of error!

2. This is the basic story of Jarrett, Rulers and Ruled, as above and also pp. 57-64.

3. Theodore V. Buttrey, “Calculating Ancient Coin Production: facts and fantasies”, The President’s Address in Numismatic Chronicle Vol. 153 (London 1993), pp. 335-351 at p. 351.

4. My best example is another numismatic one, an article by Warren Esty, “Estimation of the size of a coinage: a survey and comparison of methods” in Numismatic Chronicle Vol. 146 (London 1986), pp. 185–215, which pitted a range of statistical techniques then in use to reason up to ancient currency sizes from current surviving evidence against each other by means of a randomly-generated virtual hoard, and concluded that all were more or less rubbish but a combination of two the least rubbish way to do this, the result of which has of course been that his least-worst method is now the standard among those who do such things…

5. I look here with especially narrowed eyes at Georges Duby, The Early Growth of the European Economy: warriors and peasants from the seventh to the twelfth century, transl. Howard B. Clarke (London 1974), pp. 11-13, which does exactly the trick Ted decries (Buttrey, “Calculating ancient coin production”, pp. 349-350) of surrounding the data with all kinds of cavils and conditions and then rhetorically building on it just the same.

6. Buttrey, “Calculating ancient coin production”, pp. 349-350.

7. Ibid. p. 339.

5 responses to “The kind of maths we should not do

  1. Allan McKinley

    You see this form of thinking in most fields of science (in its wider sense of all fields of enquiry), although in the physical sciences it is called modelling and should be physically provable. Unfortunately, there is plenty of cases where a model has been produced and treated as evidence without proof: this underlies most of the public health issues such as fat consumption where established facts suddenly change because an earlier accepted model is no longer substantiated (or a new model is accepted – substantiation requires costly and long studies for public health); or to cite an example our host will be less comfortable with perhaps, the reliance on computer models to show we are experiencing relatively extreme global warming. It is useful therefore to have a reference from one of the historical sciences to argue against this invention of numbers, for which this paragraph is partially a particularly long-winded way of saying thanks.

    In the interests of provoking controversy I would like to model a suggestion that in most scientific (again including the humanities etc) fields this desire to model is linked to a wish to substantiate a longer-term political viewpoint, be this nationalism (English peculiarity expressed through counting hides to find organisation at a high level or some of the frankly scarey work on genetics or psychology in the early days of those subjects), some form of collectivism (the desire to count population and other statistics, including coin use, always seems strongest in those working in the Marxist paradigm; my examples of public health and climate change modelling above are linked to calls for government to act on these findings) or presumably an adherence to a liberal free market view (I am vaguely self-aware enough to be conscious that my biases may exclude modelling practices that draw conclusions with which I agree from my criticism – on reflection this probably means some areas of economics…). For historians, and probably all scientists, the answer might therefore be only to focus on what is evidenced and do not make assumptions built upon assumptions (some assumptions are necessary, obviously). But a little voice in my head is asking me where’s the fun in that?

    • The devil is in what assumptions are necessary, though, isn’t it? It’s very hard to make them all explicit, even to ourselves. They include, I suppose, the underlying predilections like my krypto-Marxist counting of things :-) (It’s not that krypto-, really, is it?) And in substantiation there’s also a difficult line to cut between doing a lot of work to verify something that was obvious and making such assumptions about what doesn’t need substantiation. But your second, controversial, factor is right on the button (and maybe not even that controversial, therefore): given that we don’t check everything, there is choice at work in what we do check, and in there must be subjective biases and the play of politics. That, again, isn’t always explicit, perhaps, but onnected Communities recently made it so for the UK social sciences

  2. I came here to say much the same thing – that given the number of times I’ve seen people in science and engineering try to do statistics with numbers that just aren’t up to it, I’m looking forward to being able to beat them about the head with that quote. Even a historian knows you shouldn’t do that… (And, you know, apologies for the inherent humanities-shaming in that, and I recognise that the point probably comes from a greater sense of respect for one’s responsibilities to the past and future, but on the spot, in the company of engineers, sometimes you need to go for the instant hit.)

    • I’m not sure whether to call Ted a historian, a numismatist or a philologist… but the point would stand anyway, with the proviso that the latter two fields do consider themselves scientific in their methods, and I think that is a lot of why it was someone from those backgrounds standing up and saying that.

  3. Kudos to your pal Ted Buttrey. I get utterly fed up with drivel about “but it’s the best data we have”. The question that matters isn’t whether it’s the best; what matters is whether it’s good enough for the task in hand. C’est bleedin’ tout.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s