In a volume that is now safely back in Cambridge University Library, I read something that annoyed me slightly in what was otherwise a mildly interesting paper about the use of prepositions instead of declension in the letters of Braulio of Saragossa. I realise that sounds unlikely to be interesting, but it provides a useful way to assess the supposed decline of Classical Latin if you take one of the great minds of Visigothic Spain as the marker standard. (He was pretty solid but there are particular cases, especially the use of per + acc. for agency rather than ab + abl., where he shows that things had changed since Cicero, since you ask. Oh, didn’t you? Well, thank heavens I only gave one example then.) So anyway, what’s the problem, Jarrett? Well, the problem lies in this line about that very phenomenon:
Estos datos son, nos parece, pruebas del empleo creciente de per para introducir el agente, aunque domine con mucho (en un 87%, 53 citas) la construcción clásica con ab.1
I won’t blame you if you don’t see a problem there. They don’t teach historians statistics, and even when historians learn statistics it’s often apparently only so as to mislead and bamboozle. I remember when I had to be told this myself, in fact, and it wasn’t so long ago. But the thing is, in order to have a percentage that’s meaningful you need a hundred of something. If your 87% is made of fewer than 87 units you shouldn’t be using percentage at all.
The Portal of the Judgement at the cathedral of Santa Maria de León
But surely Jonathan this is mere pedantry, I hear you cry. But it isn’t, it can matter. It’s an issue of significance and, in malicious cases, spin. Calling something a percentage implies reliability when compared to other things that are called percentages. Let me give you an example from my own work. One of the many papers currently being rallied between me and reviewers is one about persons with Arabic names in Christian León.2 In it I compare the presence of these people in the documents of three separate archives, this being a rough way to approximate a distinction by geography that would be hard to do by any more precise criterion. So, at one point I write:
At Oviedo, then, in the period of these charts 3 (out of 8 surviving) documents feature Arabic names; at Sahagún 81 out of 134 do (60%); and at León the proportion is 166 out of 226 (73%).
The reviewer that I have still to satisfy wanted to know at this point why I hadn’t given a percentage for Oviedo, and my response, not as expressed in e-mail but as shouted to an empty room when I read the comments, was roughly, “Because it wouldn’t mean anything!” Consider. The percentage, in the sense of ‘what you get if you divide 3 by 8 and multiply by 100’, is 37·5. Now. If I had a round two hundred documents that featured Arabic names from some fourth archive that I had included—and Celanova may even have that many—then if someone found one more, that would make a flybite of difference, half a percent, and my figures would be pretty safe.3 At León, one extra would make less difference even than that; my actual percentage to 2 significant figures would be the same. At Sahagún, 61%, a change but hardly a paper-destroying one. But if at Oviedo they opened a secret chamber and found some originals of documents that Bishop Pelayo altered, and one of them was from the relevant period and out of thirty witnesses at one hearing one, just one was called Zeiti or something, the ‘percentage’ that I’d given at Oviedo would be 44%, a leap of 7 percentage points, implying seven times as much shift as it would be at León or Sahagún. To put it another way, giving a percentage at Oviedo would be to imply that those three people in that tiny sample, years or decades apart from each other, are as significant to our understanding of this group as eighty-five people from León, whereas actually, because they are statistically so insignificant they probably don’t actually tell us anything. They’re so odd that each can probably be explained as a one-off: for example, two of them are frequently-appearing royal courtiers who also appear in the León documents… Whereas 85 people with such names at León do tell us something about what people were at court, perhaps where they’d come from and what it was fashionable to do or be there.
So, next time you’re tempted to use that pair of circles and a slash, well, consider if a fraction might not serve you better if a mathematician happens to be in the audience…
1. Maria Luisa García Sanchidrián, “Del sistema casual a las preposciones : una muestra en Braulio de Zaragoza” in Maurilio Pérez González (ed.), Actas, II Congreso Hispánico de latín medieval (Leon, 11-14 de Noviembre de 1997) vol. I (Leon 1998), pp. 483-491, quote at p. 491.
2. Jonathan Jarrett, “Arabic-named communities in ninth- and tenth-century Asturias and León, at court and at home” in Journal of Medieval Iberian Studies Vol. 2 (London forthcoming).
3. And actually Celanova was the first of these groups to be studied, by Richard Hitchcock, ‘Arabic Proper Names in the Becerro de Celanova’ in David Hook & B. Taylor (edd.), Cultures in Contact in Medieval Spain: historical and literary essays presented to L. P. Harvey (London 1990), pp. 111-26, now much expanded in Hitchcock, Mozarabs in Medieval and Early Modern Spain (Aldershot 2008), pp. 53-68. The Celanova documents have also now been edited in José Miguel Andrade Cernadas (ed.), O Tombo de Celanova: estudio introductorio, edición e índices (ss. IX-XII) (Santiago de Compostela 1995), though lots of problems with this edition have been noted.
A Corner of Tenth-Century Europe 
Two points occur to me:
First off, the cut off point shouldn’t be a hundred. In this case (if I have my Spanish and my math straight) Braulio used “ab” 53 out of 61 times–and 61 is a fairly substantial sample. If another letter was found in which Braulio used “per” once and “ab” none at all, that would bring the percentage down a whole two points, to 85%–so I would rate that within acceptable limits. I think the key is a the size of the sample. If we can be sure that 61 cites is truly representative of the entire corpus of letters, than using the percentage figure is acceptable–if it isn’t, then no figure is. Your 3 of 8 is an obvious example of a sample size that is too small. But if it was 30 out of 80, that would be probably acceptable.
Second, as to your own figures: are your statistics based on the total number of documents, or the total number of names mentioned? The way you phrase it, it sounds like you are basing it on the number of documents, and a document that has one Arabic name out of thirty rates the same as a document that has ninety five Arabic names out of ninetey six–which doesn’t sound quite right to me.
I, for one, would like to hear more about the prepositions. (But the statistics stuff is nice too.)
Chris, now that I’ve fixed the typo in the reference, you can, of course, look it up. (I’ve lost track of where your historical blog was, sorry! Are you still keeping it?) Kishnevi, I take your point about the solidity of the sample, but there’s a semantic one too: per cent means out of one hundred, if the figure is not one that one could reason up to one hundred in safety then that isn’t something that one should say. Also, the calculations I do elsewhere in the paper do count names individually, or at least persons, but in the section I’ve quoted I was interested in showing in what proportion of these documents such persons occurred at all. That’s important because at León they tend to occur in groups, whereas at Oviedo and Sahagún they turn up individually for the most part. A comparison by names therefore makes León seem far more prominent, which is important, but in a way that needs (and gets) separate analysis.
I think that kishnevi is correct; I also believe that the reviewer may have a point, but perhaps not the one that they thought.
You are saying that one should not use percentages to implicitly compare samples that shouldn’t be compared. You are then saying that you shouldn’t give a percentage for Oviedo as the number shouldn’t be compared (quantitatively) against the others as it’s not statistically significant. But just by giving the numbers in the form that you do you implicitly invite this comparison.
I don’t use statistics much in my work (too much theory) so can’t give you a “from experience” result, but I think you’re either going to have to be more pedantic about what should or should not be compared, or less (ie, put the percentage in even though it is statistically problematic); this halfway house doesn’t feel justifiable.
Both you and kishnevi, though your input is gratefully received, are judging a whole paper from a single cite, about three out of eight data charts. It’s not as if this is the single lynchpin of my argument. How, for example, did you know that I don’t go on precisely to say that this basically means we can do nothing with Oviedo? In fact, it goes on,
and then concludes that we can’t really do anything with this sample till the capital moves to León in 914, but that that in itself has historical import. So, y’know, if you want to read the full thing that would be great but otherwise cut my method some slack.
However! I do see your point about consistency. The numbers obviously have to be compared, if only to say, “here is the record: it is a bit useless”. The percentages were originally there mainly because 81 over 134 is hard to do in your head, but I suppose that most people can reason down to `quite like 8 out of 13 ten times over’. So I think the correct thing to do is to eliminate the percentages altogether at this point.
Pingback: Conferring in Naples, IV and final: clarity, confusion, coffee and photos « A Corner of Tenth-Century Europe