It’s a long one, best get a cup of tea and make sure you’re in a comfy seat… Ready? OK.
Science has long struggled to rank the worth of the actual science itself. There are all kinds of metrics to rank journals and papers and the contributions of authors. I have yet to meet one that I don’t profoundly dislike and it really all comes back to the same central point in that all of them seem to be so massively dependent on factors that have very little to do with the actual quality or use of the research. I also know that (and understand why) hiring committees need quick and dirty ways to cut through several hundred applications to a few dozen and these kinds of things will scream out at them for use. I am, frankly, worried that I (and many other colleagues) might be missing out badly for no other reason that these metrics can be biased against or towards various kinds of research or researcher, regardless of their actual ability or the quality of their work.
Have a few too few citations or lose a few points on an index and you might never make the shortlist no matter how good or valuable your research is. Moreover, this will promote practices that are at best, not conducive to good science.
Let’s start with the recent version set up on Google Scholar (for those interested, here’s me). This is pretty standard as they go, it looks at my citations, top papers, and a couple of indices that basically look at how often my papers get cited. I have a few problems with some of the things it’s doing at the moment and it’s new and needs work, but the basic principles are the same here as on other metric sites.
So what’s the beef? If my work is good and is being read and cited then my ranks will go up right? Well yeah, in theory. But are these things happening correctly? What things might skew how often (or rarely) a paper is cited (or counted as being cited) and how relevant is that?
First off, there and obvious one. If you’re cited in a paper you get one citation. But someone whose work is critical to a manuscript might be cited dozens of time, but a single tangential point or general review paper might pick up a mention somewhere. Both are scored equally – one citation. So right off the bat, two papers can appear to be equal when they are not. I was recently delighted to get a copy of the new Scipionyx monograph from Cristiano Dal Sasso. Included was a note telling me I was being gifted a copy as I was one of the most cited authors in the paper. Only 1 of my papers was cited in a reference list that ran to something like 400 papers – so in short on this occasion at least a single ‘ping’ for me doesn’t represent the significance of the paper.
Secondly, things can be cited often even if, or even because, they are wrong? How many papers are there out there on birds and their dinosaurian ancestry which mention the BAND group? Most of them give it at least a token mention in their introductions, which means the same half dozen BAND papers rack up the citations even though they are only ever being cited by people saying they are wrong! On a not unrelated note, a big pool of papers that make the same point may be sampled more or less at random (no need to cite 50 papers to say birds are dinosaurs) or the same few pick up all the hits, even if there are better or more appropriate ones out there.
There’s also a lot of journals out there which simply don’t get picked up by the indices at the moment because they’re not considered of sufficient calibre or are simply rather obscure and so citations in those journals won’t be added to the list. You can make a case that if only minor papers are citing your work then it can’t be that important, but I think this isn’t right. After all, the biggest journals count just as much as the smallest ones, there’s no direct rank by journal quality, and I’d argue there’s a bigger gap between the biggest and smallest journals that would count than between the lower ones that are ticked off and the best that are not.
Subject with numerous researchers are likely to rack up the citations far faster than smaller research groups. There are probably four or five people working on theropods for every one who works on pterosaurs, so assuming people publish similar papers at similar rates, theropod papers might get four citations for every one a pterosaur paper picks up, even if both are of hypothetically similar values in quality and usefulness. Chance can play a big part here too, I remember a theropod-worker colleague of mine noting wryly that his one paper on a mammal (a tooth he’d happened to find in the field which turned out to be very important) had accrued more citations than his entire back catalogue of dinosaur research combined.
Some of these ranks are dependent on rates of citations too, or only count those accrued within the first 2-3 years of publication. Well again, some journals are much faster than others, indeed some entire fields are. I know in some branches of science, 2-4 weeks in review is normal, and submission to publication can be in weeks. There are few palaeo journals that are not measured in many months for those kinds of turnaround times, so it’s simply harder to get a few citations that quickly.
So all of these have obvious problems. Someone can write a terrible paper on HIV say, but with lots of researchers out there, and all of them keen to stick the knife in, it could rack up hundreds of hits fast in major journals. But a truly brilliant and groundbreaking paper in a relatively obscure palaeo journal on a subject with only a handful of specialists might take years to get half a dozen. According to these indices (or for that matter an outside observer or non-expert) the former will look much more appealing than the latter.
Moreover, these things can also be manipulated, or at least have the potential to be. People can cite themselves where they don’t need to, to get a few more hits in. Cartels might form of people citing each other to jack their citations up, or supervisors (or even referees and editors) can pressure people to cite their work. People might start splitting big papers into multiple smaller ones, each of which can then cite a few things and bulk the number up again. Or you can put each other on your papers to bump up the number of papers you have apparently contributed to and get all the free citations that go with it down the line. A brilliant student might still struggle to get papers published in good journals if they are not getting the support they should, and a poor student can be gifted credit on papers in major journals by a generous and talented researcher (and I know the latter already happens – it’s dispiriting to meet an alleged author of a paper and discover they don’t speak English, or on one memorable occasion, realise they are on the paper you’re talking to them about….).
Other metrics have been tried or are being considered, like numbers of views or downloads, or number of pages published. Again, this will vary enormously between different fields but can also be screwed up. I remember my Microraptor paper coming out and a colleague got it early and e-mailed it to a couple of massive mailing lists. Within minutes, hundreds of researchers had a PDF (whether they wanted it or not). A few days late I checked the PloS metrics and according to that about half a dozen people had downloaded it, and only a few dozen had visited the page. But then that would happen, no-one needed it because they already had it! But not to worry, it could always be jacked up, just set it as a required reading for a course taught to a few hundred undergrads and the numbers can soon skyrocket. Or be savvy enough to get it pimped on the right media site and you can drive thousands of people to the page.
What about numbers of pages published? Stick to small format journals, make sure your figures are big, pack in extra references and use some nice big tables. The number of pages will soon go up.
In short, I have yet to see a metric which is anything but highly capricious and makes no real measure of all of these problems. Bad papers in popular fields with fast turn around times and short manuscripts will surge ahead of a field with few researchers who tend to turn in long papers of superb quality. Moreover, there’s an obvious risk of escalation – people can start tailoring their work to these ends, focusing on more popular fields, keeping papers short, bumping up their citations (especially to their own work or those of close colleagues) and so on. None of this is good for science.
Discussions with a number of colleagues show that hiring committees, promotion boards and grant bodies are actually using these metrics, or ones like them, to decide things like who gets money or a job. For someone working in a field where turn around times are huge, papers often long, and the number of colleagues small, you can see why I’m worried. I may be competing for positions with people who have apparently a much greater academic record simply because they work in a popular field. I can’t and don’t expect a prospective employer to read, let alone understand, a whole bunch of papers on theropod ecology, HIV transmission and fish mechanics, but equally, if you’re only evaluation is an H-index or the number of citations in 2 years it’s clearly weighted (or can be) for one field and against another. Sure, a theropod researcher is going to spot the better student of a pair or people working in the field, or understand that the egg specialist is likely to suffer from a lack of citations compared to the maniraptoran worker, but that’s always been the case.
I freely admit that there’s no obvious solution (better minds than me have looked I’m sure). And yes, there is certainly something to be said for these metrics: good papers will, I’m sure, on average, get more citations than bad ones. But at the same time I think it’s hard to look at these and how they are built and think that it is entirely fair and ‘on average’ is fine until you discover you’re the one at the end of the statistical tail and are getting shafted by it. Some fields, some people, are going to suffer. And these look like they can be manipulated relatively easily in ways that will not benefit the subject but will those who bother to do so.