A combination of a post by Andy Farke on the metrics of PLoS ONE and a comment thread over on SV-POW has turned me to looking at the citation metrics of Google Scholar for various articles and indeed, authors. I don’t think there’s any disagreement that Scholar gives rather higher values than do set-ups like Web of Knowledge, but what is behind this? (And yes, for the record I still don’t like ranking things in general, and nor do a lot of people).
The general answer is ‘blog posts get counted’ and while some certainly do sneak in, in my experience it’s not massive. I certainly have picked up a few hits from blog posts, but not many. What I have got rather more of though is hits from less well-known journals and indeed book chapters – things that don’t always flag up on WoK indices and the like.
This to me is a much more significant issue. While there are papers that are unreviewed papers out there in some (even good) journals, and some good and bad books (and indeed grey literature) there are plenty of good papers in minor journals and book chapters. While I think it would be reasonable that some things should be excluded from such metrics (unreviewed letters and comments and replies I don’t think should count), any paper has potentially good science in it and if it’s building on your work, you should be credited with it. (Essentially this is the approach now being pushed by the Wellcome Trust – it’s not the journal the paper is in but the paper that has the merit).
In short, while GS might well be overstating things, WoK and the like and also probably understating. I looked at a few of my papers in more detail to highlight a couple of things I found.
WoK gives my paper on the soft tissue of Jeholopterus just 10 citations, but GS some 23! That’s a massive difference. But looking over the latter in detail, it’s not massively overstated. There is a duplication in there (so the same paper appears twice for some reason), and one is from a student thesis, and one citation is from a review from the Chinese Academy of Sciences journal (so not really a formal paper, but at the same time not quite irrelevant either and are certainly not blog posts or media bits). So we can certainly remove two of these, and arguably another two. That drops it down to 19, but this is still almost double the 10 citations of WoK. All the other 9 ‘extra’ citations are in published, peer-reviewed journals (well, as far as I can tell). Oddly, WoK also has once citation GS doesn’t, from the little editorial review at comes at the front of the volume in which this paper appeared, which I would hardly count as a scientific use of the work.
Similarly the paper describing Shaochilong gets 11 vs 8 with the 3 extras in GS all coming from proper papers. The paper on sauropod necks is more of a problem – fully 3 of the 5 citations assigned to it by GS are from blog posts. But the others are from Lethaia and Biological Reviews – both well-known journals with proper Impact Factors and the rest. One papers is properly published and the other is ‘in press’ (but an accepted version, formatted etc. and with a DOI and has been out online for 6 months now) yet only 1 is picked up by WoK. Either it’s slow (not great but not the end of the world) or not counting ‘in press’ papers (which seems odd, and in the electronic age, a bit of a weak excuse) or it’s simply not counting them all.
Finally I also know I’m missing some citations even on GS (or at least by their standards). The Beijing Flugsaurier meeting had extended, reviewed, abstracts published in one of the normal Chinese journals – a far from major international journal and probably little known, but a bona fide research, reviewed journal. Even so, I have picked up a couple of citations from this volume of abstracts – but not all of the ones I should get (since it’s easy enough to flick through and check) and indeed one of my pieces in there isn’t even recorded on the list of things I have written. So clearly GS is recording some of the abstracts as papers and citations but not others.
In short, GS does indeed overcount a bit. But having had a good look through my records at least, not that much and yet I know I’m still ‘owed’ a few more which would compensate at least a little for this. And moreover WoK is so massively undercounting that in terms of what my ‘true’ citation record is, I think it’s likely closer to GC than WoK. Now obviously I still don’t like these ranks in general, and this is not an especially detailed look at only my own situation which could be very unrepresentative, but overall I don’t think either is especially good and I must say I can’t find much to really recommend WoK in any particular way over GS. It’s certainly more widely used and respected, but speaking for my record it doesn’t seem to be any more accurate or representative of how my work is being used and cited and in fact, probably less so.