On why I really, really dislike all things that rank papers

It’s a long one, best get a cup of tea and make sure you’re in a comfy seat… Ready? OK.

Science has long struggled to rank the worth of the actual science itself. There are all kinds of metrics to rank journals and papers and the contributions of authors. I have yet to meet one that I don’t profoundly dislike and it really all comes back to the same central point in that all of them seem to be so massively dependent on factors that have very little to do with the actual quality or use of the research. I also know that (and understand why) hiring committees need quick and dirty ways to cut through several hundred applications to a few dozen and these kinds of things will scream out at them for use. I am, frankly, worried that I (and many other colleagues) might be missing out badly for no other reason that these metrics can be biased against or towards various kinds of research or researcher, regardless of their actual ability or the quality of their work.

Have a few too few citations or lose a few points on an index and you might never make the shortlist no matter how good or valuable your research is. Moreover, this will promote practices that are at best, not conducive to good science.

Let’s start with the recent version set up on Google Scholar (for those interested, here’s me). This is pretty standard as they go, it looks at my citations, top papers, and a couple of indices that basically look at how often my papers get cited. I have a few problems with some of the things it’s doing at the moment and it’s new and needs work, but the basic principles are the same here as on other metric sites.

So what’s the beef? If my work is good and is being read and cited then my ranks will go up right? Well yeah, in theory. But are these things happening correctly? What things might skew how often (or rarely) a paper is cited (or counted as being cited) and how relevant is that?

First off, there and obvious one. If you’re cited in a paper you get one citation. But someone whose work is critical to a manuscript might be cited dozens of time, but a single tangential point or general review paper might pick up a mention somewhere. Both are scored equally – one citation. So right off the bat, two papers can appear to be equal when they are not. I was recently delighted to get a copy of the new Scipionyx monograph from Cristiano Dal Sasso. Included was a note telling me I was being gifted a copy as I was one of the most cited authors in the paper. Only 1 of my papers was cited in a reference list that ran to something like 400 papers – so in short on this occasion at least a single ‘ping’ for me doesn’t represent the significance of the paper.

Secondly, things can be cited often even if, or even because, they are wrong? How many papers are there out there on birds and their dinosaurian ancestry which mention the BAND group? Most of them give it at least a token mention in their introductions, which means the same half dozen BAND papers rack up the citations even though they are only ever being cited by people saying they are wrong! On a not unrelated note, a big pool of papers that make the same point may be sampled more or less at random (no need to cite 50 papers to say birds are dinosaurs) or the same few pick up all the hits, even if there are better or more appropriate ones out there.

There’s also a lot of journals out there which simply don’t get picked up by the indices at the moment because they’re not considered of sufficient calibre or are simply rather obscure and so citations in those journals won’t be added to the list. You can make a case that if only minor papers are citing your work then it can’t be that important, but I think this isn’t right. After all, the biggest journals count just as much as the smallest ones, there’s no direct rank by journal quality, and I’d argue there’s a bigger gap between the biggest and smallest journals that would count than between the lower ones that are ticked off and the best that are not.

Subject with numerous researchers are likely to rack up the citations far faster than smaller research groups. There are probably four or five people working on theropods for every one who works on pterosaurs, so assuming people publish similar papers at similar rates, theropod papers might get four citations for every one a pterosaur paper picks up, even if both are of hypothetically similar values in quality and usefulness. Chance can play a big part here too, I remember a theropod-worker colleague of mine noting wryly that his one paper on a mammal (a tooth he’d happened to find in the field which turned out to be very important) had accrued more citations than his entire back catalogue of dinosaur research combined.

Some of these ranks are dependent on rates of citations too, or only count those accrued within the first 2-3 years of publication. Well again, some journals are much faster than others, indeed some entire fields are. I know in some branches of science, 2-4 weeks in review is normal, and submission to publication can be in weeks. There are few palaeo journals that are not measured in many months for those kinds of turnaround times, so it’s simply harder to get a few citations that quickly.

So all of these have obvious problems. Someone can write a terrible paper on HIV say, but with lots of researchers out there, and all of them keen to stick the knife in, it could rack up hundreds of hits fast in major journals. But a truly brilliant and groundbreaking paper in a relatively obscure palaeo journal on a subject with only a handful of specialists might take years to get half a dozen. According to these indices (or for that matter an outside observer or non-expert) the former will look much more appealing than the latter.

Moreover, these things can also be manipulated, or at least have the potential to be. People can cite themselves where they don’t need to, to get a few more hits in. Cartels might form of people citing each other to jack their citations up, or supervisors (or even referees and editors) can pressure people to cite their work. People might start splitting big papers into multiple smaller ones, each of which can then cite a few things and bulk the number up again. Or you can put each other on your papers to bump up the number of papers you have apparently contributed to and get all the free citations that go with it down the line. A brilliant student might still struggle to get papers published in good journals if they are not getting the support they should, and a poor student can be gifted credit on papers in major journals by a generous and talented researcher (and I know the latter already happens – it’s dispiriting to meet an alleged author of a paper and discover they don’t speak English, or on one memorable occasion, realise they are on the paper you’re talking to them about….).

Other metrics have been tried or are being considered, like numbers of views or downloads, or number of pages published. Again, this will vary enormously between different fields but can also be screwed up. I remember my Microraptor paper coming out and a colleague got it early and e-mailed it to a couple of massive mailing lists. Within minutes, hundreds of researchers had a PDF (whether they wanted it or not). A few days late I checked the PloS metrics and according to that about half a dozen people had downloaded it, and only a few dozen had visited the page. But then that would happen, no-one needed it because they already had it! But not to worry, it could always be jacked up, just set it as a required reading for a course taught to a few hundred undergrads and the numbers can soon skyrocket. Or be savvy enough to get it pimped on the right media site and you can drive thousands of people to the page.

What about numbers of pages published? Stick to small format journals, make sure your figures are big, pack in extra references and use some nice big tables. The number of pages will soon go up.

In short, I have yet to see a metric which is anything but highly capricious and makes no real measure of all of these problems. Bad papers in popular fields with fast turn around times and short manuscripts will surge ahead of a field with few researchers who tend to turn in long papers of superb quality. Moreover, there’s an obvious risk of escalation – people can start tailoring their work to these ends, focusing on more popular fields, keeping papers short, bumping up their citations (especially to their own work or those of close colleagues) and so on. None of this is good for science.

Discussions with a number of colleagues show that hiring committees, promotion boards and grant bodies are actually using these metrics, or ones like them, to decide things like who gets money or a job. For someone working in a field where turn around times are huge, papers often long, and the number of colleagues small, you can see why I’m worried. I may be competing for positions with people who have apparently a much greater academic record simply because they work in a popular field. I can’t and don’t expect a prospective employer to read, let alone understand, a whole bunch of papers on theropod ecology, HIV transmission and fish mechanics, but equally, if you’re only evaluation is an H-index or the number of citations in 2 years it’s clearly weighted (or can be) for one field and against another. Sure, a theropod researcher is going to spot the better student of a pair or people working in the field, or understand that the egg specialist is likely to suffer from a lack of citations compared to the maniraptoran worker, but that’s always been the case.

I freely admit that there’s no obvious solution (better minds than me have looked I’m sure). And yes, there is certainly something to be said for these metrics: good papers will, I’m sure, on average, get more citations than bad ones. But at the same time I think it’s hard to look at these and how they are built and think that it is entirely fair and ‘on average’ is fine until you discover you’re the one at the end of the statistical tail and are getting shafted by it. Some fields, some people, are going to suffer. And these look like they can be manipulated relatively easily in ways that will not benefit the subject but will those who bother to do so.

43 Responses to “On why I really, really dislike all things that rank papers”

  1. 2 Mike Taylor 04/01/2012 at 10:23 am

    So true. Every metric measures the wrong thing. Because the one thing we should care about is how GOOD a work is, and there’s no way to determine that from citation count, page count, download count, etc. Hiring committees using such metrics get exactly what they deserve: less than the best candidate.

    • 3 David Hone 04/01/2012 at 10:26 am

      And more my worry Mike, good candidates don’t get jobs.

      • 4 Mike Taylor 04/01/2012 at 10:44 am

        Of course, that is a much more important and serious consequence! The point I was trying to make is that when selection committees count up points rather than taking the time and trouble to engage with someone’s research record at more than a superficial level, they cheat themselves as well as the candidate. Really: when they are deciding whether to take on an expensive professional on a long-term contract, you’d think they’d make the effort to consult an expert in the relevant field.

      • 5 David Hone 04/01/2012 at 11:04 am

        Well I do sympathise with the issue for them Mike, I know of jobs that have attracted hundreds of applications. If that is to teach something like ‘evolutionary biology’ then it’s jsut as valid for someone like me to apply as someone who works on fish genetics, or caterpillar populations or whatever. How can any committee judge those people against each other (and everything in between). And they can’t realistically seek out hundreds of referees / experts either, and even if they did, there’s always the risk they’ll pick someone apparently independent who has nothing but praise or borderline slander to say about a candidate, regardless of real value. I can only imagine it’s a nightmare. But I *still* don’t think these metrics help.

    • 6 Richard 04/01/2012 at 11:11 am

      I broadly agree with most of what Dave has said about citation metrics, although I don’t know what the alternative is to be honest, given that science is fragmenting to the point where it can be difficult to accurately judge the work of anyone outside of your own field. However, Mike – where’s the evidence that jobs are not going to excellent candidates? All of the palaeontology jobs I can think of recently went to exceptional candidates, with other exceptional candidates shortlisted and a lot of outstanding scientists not even making the shortlist. The real problem is not that jobs are being filled with substandard candidates, it’s that there is nowhere near enough jobs for the number of excellent palaeontology PhDs that are being produced at the moment.

      • 7 Heinrich Mallison 04/01/2012 at 11:13 am

        You have a point here: if you just roll the dice and pick one out of 100, with 98 excellent and two OK candidates having applied, you have a high chance of getting an excellent candidate.

      • 8 Mike Taylor 04/01/2012 at 11:26 am

        Richard asks: “Mike – where’s the evidence that jobs are not going to excellent candidates?”

        Oh, no evidence at all. All I’m saying is what if committees do this, then that will be the result. I agree that in palaeo there are far more excellent candidates than there are posts for them to fill, so a search committee could hardly screw up if it tried. I assume that’s not true in all the other sciences!

  2. 9 David 04/01/2012 at 11:23 am

    The main valid point here IMO is that you must compare all citation metrics to norms in your discipline or subfield. So for example using the Journal Citation Reports, journals should be ranked relative to the top journal in their field. Based on your Google Scholar citations (by the way Google Scholar picks up everything on the web) your PhD age adjusted h-index i.e. Hirsch’s m is 1.66 which is very respectable regardless of field (same as mine more or less). Within vertebrate paleontology your citation record is very good (your advisor Benton has a lot of citations) but other big names (search “dinosaurs” or something on the Google Scholar Citations site) have a lot less than him. Rather than complain about citations metrics I would compare your record to others in your field in your cover letter (I did when I was applying up till August and I know some of my letter writers did too). You don’t need to name anyone, just say or get letter writers to say that this is good for your field.

    • 10 David Hone 04/01/2012 at 11:31 am

      Minor point first, Google Scholar doesn’t pick up *everyhting* online, there are very few blogs etc. that have sneaked in there, though how it picks them god only knows as it seems in some cases to have given me hits for a single post, but not for others that are cited in the exact same post.

      Anyway, more generally that’s not a bad point. The issue for me though (which ties in with Richard’s comments) is that I’m more interested in (and I think most suited for) jobs in biology departments etc. and I really have been competing with people from across the spectrum of biology (in my last effort there really was a fish population ecologist for interview before me). I guess it would help me to be able to say “I’m doing well compared to other vertebrate palaeo / dinosaurs / geology people” but that still might not cut much slack if the overall numbers are massively below the next guy who works on cancer say (a bit extreme, but not implausible). And again, looking at something in detail (comparing lots of dinosaur people for a dinosaur job, or even me vs fish guy) doesn’t necessarily work when the comittee has to whittle down 500 people to 50 or even 10 quickly.

  3. 11 David 04/01/2012 at 11:30 am

    PS – having been on a few selection committees none of us have used citations as a cut-off or proxy for choosing the best candidate. But we do look at the citations of finalist candidates and if someone is getting hardly any citations relative to field norms we’d begin to question the quality of the work and vice versa. We are an interdisciplinary school in the social sciences. Now in biomed or something maybe they use citations more mechanically, but anyone with a paper in Nature like you would instantly be looked at I think. We are impressed by people with papers in Science and Nature too (in hard core econ departments they’re not).

    • 12 Mike Habib 04/01/2012 at 2:54 pm

      I have also acted on a few search committees now, and we also have used the same approach that David describes: we compare the overall records applicants to some kind of standard for their field. This is much improved over an unweighted comparison, but there are still some significant concerns.

      One of those concerns, which David H noted astutely in the blog entry, is that weighted comparisons within fields does nothing to avoid simple preference for a highly cited field over a less cited one. In competitions for jobs as general in nature as what David H describes (departments of “Biology” for teaching Evolutionary Biology) it can very well become a battle of the disciplines more than a battle of the researchers. A similar effect can occur when overall readership is emphasized for the sake of PR: many institutions are eager to pull in individuals that get a lot of public bang for their papers (I have been on the winning end of this trend, in fact – I have not yet accrued many papers, but I tend to get a lot of publicity and readership out of each paper). On that front, subjects such as paleontology can actually do quite well, but it is still a bit touch and go.

      Honestly, I think the current metrics can be useful, but I also agree that they can be problematic. It may also be worth asking whether or not hiring bodies are actually looking for the “best” scientists. I have the impression that, in terms of employment, many institutions are not actually looking for the most brilliant scientists. Instead, they are looking for productivity and notoriety. We might hope that all three aspects would go hand-in-hand, but there is no reason that they must.

      • 13 David Hone 04/01/2012 at 3:03 pm

        Yeah I think the simple thing is that people dreamed up these metrics because, well they’re not inherently a bad idea and are useful for individuals and others to keep track of things. However my suspician is that because *they were already there* they started to be used to rank people etc. Not because they were necessarily good, or accurate, or reflected X or Y but simply because they were accessible and already calculated for everyone. And that’s not really how you go about creating something to actively compare work like this.

  4. 14 Schenck 04/01/2012 at 3:34 pm

    Is the point of the job to advance science or advance the institution? I think in nearly all cases the institution granting the job wants the institution itself to be advanced.
    So given that, most citation metrics seem pretty sensible. Someone like Feduccia, even though he’s ultimately wrong, he’s generated a lot of scholarship in ornithology-paleontology, and that counts for something for science itself. For the institution, the high ranking means he’s productive, that students will have something to do under him, and that he’ll bring in grant money, which the inst. skims off of.
    Between ‘impact’ rankings and grant funding, that really tells any organization hiring what they’re really interested in.

    Think of it another way, /should/ the institution be the one deciding if the science is ‘good’ or not? Do you really want to leave it up to them?

    • 15 David Hone 04/01/2012 at 3:45 pm

      I’d take issue with ” even though he’s ultimately wrong, he’s generated a lot of scholarship in ornithology-paleontology, and that counts for something for science itself. For the institution, the high ranking means he’s productive, that students will have something to do under him”

      But think of the work someone like that could do that’s not bad. Or the effort taken to produce good science rather than argue with them. And the training the students would get from someone better. ‘Bad’ science does push people to produce better science yes, but they still direct money, time, research and trainsing from better people. That’s bad.

      • 16 Schenck 06/01/2012 at 4:18 pm

        I’ll agree that ‘correct’ science is preferred to ‘incorrect’ science, but consider Wegener. His ‘impact’ in his time would be from citations that reject his ideas, so no one should hire him. Turns out, he was right.

        Further, consider geosyncline theory, which beat out continental drift. It was totally wrong, but widely accepted. How do we we assess the impact of the grandees of geosyncline theory? On the one hand, they forced the ‘drifters’ to really sharpen their arguements. On the other hand, they were extremely productive while being more or less wrong. And at the same time their productivity did just what you are worried about, it meant that continental regional geology was very active, but global geology didn’t have a lot of people working in it, and also that marine geology was understudied and ‘de’ mainstreamed.
        And to be clear I wouldn’t compare Feds to Wegs at all.

        Science is especially tricky, we have no idea if the work is good or bad or right or wrong, really, it’s at best provisional, the most we can really say is ‘so and so is doing the right /things/.’ But that’s a different issue than citation impact measurements, so I am getting off track.

      • 17 David Hone 06/01/2012 at 4:28 pm

        Well I would say our ways of doing work is more formalised these days and ideas (even crazy ones) do get a lot of investigation and evaluation by lots of people in lots of ways. I see your point, but at the same time I don’t think we’re in a world anymore where these kinds of things could happen anything like as easily. I guess another way of putting it is has there been anything like this in the last 20-40 years?

        I’m not talking about theoretical stuff where obviously it’s tricky but in a field where people can go out and measure and evaluate lots of evidence quickly, has there been something like drift that sat around being ignored or considered wrong before suddenly springing back to life and overturning another major theory? I can’t think of one. Lots of to-ing and fro-ing on issues (like bird origins) but nothing quite like that (that I know of). I suspect not because new ideas get tested much more rigorously and often come with stacks of supporting data, and researchers tend to be more open and accomodating about the problems in the existing hypotheses.

        But yes, it is a different kind of issue. 🙂

  5. 18 Foghorn W. Leghorn 04/01/2012 at 5:47 pm

    In response to Mike Habib’s comment, if Institutions are looking for ‘productivity and notoriety’ then why is it that Dave Hone can’t seem to get a job? He is certainly productive (I checked out his google site) andd notorious. Perhaps an online petition would shake some of these petty-minded academics out of their trees?

    • 19 David Hone 04/01/2012 at 6:05 pm

      Well at least in a very large part because there are so few jobs out there in academia full stop, and even fewer in vertebrate palaeontology, and a great many candidates. There are people a dozen years older than me competing for positions as well as those my age and newly minted PhDs. Competition is, frankly, brutal and you need some luck to go with ability and experience. Which is partly my concern outlined above.

      I appreciate the support and the sentiment, but who would you petition? There just aren’t enough jobs going and a lot of people who want them ans are well qualified.

    • 20 Schenck 06/01/2012 at 4:25 pm

      But why should ANY institution hire a paleontologist, and a vertebrate paleontologist, and an archosaur vertebrate paleontologist at that?

      Most people that study paleontology, I think, have been told that, by the very fact that they’re entering this field they are radically reducing their change of getting gainful employment. In truth, there are WAY too many students of paleontology out there (I honestly believe at least a small part of this issue is that, I kid you not, when Jurassic Park came out, a whole generation of kids got ‘inspired’ by /it/, and now we have a ‘bubble’ of paleo-students, sort of like the demographic bubble of baby-boomers).

      If you’re on a hiring committee for a biology department, and you have a choice between a person doing modern ecological studies versus a “mammal-like reptile” paleontologist, who’re you going to hire? Who is more likely to bring in funding over the course of their academic career, especially given broad and deep cuts to science research?

      Or consider that many Universities have switched from Geology departments to Environmental Studies departments.

      Now, to be clear, I think paleontology, especially dinosaur paleontology, is the ‘kick ass king of all science’, but let’s be realistic.

      • 21 David Hone 06/01/2012 at 4:57 pm

        The usual answers really:

        – all knowledge is valuable, and we don’t always know which will be more useful / important later on

        – different field cross-polinate differently and can add substantually to a field you would not expect (like the work i’ve assisted on with pterosaur fossils influencing aircraft and material design)

        – different people can teach different things (the logical extension of your argument is to hire only cancer researchers as they’ll get the most money and do the most exciting stuff, but then you can’t teach much botany, palaeo, ecology…..)

        – there might be more funding in ecology vs palaeo, but there’s not enough funding for all of either (i.e. 1000000 ecology people will have more money to plsu but more way than 10 palaeo people will)

        -students go to uni for a rounded education.

        I think we *are* realistic. Competition is brutal, full stop. There’s not many jobs. But equally there’s scope for all fields, and while you can make a case for cancer research gets more money and ecology is hot this year and whatever, at the end of the day there should be a place for fossil echinoderm moulting research as well as HIV cures (even if the latter should get more money).

  6. 22 Paul Barrett 04/01/2012 at 11:04 pm

    Most job search committees have a list of criteria that applicants are checked against – few of these rely explicitly on looking up scores on Google or Researcher ID. Those candidates that check these boxes (which are skills and experience related as well as output related, the latter being the only one that might rely on citation counts) get on to the short lists. This has been the case on every search I’ve been involved in. References also play a major part in getting on short lists: pick your referees carefully.

    Despite some outliers caused by people who consistently publish work that is only cited as a ‘contra’, the numbers of citations almost always reflect either longevity (older researchers accrue more citations on average than postdocs), genuine prominence within a subject area, or membership of a dynamic or large subject area. Citations within VP follow these trends: the established Bentons and Norells of the world have high numbers of citations and H-indexes, ahead of mid-career workers, who are in turn ahead of more recently minted post-docs and students. Exceptions to this general pattern are more likely to be excellent, prolific workers than crap prolific ones (in most decent institutions, the latter do not get tenured jobs).

    As for using quality as a criterion it would be totally impossible to operationalize in any simple format. Quality is a subjective criterion, whose definition is going to vary from individual to individual and this can only be assessed with the benefit of long-term impact and hindsight. Number of individual citations within a paper would be futile – I probably cite papers that I criticize more often within a paper (i.e. providing a critical discussion, with correct attribution of the information being reviewed) than the good papers that I cite in support of my arguments (which are evidently good and don’t need continual dissection).

    Railing against the metrics is futile – they’re a rough imperfect guide, everyone knows this. That’s why job committees use plenty of other criteria and interviews to sort people out. All small subject areas are at a disadvantage when applying for general posts, VP is not especially vulnerable. Indeed, VP enjoys the advantage of being a media-friendly, public outreach aware science: many institutions are quite happy to absorb VP people given the public awareness they generate in lieu of mega-grants or a constant stream of high-impact papers. I can think of several UK posts that have gone to VP people in large Departments where VP is only a marginal subject. They ended up recruiting excellent people. Richard is correct to note that the major problem is that there are simply too few jobs for the large number of talented PhD students/postdocs currently in the system.

    • 23 David Hone 04/01/2012 at 11:26 pm

      Interesting stuff Paul. I won’t disagree that *my* primary concern is job hunting because, well, I need a job. However, informal chats with a number of colleagues in VP and other fields show that this these metrics are being used in places. I’ve been told by various people from a wide range of scientific subjects that these kinds of things have explicitly been used to rate people for promotions and grants at least, and a senior colleague of mine advised me that these kinds of ratings would help me get a job.

      Now clearly other criteria are being used and I’m not pretending that bad people are getting jobs or that any one person or group is especially vulnerable. However, as long as I’m being warned by some people that these things count and by other that I need to work on them, then I’m going to worry a bit, and I’d imagine there are *some* committees *somewhere* who are using them more than they should. And I can’t be the only person who is getting this advice and by extension I assume other people are worried by it or are actively trying to influence it.

      As such I still don’t like them, even if the effect is minor overall and I am being overly concerned, I’d still call it an insidious influence on some. And judging by some comments here from other people and what I’ve heard elsewhere I’m not alone.

  7. 24 Robert Sloan 05/01/2012 at 5:40 am

    Wow. Dave, you’ve given me an eye opener in this rant. It doesn’t bode well for science the way these metrics can skew in favor of a lousy frequently flamed researcher over a brilliant researcher in an obscure specialty even if that brilliant researcher’s done groundbreaking science that affects far more than his specialty. Let alone what happens if someone who’s brilliant has done his or her very first paper and it’s groundbreaking. I’m stunned. I sure hope these are not the only criteria, because that’s just loony.

    I hope they consider blogs carefully.

    Archosaur Musings combines great readability to the nonacademic web surfer with deep science and sound information. It’s a winner. I categorize this blog with Stephen Jay Gould’s “This View of Life,” the column that got collected into all those lucrative essay volumes that usually didn’t even get pretty paleo-art on the cover. If you ever published an essay collection or mainstream book on any paleontology topic, I’d preorder it – and hope you were planning a Kindle version so I don’t also need to buy a bookcase I don’t have space for.

    If I were hiring, I’d see this blog as proof you can get Biology 101 into the heads of freshmen who might have graduated high school without even getting to the “Evolution” chapter in the standard textbook. I think you’d become one of those memorable teachers, the ones that set students on fire for science and connect it with life.

    The combination of someone who’s both a good teacher and a good researcher is a high standard. I hope that’s what they’re sensible enough to look for instead of just for points on this Google Scholar system, or there are going to be a number of Bio 101 students floundering

    • 25 David 05/01/2012 at 7:40 am

      For UK jobs it’s all about how you will contribute to the RAE from what I hear and blogs like this and the Ask a Biologist site should qualify for the broader impact category that everyone seems to be concerned about. I think the problem is simply that there are a lot of good people chasing a too small number of jobs.

  8. 26 Paul Barrett 05/01/2012 at 5:04 pm

    The best things to worry about in terms of getting a job are publishing high-profile papers and getting grants – those are the primary criteria for any academic job. What your referees say about your team-skills and attitude can count for a lot too. Yes metrics are used for promotions, but that’s after the initial hurdle of getting the job, which is based on a much broader set of criteria.

  9. 27 John R Hutchinson 06/01/2012 at 10:45 am

    I agree with Paul B’s comments. The reliance on metrics may be higher in departments where no one has a clue who you are or what paleobiology is, e.g. Podunk Technical College of the Hillbilly Nation, but at a reasonably esteemed university department that is not the case; committees badly want to hire a great person and so if they don’t know them well (and often even if they do) then they do tons of homework on them– googling, checking webpages, going to conferences and getting the scoop on who’s the best or just calling up experts. If they don’t know them well that may be a bad sign in the first place.

    And then for many jobs the candidates are all quite well known to at least 1 person on the committee, and so their experience with the person will weigh heavily, including reputation, quality/intellectual cohesion of work, experience getting funding, vision for the future, Once you have 2, 3 or more people involved in the search that know the person fairly well enough to judge the totality of their merit, then it can become a quite rigorous process.

    No one pretends it’s perfect, there is always an element of luck involved (particularly in what kind of person is needed for the job- i.e. who fits in best with others in the dept/museum), there is always subjectivity– because there are humans involved, not robots looking blindly at metrics.

    Certainly, it’s very tough times right now globally, so it is a hard time to get a job and there are plenty of good people out there unemployed or nearing that state. The postdoc-to-faculty transition is really hard and stressful on individuals and relationships, but generally good scientists get hired eventually and will continue to be. The best thing one can do is keep plugging on publishing the best science they can and building a well-rounded CV as well as a sound reputation as a person you’d want to spend your career working alongside.

    • 28 David Hone 06/01/2012 at 11:54 am

      Hi John,

      Again, all of this makes sense and what you and Paul and Mike and others have said is, I hope, normal and standard practice. But again, I’ve heard enough behind-the-scenes horror stories and had enough hints from people in authority in good institutes to tell me that my concerns are not exactly unfounded. Yeah, it’s probably not even an issue 95% of the time, but as we all note, when there are only a handful of job available a year, you don’t want to be caught out on the wrong side of that lingering 5% or even 1%.

      As noted there are things beyond job hunting, in a discussion of this issue in another forum someone said that they had had an e-mail about academic promotions that specifically included wanting to know the citations of every paper published and H indices etc. which I would argue suggests that in some places there is an obvious incentive for people to try and jack their indices up.

      On last thing, I’m sure departments do do tons of homework on people once they have something approximating a shortlist, but again, I can’t imagine they do it for each of several hundred applicants. Someone must be whittling down 300 people to 30 people, and even there I strongly suspect that not every CV, personal statement and letters of reference are read in detail. That’s where I worry that people might well be tempted to look at something this quick and dirty to skim off the bottom 50% or so. Now admittedly this is unlikely to filter out good people (if you have a half decent number of papers and a few citations you should get up that far), but it might.

      I’m quite happy to concede (given all the comments coming in from all over the place) that my original post is in an overly worried place as it were. But equally I still think there is cause for *some* worry given other discussions I have had. At the very least, lots of other people seem to be worried about it, so whether or not we should worry about it, it’s caused concern and people who’ve not had (or read) this kind of discussion might still be very worried and trying to manipulate things or even being advised to (as I was once).

  10. 29 Mike Taylor 06/01/2012 at 12:00 pm

    A simple observation: it seems that in this comment thread, the people who have good jobs at good institutions think there is no problem with the way hiring is done, while those currently without jobs think there is a problem. Hardly surprising on either count, I suppose!

  11. 30 John R Hutchinson 06/01/2012 at 3:47 pm

    Cute simplistic summary, Mike, but not a fair summary of what’s being said. “No one pretends it’s perfect” hardly is “no problem with the way hiring is done.” The harsh reality is that with humans involved there is no 100% fairness and objectivity and never will be.

    I don’t think many people understand how hiring is done– it’s hard to understand until one has sat in a job search committee and seen how it actually works, warts and all.

    • 31 Mike Taylor 06/01/2012 at 3:49 pm

      I’m sure all that’s true, John. Of course, you’re right, no-one’s said it’s perfect. Nevertheless, there is a big disconnect between the perceptions of those with and those without jobs. (Having no horse in the race at all, of course, I am perfectly positioned to see all with perfect objectivity :-))

      • 32 John R Hutchinson 06/01/2012 at 5:53 pm

        This is where I should clarify that once I did not have a job, Mike. All “with jobs” people have gone through the horrifically stressful period of finding an academic job. We all worried. We all thought it was fraught with peril, uncertainty, unfairness, and the full range of human pathos. It is! I was employed since age 12 (seriously!) and the periods of unemployment after my PhD were terribly stressful for many reasons.

        So frankly, I think you are full of it here, Mike. You are making up perceptions that seem to fit some preconception of yours that does not fit reality; not sure if it is “stick it to The Man and cheer the underdog” gimmicry, but it’s not constructive and smacks more of a straw man argument than one based on facts and experience.

        Yes, I’m lucky, I do have a job now. A great one. I feel damn lucky for that. A lot of blood, sweat and tears (literally, all 3) went into all that. But those without a job right now will get one too, eventually, tough as the market is right now, if they stick to their guns and focus on being the best scientists they can be. I have many people on my team that are worried about jobs too, so like many researchers I am far from out of touch with the “plight of the common researcher” as some seem to be implying here. I’ve lived it and continue to live it through my students and colleagues. I am deeply sympathetic to that plight because of this ongoing personal experience with it. But I am also a man of evidence and reason.

        I don’t see an argument of substance being made in most of the comments here; the situation with the PhD-to-faculty transition is complex and difficult and not easily summarized in snarky snippets or tweets, or “old people don’t get it” nonsense. The point of the blog seems to have been that paper metrics aren’t great (duh; that’s why real job searches rely on vastly more information) and has drifted from there.

  12. 33 Andy Farke 06/01/2012 at 7:50 pm

    This thread speaks to a much, much larger problem in science right now–far more Ph.D. students are being produced than there are jobs at major research universities, and there is a prevailing attitude that only these jobs are worthwhile (a perception held by many people on both sides of the employed/unemployed divide). I think this is the much more serious and real disconnect, rather than employed/unemployed. (to be very clear, I am not attributing this attitude to anyone on this thread – John, Mike, Dave, Paul, etc., – I’m just pointing out a general attitude in science)

    Case study – I was at the SVP open executive meeting this year, and there was hand-wringing from the floor as to how all of the students were going to get university jobs, and who is going to fund their research. The implicit assumptions were that 1) research is only happening at big places like Michigan, UC Berkeley, Harvard, etc.; and 2) those are the only jobs worth pursuing. Of course, the solution too all of this is to get more NSF money for big research universities. (I am oversimplifying a little here, but not much – I asked one person where small museums like mine would benefit, and the answer was basically that maybe one of the big institutions might trickle down their largesse – and we all know the utility of “trickle-down” economics!)

    In any case, I stood up at the meeting and pointed out that there _is_ life outside an R1 (e.g., museums of all shapes and sizes – I have a wonderfully satisfying position at one!). This doesn’t mean the jobs are easy to get or plentiful, but there are far more viable, satisfying positions out there than often acknowledged. Still, the “R1 or bust” attitude that is explicitly or implicitly promoted is not realistic or healthy. When I took my current job (which is a permanent, endowed curator position that I like very much and affords lots of research support and freedom), more than one colleague told me that, “Oh, you can do better than that, but it’s not a bad place to stay until a [real/R1/university] job opens up.” Most of them now get that I’m here to stay (for reasons of high job satisfaction, stability, and family), but it took some convincing!

  13. 34 Tor Bertin 07/01/2012 at 7:17 am

    Another issue that I’ve found with citation metrics is that the generality of the papers/books written can greatly influence citation numbers. One of my professors wrote ‘The Book’ on the use radio telemetry in the mid 1990s, and it’s since garnered over 2,200 citations. His next highest is much more typical for a decently cited paper in the field, at ~200. Two candidates with equally ‘quality’ work may have quite divergent citations numbers if one candidate’s research is more broadly applicable than another’s. Of course, the search committee might find this a bonus, but it’s something to consider all the same.

  14. 37 mAnasa-taraMgiNI 07/01/2012 at 8:00 am

    You raise a point that so many scientists privately converse about but few articulate in public. While the best candidates still continue top the metric and get jobs, there is need for compassion for the scientists who do not make the cut and a provision for alternative tracks for them to remain in active science. Perhaps, the whole pyramidal structure of the biological endeavor with professor Bigs, with big grants, spawning many post-docs/PhDs, who are then expected to become professors themselves has reached its limits. Perhaps an alternative to the PhD degree and the standard career path in biology needs to be considered. Many of us with multiple generations of pupils know that not all of them have in them to become professors. Yet some of them are good scientists who could be extremely productive under a senior scientist. But the current system sort of pushes them into the biology department faculty market due to the absence of alternatives. Some of these might be more productive working under the patronage of more established professors without needing to found their own little kingdoms.

    • 38 Mike Taylor 07/01/2012 at 11:44 am

      There’s a simple mathematical analysis of this situation. For the world to continue with the same number of VP labs, each headed by a “senior scientist”, that senior scientist needs to spawn just one more lab leader in his entire career. Unless the whole market grows (and of course right now it’s shrinking), simple arithmetic says that the other n-1 Ph.Ds emerging from the lab over the course of that senior scientist’s entire career can’t possibly get similar jobs. That is truly frightening.

      • 39 David 07/01/2012 at 12:38 pm

        Yes, and some others will get jobs at teaching oriented universities or universities without PhD programs, museums, etc. In the US system at least PhD programs are concentrated in the R1 universities, but the non-R1 universities etc. need to be staffed. Of course, the amount of science going on in the world is still increasing as countries like China and India are developing. A lot of the PhD students being trained in the developed countries are going back to their home countries in Asia etc.

  15. 40 John R Hutchinson 07/01/2012 at 1:46 pm

    Sorry to be harsh on Mike T in my reply. I was feeling grouchy and Mike was making some oversimplifications that I did not find helpful to those in the job hunt or a fair characterization of senior researchers’ attitudes and knowledge. I’ve seen a lot of people “fail” (leave academia, at some stage) so I know very well what the state of affairs is like- it’s grimmer than ever, as everyone above has been noting. And it’s my job to know that, too.

    I was especially objecting to an implication that senior researchers do not know what it’s like for junior researchers right now. I regularly get students/staff/postdocs crying on my shoulder about the job market, so I see the worried aspect that most commenters are focusing on and am reminded of the strife I had too. We senior researchers see the situation from essentially all sides; the decreased research funding, the dwindling of jobs available, the competitive job interview process, etc. And it is our job to understand that situation and use that understanding to maximize the success of those that pass through our programme- or steer those that are not ideally matched to a certain career path into alternative paths.

    I’ve noticed an interesting difference in attitudes toward doing a PhD in the UK vs USA- many people will do a PhD in the UK just for a fun education or to bolster their chances at getting a non-academic job. Much more so, in my experience, than in the USA. Probably 1/3 or so of PhDs don’t aim to go the “normal” path, in my experience here (mine might be a bit unusual as it is not a VP programme). I am not sure how that is changing as most PhD programmes shift from 3 to 4 yrs duration, but assume it will become more like the USA.

    As for the increasing ratio of PhDs to jobs, that is probably going down. At some point, it is inevitable. It would be interesting to see some broad stats. The number of PhDs available has plummeted in the UK- for example, we used to (circa 2009) get 20 PhD posts/year, and now that has dropped to 3-6! So now many academics really have to struggle to even get 1 PhD student/year or so. And external funding to augment PhD numbers has almost dried up. So right now I generally advise students not to go for a PhD unless they are of the highest caliber; it is not a good time to do it, too competitive, too few jobs.

    It’s even worse for students in the USA that want to go abroad for a PhD, which is a rich experience to have. I have several top notch students interested in doing PhDs in the UK but of course they don’t want to pay the £19k/year fees (x 4 yrs = £78k, plus living expenses!), and precious little funding exists to cover the fees; ditto even for excellent Chinese students.

    But there is still hope and a matter of time before the economy turns around… and only so much any current junior researcher can control. Do great science and be a respected colleague, and try to check you’re on the right path for you.

  16. 41 Mike Taylor 07/01/2012 at 2:14 pm

    Sorry to be harsh on Mike T in my reply. I was feeling grouchy and Mike was making some oversimplifications that I did not find helpful to those in the job hunt or a fair characterization of senior researchers’ attitudes and knowledge.

    No worries, John. Thanks for clarifying, and apologies for any oversimplifications.

  1. 1 Catchup « Dave Hone's Archosaur Musings Trackback on 09/01/2012 at 11:02 pm
  2. 2 Google Scholar citations « Dave Hone's Archosaur Musings Trackback on 04/05/2012 at 10:44 am
Comments are currently closed.

@Dave_Hone on Twitter

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 548 other followers

%d bloggers like this: