Quelle surprise: academics gaming the system sank the era journal rankings

So here is the problem: research quality is a nebulous concept and it takes many years to work out whether someone’s output has actually high quality or not. This is especially hard for

non-insiders. The Government is a non-insider that funds academic research in Australia and so would like to know what it is getting for that money. So here is a typical solution: use the

quality of research publications as a proxy for likely high quality in the future and use that to measure performance. Academics will recognise that solution as it is precisely what they try

to employ internally. Sometimes it is transparent, sometimes it is subjective. But it is always done. So what does the Australian Government do in response to all of this: try and mimic the

academic solution to the performance measurement issue but scale it up. Alas, the only way to scale it up and to be transparent is to invest and use an objective measure. That is where the

Excellence in Research for Australia (or ERA) journal/conference ranking came in. It was not original. The British government had already embarked on that. It was also not new. Australia had

been down such paths before. What is the problem with such objective measures of quality? First, there is a major cost — a huge cost — in coming up with the measure. The reason for the cost

is that off-the-shelf measures rarely exist. Actually, that isn’t true. Every discipline had an off-the-shelf measure that managed to get 90% of what any ranking would eventually have in

it. But there are always problems. The big one is ranking Australian-based research. Another is inter-disciplinary research. Then there are newer journals and fields. All of these problems

bite internally in academia. And so what happens? A big fight ensues. That is what happened with the ERA. Huge fight, lots of rent seeking and mostly a waste of time because the choices

impacted on a few people in a big way and everyone else not at all. Second, once you have an objective measure, people will try to improve their own ranking. Remember that is the point. The

problem with research is that the best way to improve your own ranking is not to produce better research (that takes time) but to buy it in (getting good researchers into your institution;

see here for an example) or direct publications (to the journals that rank highly but are actually not of great quality). The first sort of gaming is great for high output academics whose

salaries went up. The second doesn’t really have any consequence. Actually, both reactions are distributional but involve “costs of gaming.” With these schemes you get what you pay for (in

this case, performance directed to improving ERA ranking scores) but at a cost (not getting any real increase in research quality). Yesterday, Minister Carr announced that the ERA was

effectively dead). The journal rankings will be replaced by a new ranking based on “frequency of publication.” This is so useless a measure I read it as giving up. Of course, I could be

wrong and the Government may reward Universities based on it, in which case it is one of the most insane measures ever put forward. Anyhow, here is the cited reason for the change: “There is

clear and consistent evidence that the rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor

understanding of the actual role of the rankings. One common example was the setting of targets for publication in A and A* journals by institutional research managers.” To which I say: what

did you expect? Every academic told the Minister this would happen — it is just basic economics. Secondly, it is exactly what happened in Britain. So we had evidence of the theory. Now the

Government is giving the impression that they are still going to do something. They have opted for a lack of transparency but it seems to me they are packing up and going home. To be clear,

these short-term issues aside, that means no measurement of research performance and hence, no reward for it. This would be fine if we had real competition for funds from students in our

Universities but we don’t. It is unclear how pretending issues don’t exist will improve matters. But we should be more angry about this. Many academics’ comments on hearing about the demise

of the ERA is good riddance. Why? Because they bore the costs of fighting about the measure and then the gaming. But those costs have been borne. I personally bore a ton of them and so did

so many others. A complete waste of time. And for what? Nothing. Just to prove to the Government what we all could have predicted four years ago! It is an outrage. The Minister had the

information. Carr is the same person who pushed the decision in the first place. Has he admitted he was wrong? No. He pretends that it is news to him that academics were strategic. Well, let

me tell you, someone who doesn’t know that academics are strategic is not equipped to manage the University sector. This is an embarrassment to the “Education Government.” _This also

appeared on the Core Economics blog and has been reproduced with permission from the author._