How to reference the REF

Or how to go beyond a league table position in evaluating a UK university department

Every five years or so all UK university departments get their research assessed in one gigantic peer review exercise, which is now called the REF. Each discipline is assessed separately, and scores can be used to compile a league table. The exercise has direct financial implications: the better the research, the more money universities get from the government. But if you know what academics are like, you will not be surprised to learn that those in the UK obsess about this exercise and its results to a far greater extent than the money involved would justify. The results of the latest exercise have just been published, and turned into league tables by Times Higher Education (THE) here. 

You could say that the REF now provides the same sort of incentive system for UK universities as profit does for a firm. In some cases academics whose research is below their departmental average are put under pressure to leave by one means or another, and most academics feel acutely the pressure to improve on how their own output will be assessed by this exercise. In contrast, poor performance in teaching or administration is not nearly such a serious issue.

Many academics complain bitterly about the indignity of all this. An alternative system would be one where getting tenure was the last performance hurdle an academic had to pass, and from then on they were free to do what they liked. Research money could all be allocated on a project by project basis. I personally doubt that would be a better system from society’s point of view, and I do find it annoying how academics can complain so much about pressures that are taken for granted elsewhere.

It would be a mistake, however, to think that the position in some REF league table told you all you needed to know to evaluate the quality of research in a department. The REF releases a wealth of data, and going beyond the headline number (usually the GPA score) can be informative. In the latest exercise departments were evaluated under three headings: outputs, environment and impact. Details about what is involved for each category can be found here.

Outputs, which has the highest weight in the total (65%), looks at the quality of the four best recent publications of each submitted member of staff. The key word to note here is ‘submitted’. A department/university can choose not to submit all its staff to the REF, and by not submitting staff that a department/university considers are well below average it can raise its GPA score (if it gets its assessments right). So to the extent that staff are not submitted, the GPA will overestimate the average quality of the research done in that department. As I said, league tables normally just look at the GPA score [1].

To some it may seem strange that this is allowed, but there are arguments to justify it. Departments do pay a significant financial penalty for leaving staff out - they only get money for submitted staff. To get a guide to the total amount of quality adjusted research done in a department, simply multiply the GPA score by the number of people submitted (called ‘power’ by THE).

The decision about whether not to submit a member of staff is an agonising one [2] that involves many difficult trade-offs. To the individual not being submitted it is a nasty slap in the face. For the department, the perceived benefits in getting a higher position in GPA based league tables may outweigh the financial cost of not submitting staff members. Decisions on this front do vary from university to university, and from department to department: in economics, compare the third and fourth columns of the table below.

Although it only counts for 15% of the total GPA score, the ‘environment’ heading may be of particular interest to potential PhD students. It is based on a number of different criteria, including the number of PhDs, the support provided for research, and research income from outside grants. Only three economics departments had all elements of environment judged to be of the highest (4*) quality this time: UCL, LSE and Oxford.

Impact is a new category, accounting for 20% of the total. It is based on case studies where research has engaged with public, private and third sector organisations, or directly with the public. For example, one of Oxford’s case studies for economics was my own work on fiscal councils. A quick look at the results suggests that this new element has had a significant influence on the overall results. In economics, for example, the only department where all the submitted case studies were judged to be of the highest quality was Bristol. So while Bristol only came 12th= on published outputs, a strong impact and environment score lifted them to 6th in the overall ranking.

As with any evaluation system, there are difficult judgements to make on the details, and these can lead to possibilities to ‘play the system’. Chris Bertram focuses on one particular issue at Crooked Timber. Each iteration of the assessment exercise attempts to change the details of the rules to avoid this, only to allow some new possibility to exploit the system. Partly as a result, after each exercise many academics feel that there must be a better and less time consuming way to judge the quality of research produced by individual academics or departments, but perhaps the fact that we keep returning to the same basic procedure suggests otherwise.

REF 2014 results: economics and econometrics

University	GPA Score	No. of staff submitted	Eligible staff	Power	% 4* Outputs	% 4* Environ.	% 4* Impact
UCL	3.78	37	45	139	70	100	92
LSE	3.55	51	56	182	56	100	87
Oxford	3.44	84	97	289	43	100	64
Cambridge	3.42	27	38	92	55	13	50
Warwick	3.41	42	52	142	43	38	60
Bristol	3.32	19	25	62	22	63	100
Essex	3.25	33	40	108	29	63	20
Edinburgh	3.14	18	28	55	31	50	13
Royal Holloway	3.11	14	23	45	35	0	60
Nottingham	3.05	35	46	107	20	13	18
UEA	3.04	14	22	43	20	0	20
Surrey	3.01	21	25	62	27	13	0
Queen Mary	2.98	24	31	73	20	13	13
York	2.93	28	46	82	14	0	40
St. Andrews	2.92	21	31	60	24	0	0
Manchester	2.89	33	45	96	11	13	40
Glasgow	2.86	24	30	68	18	0	0
Sussex	2.84	17	24	49	15	0	37
Exeter	2.78	25	31	68	13	25	13
Birmingham	2.78	24	27	67	8	0	27
Southampton	2.70	22	28	59	22	0	10
Birkbeck	2.60	25	32	65	10	0	0
Leicester	2.59	22	29	58	19	0	0
Sheffield	2.58	15	26	38	8	0	40
Aberdeen	2.48	19	26	48	5	0	0
City	2.44	14	26	33	17	0	20
Kent	2.32	22	26	51	3	0	13
Brunel	2.20	26	29	58	2	0	0

Note that many economics departments are assessed under Business and Management, and are not included here. Sources: columns 2,3 and 5: Times Higher Education, column 4: HESA, columns 6-8, REF.

[1] THE publishes an alternative university wide ranking (aggregated across departments) that multiplies the GPA by the proportion of staff submitted, but that implicitly gives the research of non-submitted staff a score of zero, which is likely to be too extreme. It is better to simply note either the power score, or the proportion of staff submitted. Approximate information on the number of staff eligible for submission by department can be found here.

[2] This is based on my own experience at my previous university, where I was research director for the school. Thankfully I have played no role in these decisions at Oxford!

Monetary Impotence in context

Mainly for macroeconomists

There are a significant group of people who think that monetary policy must be the right answer even in a liquidity trap because of the centrality of money in macroeconomics, and because of ‘basic’ ideas like money neutrality. Call them market monetarists if you like. They dislike fiscal stimulus because - in their view - it just has to be second best, or a fudge, compared to monetary policy. Their view is not ideological, but essentially based on macro theory. Now it may not be very relevant to the real world, but for many holding the theoretical high ground is important, because it colours their view of the real world.

That is the group that Paul Krugman has been arguing with recently, and why the point he made in his post yesterday is so critical. It is set in an idealised two period world where Ricardian Equivalence holds, but that is entirely appropriate for the task in hand. If people believe something because of (in their view) basic theory, and you think they are wrong in terms of basic theory, then that is the level on which to argue.

The argument is that in a liquidity trap, when prices are sticky, temporarily expanding the money supply - even if it involves helicopter money (i.e. money financed tax cuts) - will not do anything to get you out of the trap. Another, and more modern, way of saying the money supply increase is temporary is saying that the inflation target is unchanged, so the long run price level is unchanged. (Long run money neutrality does hold in this world.) I will not go through Paul’s argument in detail - I have gone through the same logic before. The basic point is that the temporary increase in money is saved, not spent, because agents know it is temporary. Short run money neutrality does not hold, and not because prices are sticky, but because of Ricardian Equivalence.

It is exactly the same reason why the Pigou effect is no longer discussed. Ricardian Equivalence killed the Pigou effect as a fundamental theoretical idea. If the inflation target is unchanged, when prices fall today the future price level must fall pari passu, reducing the future nominal stock of money. There is no wealth effect. As I noted here, even allowing money to be special in not being redeemable does not get the Pigou effect back, because with irredeemable money any wealth effect comes from the long run stock of money.

Money is not a hot potato in this world. The potato has gone cold because of the liquidity trap, and the money is happily saved to pay the future tax increases that will be required to keep the long run money stock (and price level) constant. 

While in this largely frictionless world money is impotent, additional government spending is a foolproof way of expanding demand. So is raising the long run price level, which means at some point raising the inflation target.

All I really wanted to do in this post was make an observation. The theoretical point that Paul makes depends crucially on thinking in an intertemporal manner, which gives you Ricardian Equivalence. Just as price rigidity kills short run monetary neutrality, so does Ricardian Equivalence in an inflation targeting liquidity trap world. So here is modern microfounded macroeconomic theory providing support to increasing government spending rather than monetary policy in a liquidity trap. Modern theory is not inherently anti-Keynesian. .  

Why these updated fiscal rules are a backward step

The coalition government in the UK has just updated the fiscal rules that govern how it decides budgetary policy. I think it is helpful to distinguish between the form of those rules and the particular numbers attached to them. I and many others have already discussed the numbers on various occasions. My bottom line: it is stupid to commit to further significant fiscal contraction (‘austerity’) when interest rates are still at or close to their lower bound. It means we become more vulnerable to adverse shocks to demand. An academic quibble? No, it is what happened in 2010. Unlike 2010, however, no one with any sense thinks we must have austerity to avoid being punished by the markets.

I want to talk in this post about the form of the rules. Here the change compared to 2010 appears minor. The primary mandate now has a 3 rather than 5 year rolling horizon, and the date for the secondary target of a falling debt to GDP ratio has just been shifted ahead to 2016/7.

In my paper with Jonathan Portes (here or here), we argue that fiscal rules can have two goals. They can try to mimic optimal fiscal policy, or they can be effective at restraining a government that is subject to deficit bias (basically spending too much and taxing too little). The main point about optimal fiscal policy is that government debt and deficits should be shock absorbers, and spending and taxes should be adjusted slowly to meet any debt target.

In this context the coalition’s secondary target remains a bit of nonsense. Getting the debt to GDP ratio to fall at some stage is a good idea, but having a target for a specific year is silly. It is not optimal because if some shock hits the economy before 2016/7 which means debt tends to rise relative to GDP, it is crazy to try and counteract that to meet the target in such a short space of time. It is not effective because it can be gamed by the government fiddling the timing of expenditures.

In contrast, Jonathan and I argue that the form of the original primary mandate makes a lot more sense, as long as interest rates are not at their zero lower bound. Having a five year rolling target for the deficit allows fiscal policy plenty of time to adjust to shocks. We saw this in action over the last few years, as the Chancellor was able to reduce the pace of fiscal consolidation from 2012 when the economy failed to recover as quickly as he had hoped. Changing this mandate from five to three years gives any Chancellor less time to adjust, which is why it is a backward step.

In talking about the change, the Treasury says it “reflects the progress that has been achieved in tackling the deficit, which means that the mandate can be safely shortened to create a tighter constraint on future fiscal policy choices.“ This is one of those lovely phrases that sounds plausible until you think about it. The whole point of having a rolling target is that it gives you time to adjust to shocks, when too rapid an adjustment would be costly. Has the expected size of shocks got smaller, so we can safely create “a tighter constraint on future fiscal policy choices“? I don’t think so.

Of course I know, and indeed everyone knows, that the reason for this change has nothing to do with economics and everything to do with politics. Whether it is clever politics or not I will leave to others to debate. In addition on this occasion the numbers in the rule, and the risk he is taking because we are still at the zero lower bound for interest rates, matter more than this change from a 5 to 3 year rolling target. But it is still a shame we are going backwards. I have just finished an article, due to appear in February, on the coalition’s macroeconomic policy, and to set against the obvious mistake I was able to count two successful innovations: the creation of the OBR and the 5 year rolling fiscal mandate. Now we are left with just one.    

Robert Peston, Mr Market and me

Paul Krugman picked up on my post commenting on the views of Robert Peston’s pals in the bond market, and Robert Peston has now responded. To characterise this as a debate would I think be wrong. Being the excellent journalist that he is, Peston is reporting the views put to him, rather than taking a particular side. In his response, he does a reasonable job of presenting Paul and my views, and does not argue too hard against them. I have two significant clarifications, and one key point.

The first clarification is that Peston still downplays the importance of having your own central bank and borrowing in your own currency. He says that is ‘partly’ why the UK has avoided the fate of the Eurozone PIIGS. I would argue that is entirely why the UK (and the US, and Japan) have done so. Both Paul and I would argue that fiscal policy should have been expansionary and not contractionary in 2010 [1], and in my view this would have had no detectable influence on any risk premium. The recent IMF self-evaluation that I discussed here also argues at a global level that the switch to fiscal contraction ('austerity') in 2010 was a mistake caused by a misreading of the Eurozone crisis.

Second, I think what he writes at the end could be misleading. He says: “Now to be clear, there are economists who attack the idea that the deficit and debt can be cut by growing the economy with all the vehemence of Krugman's and Wren-Lewis's lampooning of me and Mr Market.” The minor point is that I did not lampoon either him or the market, but simply what his pals thought about how the market worked. But more seriously, the argument that debt could eventually be lower as a result of growing the economy through fiscal expansion - advanced for example by DeLong and Summers - is different from the point that Paul and I make about the irrelevance of default risk for the UK or US and the consequent foolishness of trying to cut the deficit when interest rates are at their zero lower bound.

Those clarifications apart, the main thing I wanted to say was this. Of course he was just quoting what was said to him by his pals in the bond market, and I would not want him to suppress those views, although I did get some feedback from others in the markets that their own view would have been rather different to his pals. My complaint was that he did not talk to some academic macroeconomists as well, and I explain here why their opinion on issues like this may be at least as useful as some players in the market. If he ever wants my opinion, he just needs to email!

 

[1] See for example Paul’s recent book, and Jonathan Portes and my recent paper.

Deficits, Mediamacro and Popular Opinion

The level of the government’s budget deficit is not the most important macroeconomic problem facing countries today. In the UK, for example, the fact that labour productivity has been stagnant since the recession is potentially far more important. If we fail to regain this lost productivity growth in the next few years, the average UK citizen will be substantially poorer as a result, in terms of consumption of both private and public goods and services. If the budget deficit rises by a few tens of billions, leading government debt to eventually rise or fall by even ten or twenty percentage points of GDP, the consequences will be negligible by comparison. To misquote George Osborne, the world will not fall in.

Yet mediamacro presents a very different picture. When Labour leader Ed Miliband forgot to mention the deficit in his party conference speech, the media could talk of almost nothing else but this ‘huge gaffe’. When Prime Minister David Cameron said nothing about the productivity slowdown in his party conference speech, no one in the media bothered to even mention this.

One of the fall back positions of the media on issues like this is that they are only reflecting popular opinion. In the case of the deficit it would hardly be surprising if this were true. If people are not told otherwise, they are bound to think about the government’s budget as they think about their personal budget. The Eurozone crisis was in the news constantly for two years, and the number of people explaining that the Eurozone was special because of the ECB was dwarfed by those who suggested it could happen here. A popular fear of market reaction is also not surprising as we are still suffering from the impact of the financial crisis.

But is mediamacro reflecting public opinion? Here is a question that YouGov recently asked in a poll for the Sunday Times (full poll results here, HT Duncan Scott).

Thinking about how the next government handles the issue of Britain's deficit, which of the following best reflects your view?

A.    The next government should prioritise reducing the deficit, mainly through making cuts to spending on public services

B.    The next government should prioritise reducing the deficit, mainly through increasing the level of taxation

C.   The next government should not prioritise reducing the deficit, and should spend more on public services or cutting taxes to try and promote growth instead

(A) could be described as the Conservative view, (B) as maybe the Labour or Liberal Democrat view, and (C) is the position taken by some nutty academics. Of those that chose one of these three options, 27% went for option (A), 25% for option (B) and 48% for option (C).

So around half said not only that cutting the deficit should not be a priority, but also agreed that fiscal policy should be used to promote growth. For these people, none of the major political parties represent their position. Of course Labour’s positioning could still be shrewd politics. It may be correct in thinking that this 48% will still prefer to vote Labour because they intend to reduce the deficit by (a lot) less than the other parties, but by pledging to make cutting the deficit its first priority it may attract some of (B), and also get mediamacro off their back.

The other implication is that maybe forgetting to mention the deficit is not, for many people, quite the gaffe that mediamacro thinks it is. Of course a large part of the UK media decided it was a major gaffe for purely political reasons, as another stick to beat Ed Miliband with. If the non-partisan part of the media went along with this because they thought reducing the deficit should be the top macroeconomic priority, they are simply wrong, as most economists will tell them. If they went along with this because they thought they were just reflecting public opinion, then maybe they should think again. If mediamacro continue to obsess about the deficit, the only conclusion to draw will be that we have a media consensus driven by a partisan press.