Winner of the New Statesman SPERI Prize in Political Economy 2016

Saturday, 19 October 2013

The decline of evidence based policy

There is a brief description of Cartwright and Hardie’s book ‘Evidence-based policy: a practical guide to doing it better’ on Amazon. It starts: “Over the last twenty or so years, it has become standard to require policy makers to base their recommendations on evidence. That is now uncontroversial to the point of triviality--of course, policy should be based on the facts.”

My immediate reaction is to say ‘unless that policy involves the macroeconomics of fiscal policy’, but once you see one area where evidence is ignored, you begin to see many more. Here are just two that I have read about over the last few weeks, courtesy of the UK government. They can better be described as ‘emotion-based policy’, or ‘election-based policy’, and I fear they may now be the rule rather than the exception.

The first involves what is generally known as the ‘bedroom tax’. The policy reduces the housing benefit payable to tenants deemed to be under-occupying their homes. This policy has undoubtedly caused considerable hardship to many of those affected, but it also saves public money. So crucial in any assessment of the desirability of the policy is how much money it saves. Getting such an estimate is complex, because it will depend on a lot of factors, like whether individuals move in response to losing benefit, where they move to, and so on. The UK’s Department of Work and Pensions has a model that calculated savings of £480m in 2013/14, an estimated it published in June 2012.

So the first thing to do in trying to assess the realism of the £480m figure is to look at the model. We cannot do that, as it is not published. However, Rebecca Tunstall of the Centre for Housing Policy at the University of York has managed to obtain some spreadsheets, using Freedom of Information requests. In a report, she looks at some of the assumptions behind the department’s calculations. Many look very questionable, and as Alex Marsh observes here, there seems to be a pattern - the questionable assumptions tend to overestimate the savings involved. The calculations also ignore some of the other financial costs and consequences that an overall assessment of the policy should take into account. Alex Marsh suspects that allowing for these, and taking into account the evidence we are now getting about how people are responding, the savings created by the policy could disappear completely.

If policy was evidence-based, then either the government would be disputing in detail Professor Tunstall’s analysis, or rethinking the policy. Neither are likely to happen for one simple reason. The policy was never evidence based. It was instead inspired by the tabloid led attack on welfare recipients, and the idea that it was unjust that ‘hard working taxpayers’ should fund a spare room for benefit recipients. Here for example is Stephen Glover writing in the Daily Mail: “The notion that many families not on welfare don’t have the luxury of a spare room, and may have to have one or two people in every bedroom, is foreign to the head-in-the-clouds types that proliferate at the BBC.” Now that feeling of unfairness is real enough, but it is not based on evidence either about what impact the policy might have, or whether it will actually save any money.

The second example is the recent ‘clampdown’ announced by the UK Home Secretary on ‘health tourism’. Among the measures included in the government's new immigration bill is a £200 charge on all temporary migrants for using the NHS and a requirement for GPs to check the migration status of new patients. Both policies are aimed at preventing migrants travelling to the UK to seek free healthcare. Conservatives has been particularly keen to attack ‘health tourists’ from other EU countries, thereby clocking two election sensitive issues among potential UKIP voters: migration and the EU.

So what evidence does the UK government have on the scale of this EU health tourism? The answer is that it has none. A classified document had the following very telling phrase: "we consider that these questions place too much emphasis on quantitative evidence". So considerable extra costs are going to be imposed on GPs and other parts of the NHS to help deal with a problem which may be trivial.

However the European Commission has succeeded in getting some evidence on ‘benefit tourism’, which it recently published. As Jonathan Portes reports, approximately 4% of those claiming unemployment benefit (job seekers allowance) in the UK are EU migrants, although they represent well over 5% of those in work. More generally, as the Commission says: “Mobile EU citizens are less likely to receive disability and unemployment benefits in most countries studied.” For much the same reason, the proportional demands made by EU migrants on the UK health service are likely to be less than the native population. They may also choose to avoid the NHS, for a variety of reasons.

Which you might think is a bit embarrassing for a government that is about to create a whole raft of additional bureaucratic costs to deal with this ‘problem’. Well not embarrassing if you read the Daily Telegraph or Daily Mail. Their reading of the Commission report was that it showed “more than 600,000 unemployed European Union migrants are living in Britain at a cost of £1.5 billion to the NHS alone”. Only one slight problem - the true number is 38,000. So how did the Daily Telegraph manage to inflate the true number by a factor of 15? Because they ‘confused’ unemployed with non-employed, where the latter include students (of which there are many), retired people, carers and others with family responsibilities. As Richard Exell says: “Our government may be chumps when it comes to evidence-based policy making, but they can always rely on world-class distortion to see them through.” The report remains on the Telegraph’s website, uncorrected.

I’ve included this last paragraph to make a very simple point. Some commentators tend to argue that we make too much of what happens in the media (see Chris Dillow here for example). I think this is plain wrong. Information is vital. People make judgements based on the information they receive. As one student said to me this week, they knew that the Obama stimulus package had had little impact on the US economy, because they had read this in the FT and the Economist. (For better analysis, follow the links from here.) If people have distorted information (or if those bringing a bit of reality are vilified), politicians have no incentive to base policy on actual evidence. The decline in evidence-based policy, and the declining importance of facts for much of the media, are not unrelated.  

Postscript, 20th October

If a reputable newspaper makes a mistake, it acknowledges that mistake. Another kind of newspaper, it seems, can try shouting down those who pointed out the mistake. That was what the Daily Mail did over its attempt to slur Ralph Miliband as the ‘man who hated Britain’. So today we find the Telegraph doing the same over its reporting that “more than 600,000 unemployed European Union migrants are living in Britain”. Rather than admit that it got that wrong, it instead tries to pretend that it is in fact battling some kind of great conspiracy emanating from Brussels.

The lead article on its website today (19 October) by it chief reporter Robert Mendick could not be a better illustration of the points I made in the last paragraph of my post. The article’s first paragraph says
“The [Commission] study — whose details were first disclosed in The Telegraph — showed that more than 600,000 “non-active” EU migrants were living in the UK at a possible cost to the NHS alone of £1.5 billion a year.”
Here is the opening paragraph by the same reporter in the original (12 October) piece:
“More than 600,000 unemployed European Union migrants are living in Britain at a cost of £1.5 billion to the NHS alone, according to an EU report.”
Virtually identical – except for the little details of replacing unemployed with non-active (the writer puts non-active in quotes, apparently unable to translate this into ordinary language, like students, housewives, retired people), and the insertion of ‘possible’ before cost. No acknowledgement of these changes, but instead an attempt to manufacture another story, the ‘developments’ of which are

(a) some politicians did not like the conclusion of the Commission report. (No, really?)

(b) quotes from an academic at Oxford who says that “There is no problem with the numbers [in the report]. The issue is the interpretation of those numbers.” (Heady stuff!)

(c) the paper has found out that the “independent consultancies who wrote the report were awarded EU contracts worth more than £70 million over six years.” (Ah, the plot thickens.)

(d) a BBC report which attempted to reflect the facts had been accused as being unbalanced by a Conservative minister. (My god, this is serious.)

Then we have: “Evidence of mounting public concern in the EU’s biggest economies over migration emerged in a poll yesterday which showed that the introduction of restrictions on EU migrants’ rights is backed by 83 per cent of Britons..” I wonder where people are getting their information from!

And then there are attempts to discredit opponents. The European commissioner in charge of the department that published the study is quoted in a paragraph that begins: “Mr Andor, a socialist, said: ...”. Later on we have: “It also emerged that one of the main supporters of the report ... was in receipt of more than £600,000 of EU funding for the year ending March 2012 for his think tank.“  Yes, that is Jonathan Portes, director of the National Institute, which has been doing – shock, horror - research funded by the Commission. As Portes points out in the article, it has also been doing research funded by the UK government, but clearly the Commission has managed to exert its evil influence on Mr Portes where the UK government has failed. How can that be? The article helpfully informs us that he was a “former senior economics adviser to the last Labour government “. He was actually a civil servant working under both governments (he has an interesting account of his role advising Norma Lamont in 1992 here), but I guess that is another one of those little details, like the difference between ‘unemployed’ and ‘non-active’.  


  1. If you had listened to Toby Young on the BBC, you would know that the US had a one trillion dollar stimulus, apparently. Not that he was challenged on the figure, which I expect came third-hand from someone who knew someone who knew what Joe Stiglitz had recommended (Krugman went for 1.2 trillion dollar stimulus).

    I have just read Stefan Collini's third brilliant article in recent years on university funding on the London Review of Books (no subscription required). It finishes like this:

    "Future historians, pondering changes in British society from the 1980s onwards, will struggle to account for the following curious fact. Although British business enterprises have an extremely mixed record (frequently posting gigantic losses, mostly failing to match overseas competitors, scarcely benefiting the weaker groups in society), and although such arm’s length public institutions as museums and galleries, the BBC and the universities have by and large a very good record (universally acknowledged creativity, streets ahead of most of their international peers, positive forces for human development and social cohesion), nonetheless over the past three decades politicians have repeatedly attempted to force the second set of institutions to change so that they more closely resemble the first. Some of those historians may even wonder why at the time there was so little concerted protest at this deeply implausible programme. But they will at least record that, alongside its many other achievements, the coalition government took the decisive steps in helping to turn some first-rate universities into third-rate companies. If you still think the time for criticism is over, perhaps you’d better think again."

  2. “My immediate reaction is to say ‘unless that policy involves the macroeconomics of fiscal policy’, but once you see one area where evidence is ignored, you begin to see many more.”

    Quite. IP policy is a good source of examples (e.g. ). If people demand irrational policies, as they often do, the policy makers will oblige.

  3. Politicians need the votes of the public, not experts. So the thing is that the public need to be swayed by evidence.

    No doubt they often don't get the evidence - the media isn't always great, as you say.

    BUT, even if they were better informed, I'd still be pessimistic about evidence based politics. Too much has been found about people's biases - confirmation biases and motivated reasoning. People only give weight to those facts that support their established opinions, and underweight contrary facts. As Simon & Garfunkel sung, "A man only hears what he wants to hear, and disregards the rest."

    For example, the Lee Ross and colleagues study from 1979, where supporters and opponents of the death penalty were given facts which gave support to both positions. The people came away more polarised, not less. The evidence hadn't shaken their previous opinions.

    So I think that human nature ultimately places a limit on rational political discourse (where the issues are often very complex anyway, and cause disagreements among specialists).

  4. I look at my own family in the US (I live in Canada, thank God.) Many of them are far poorer, even destitute, in the wake of the 2007-8 financial collapse, or as a result of the job-linked health care system. Two brothers, in particular, lost crucial health care when serious illness lost them their jobs - one was homeless, though not jobless, for three months ( he's the one who lost 11 years retirement savings from his 401k in 2008) The other survived a serious heart attack while he had insurance through his company, but of course there were other, huge costs that insurance didn't cover. He lost the job and his house and vehicle, and only survived by taking a tiny apartment with a friend, and going back to college, at age 60, under the GI Bill.

    Yet most of my US relatives are deeply skeptical of the ACA or of any government programs set up just for people in their situation. In particular, the younger generation, my brothers children, have bought into a mindset that ensures that they will actively avoid and oppose the very help they so desperately need. Stockholm Syndrome on a grand scale. It is this distortion of the public mind, more than the parasitism of the insurance and financial systems, that most infuriates me. Fraud and extortion you can fight, but if you don't recognize them as such, then what?

  5. I could add the following as supposedly 'evidence-based' policies that aren't:

    Minimum pricing for alcohol
    Plain packaging for cigarettes
    The smoking ban in pubs
    Campaigns against salt and saturated fat
    Proposals for a 'fat tax' and taxes on fizzy drinks
    New York's ban on smoking in parks

    Indeed almost the entirety of public health policy development is so steeped in the ideology of choice control that no of them have spotted that it's just not working.

    Oh there's lots of so-called 'evidence' but it's entirely designed and implemented to support the previously determined policy.

    1. "So called evidence"

      That's priceless

    2. So let me get this right:
      You've published a list of "supposedly 'evidence-based' policies" such as minimum pricing for alcohol, plain packaging for cigarettes and the smoking ban, and claim they are not based on any evidence at all or are based on "so-called 'evidence'" that you claim is not evidence.

      However, you are not offering any evidence to support your assertion that these policies are not based on evidence and are merely assertions.

      If you don't offer evidence to support your assertion that these policies are not evidence-based, how can we ascertain whether they are evidence-based or not?

  6. Political ideology now trumps any evidence based fact.All you have to do is look at the current immigration debate, Austerity is based on the Conservative need to reduce taxes which inevitable leads to greater inequality and poverty for those who are deemed to be less deserving, Adam Smith is turning.

  7. dementia caregiving is a free social network that care & Support for the Caregivers, eldercare, senior caregiving, senior helpers, aging parent and family caregivers.

  8. I would argue that basing policy on models rather than evidence, or taking the evidence to fit the model (rather than the reverse) is also very bad.

  9. Keep up the excellent work exposing and exploring these issues Simon. Recognising what is happening in government and society at large, with help from you, Jonathan Portes, Paul Krugman and others, is inspiring me to try to do something active about it.

    I do think there is a problem with how well (or badly) the general public is informed on important issues. My own mum will regularly opine on how too many people are coming to Britain for welfare or health reasons, and that there isn't room for them. When I ask her how many people are immigrating for these reasons, she has no clue, but this does nothing to make her question her deep-seated, emotionally-rooted beliefs. She has no desire to seek out the answer to my question. And so it is for society more broadly: reasoned argument may not be sufficient to change people's minds and influence mainstream views. And politicians as well as journalists reflect these mainstream views.

    Behind it all is the identification with certain groups (e.g. "British", or political/social groups), and the mistrust, fear or even hatred of outsiders. This is a subconscious process for most people, it seems to be part of who we are as humans, evolved over a long time, but is today a hidden taboo. The problem is it can lead to destructive behaviour, including public policies, that harms everyone for no good reason.

    I feel like joining or starting a campaign against ignorance.

  10. Don't forget the Badger Cull, proposed changes in educational policy, ongoing attempts to toughen up law enforcement despite the fact that crime is at a record low, so on and so forth. The contempt of the government for reality is absolute.

  11. I would go so far as to postulate that the entire conservative political ideology is driven by emotion rather than facts and evidence. Tropes such as the hard-working conservatives versus the lazy welfare-ridden left, the health and welfare systems being overrun by millions of scroungers, or dictatorial lefty eggheads imposing their pie-in-the-sky egalitarian social schemes, have been relentlessly hammered home for the past few decades now. It just seems the modus operandi is to 1) go on a Two Minute Hate rant against the target of the week, 2) push a policy through based on no evidence whatsoever, 3) print a notice of error (if any) six months later in small print on page 52, but of course they make no mention of their viscious propaganda and the legislation remains on the books.

  12. Politics is generally based on preferences and not on evidence. (And often in that process the evidence that agrees with the the preference is cherrypicked).

  13. Excellent article.
    Though I was wondering if you have seen the research paper that published these findings: and have any comment / critique.

    1. The Guardian itself has a good discussion:

  14. Your blogs are doing a great public service. It is good for the public and democracy that economist's work is opened up and explained and good for economists that their work gets scrutiny by the public and people working in other areas. I understand your arguments about immigration, but I am not in complete agreement. While I do not think it is the cause of inequality and little progress in tackling low regional and inter-industry mobility and hysteresis in the labour markets, I do think, EVEN IF there has been a real GNP per capita increase in wealth with immigration, the income effects have not outweighed the substitution effects. One of the great tragedies of the Blair years was that we had massive employment growth, yet long term unemployment remained entrenched. We should have got these people working in the hotels, as security guards - any way we can get them in the workforce and earn a decent wage that covers the high costs of living where the jobs are. Instead things like highly elastic labour supply from Eastern Europe and a well-developed apprenticeship system that was left over in Poland from the communist era supplied this labour demand for us on the cheap. I think it is also important not to patronise people with different views. Some people feel a sense of alienation in their own country. Crowding, road and other congestion in Britain is high by any standard - compromising the living standards of the lower and middle income classes who are not part of the cosmopolitan political elite. While this is probably only partly a result of immigration, the views of a large proportion of the population should be listened to.

  15. This comment has been removed by a blog administrator.

  16. This comment has been removed by a blog administrator.


Unfortunately because of spam with embedded links (which then flag up warnings about the whole site on some browsers), I have to personally moderate all comments. As a result, your comment may not appear for some time.