My guess is that the majority of pieces of empirical work by
economists will contain at least one error somewhere. Errors become almost
inevitable when large and diverse data sets are involved, like those
constructed by Reinhart and Rogoff
and Thomas
Piketty. So finding these errors is not headline news. Nor, for this
reason, is it particularly embarrassing for the economists concerned when these
errors are found, particularly if they have made their data public or available
to others.
If you think that shows up empirical economists in a bad light,
the best economic theorists can also make errors. One celebrated paper from my youth, Does Fiscal Policy
Matter? by Alan Blinder and Robert Solow, contained an algebraic error (pdf). And if you think this shows up all
economists, probably the most famous mathematical event of the last few decades
- the Andrew Wiles proof of Fermat’s Last Theorem - contained what a journalist
would call an error in its original form.
It is also often necessary to adjust data for a number of
reasons. In an ideal world each adjustment would be carefully documented, but
they rarely are. Of course official data series often involve many similar
adjustments before they are published. If you can, it is always a good idea to
talk to the statisticians involved in constructing your data before you use it,
although it can put you off doing any empirical work ever again!
Errors and adjustments only matter if they influence key
results. The Blinder and Solow algebraic error was not critical to the main
results in their paper. The gap in the original Andrew Wiles proof was
critical, but after what must have been an agonising year, he found he could
bypass the problem and the revised proof was sound. The Reinhart and Rogoff
spreadsheet error had a relatively minor impact on its own - the really
important issues lay elsewhere.
With all this in mind, I have very mixed feelings about Chris
Giles’s Financial Times splash. I applaud a journalist who is
unwilling to take academic results or official figures on trust, and is
prepared (and I guess has the resources) to get their hands dirty with data.
Chris has consistently done this. For example, when David Cameron claimed
mysteriously that George Osborne’s first austerity budget would increase public
sector employment compared to Labour’s plans, Chris got to the bottom of how this trick was
achieved. Yet I groaned when reading his latest FT article, with its emphasis
on “mistakes and unexplained entries”. As far as I can see (read Ryan Avent here, and the longer Chris Giles
post here, and Jonathan Hopkin here),
the only issue of substance involves trends in the UK wealth income ratio, but
of course an article headlined ‘Data sources on UK wealth income ratio differ’
would not have had the same punch.
Now you might say, as journalists always do, that people who
become famous - including economists like Reinhart and Rogoff or Piketty - have
to accept having their work treated in this way. They become ‘fair game’. I
actually think that is wrong. Misleading reporting and commentary - by
journalists or bloggers - is what it is: misleading. The fact that it can be
commonplace does not excuse it. I understand the temptation to hype up simple
spreadsheet errors even when they have no significant consequence, but I’m glad
to say I did not succumb to this temptation in the case
of Reinhart and Rogoff spreadsheet
affair.
It is perhaps worth noting one other point. The Reinhart and
Rogoff affair became notorious because governments had used this work to
justify their austerity policies. The spreadsheet error was brought to light as
a result of work by academics rather than by any journalist. In the case of Piketty,
no policies have yet been implemented using the results in his book as
justification. In that rather important sense, the two stories are different. Whether this asymmetry reveals anything of interest I will leave you
to judge.
Austerity measures in Greece (based on erroneous estimates of fiscal multipliers) have produce of an explosion of new of HIV infections in Greece (documented here : http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62291-6/abstract) according to The Lancet. Piketty's book have not until now induced such dramatic consequences...
ReplyDeleteJamel Saadaoui
Great piece. Hopefully the debate will move on to what we all want to know - is inequality rising? Secondly, and better still, if it is, what are we going to do about it?
ReplyDeleteI found the comment Giles made about weighting according to population odd. Rogoff also made a point in his review of Piketty that inequality worldwide is decreasing. But this is explained by the high growth of China which accounts for 20 per cent of the world's population, since it entered the international economy . Remove this and things do not look so good, especially in Africa. Weighting can distort the picture. Surely Piketty was interested in the cases of growing inequality, whether there were common cases across countries. He wanted to see whether on average countries' inequality was rising or falling. A weighted average in such a case is not really appropriate. I fear this might be indicative of the level of quality of discussion in relation to Gile's critique.
ReplyDeleteA weighted average makes sense if the datasets are consistent, i.e. you could mix them up without affecting the individual data points (e.g. occurence of blue eyes among UK, French and Swedish populations).
DeleteBut it is problematic when doing international comparisons of a relative measure within a country, such as wealth inequality, which is heavily influenced by national factors, such as taxes on wealth, tax morale/avoidance and rates of home-ownership. This is a problem that has long bedevilled the use of Gini coefficients, which tackle the similar, but much easier, task of measuring income disparities.
The implicit question is whether a poor person in the UK is comparable with a rich person in Sweden, and the answer is both yes and no. Clearly, translated to a common scale (e.g. buying-power in Dollars), the Swede is wealthier than the Brit, but inequality as experienced daily is a measure of that Brit's wealth relative to other Brits, not to Swedes. Someone on a comfortable income in the UK will feel uncomfortable buying a round in a Swedish bar.
Sweden started with lower levels of inequality than the UK and France in the 70s, but it has converged on the same levels since. This means it has seen a faster rate of growth in inequality over the period, which explains why Giles is keen to under-weight it at the aggregate level.
So, just to be clear, your claim is that journalists have unacceptably high standards for accuracy?
ReplyDeleteGiles hasn't just pointed to one or two slips here.
My claim is that Giles has written a misleading article because he wanted to make a splash.
DeleteCan you imagine what the reaction would be in a proper science, like chemistry say if a peer reviewed paper had been revealed to contain as many "slips" (columns misread, at very best misleading interpretations of the data in relation to large sectors of the set under survey, a clear bias in the way the findings are presented.)
DeleteIf the claim, seriously, is that this is all just par for the course in economics, so much the worse for economics.
As for the whataboutery ('R & R were even worse'), well yes there have been even worse lapses in high profile economics papers in the recent past. Disgracefully bad. I am at a loss to see how that makes this any better.
I'm afraid you illustrate exactly why Giles's article was misleading. You glide effortlessly from 'columns misread' (it happens), to 'misleading interpretations of the data' (not established, just alleged), to 'a clear bias' (a very serious charge, in no way established by Giles).
DeleteThat is exactly what the article wanted you to do. It knew there was a large section of the readership who would love to read 'Piketty got it wrong', and would not look too critically at any article that said that.
If Giles had been mainly interested in the truth, he would have given Piketty more than a day to respond to these detailed criticisms. He was much more interested in making a splash. Because he knew people like you would lap it up.
No need to stoop down low and waste your time by responding to people who start their comments with "what you're saying is..." or "you claim that...", who then go on to ascribe to you a completely fictional, non-sensical position.
DeleteKeep up the outstanding work.
Prior to Piketty the received wisdom (or something close to it) was that growing wealth inequality was an issue in the US (the hollowing out of the middle class) but much less of an issue in Europe. The conclusion we could then draw was that some kind of European style social democracy was enough to keep the problem in check.
DeleteWhat seemed so important about Piketty was the claim that this was not so, and that the European data supported growing inequality too.
Now if Giles is right, the errors in particular in relation to the UK don't really support the European thesis. UK inequality grew for a decade (Thatcher) but hasn't inexorably grown in the way Piketty claimed.
Which is why Krugman's claim that Piketty is still right in relation to the US is to miss the point
http://krugman.blogs.nytimes.com/2014/05/24/is-piketty-all-wrong/
The issue is whether we need the radical measures Piketty claims, or whether something like New Labour vintage 1997 is enough.
Which is why getting the UK all wrong (and partially as a result Europe as a whole) as Giles claims Piketty does is actually important.
On this point Giles may of course be wrong and Piketty right, but as presently advised he looks right to me.
Journalist deliberately makes a splash?
Nothing to criticise, that is his job.
Academic makes a splash, sacrificing the credibility of the academy and his discipline as he does so?
I think your penultimate sentences say a lot: "Journalist deliberately makes a splash? Nothing to criticise, that is his job." I would say a good journalist presents facts and issues in a reasonable way. If those facts warrant a splash, that is fine.
DeleteIn this case they do not. In terms of the data, the only case that matters is the UK. The only consistent series we have is the IRS data, which has an upward trend. Giles wants to use data from another source to suggest UK wealth inequality has turned down in 2010. He could have written something interesting on that, raising obvious questions like is 2010 a representative year, how come rising income inequality and house prices have not raised inequality etc. That would have been good journalism.
But instead Giles has played the man rather than the Ball, with all this stuff about “mistakes and unexplained entries”. Instead of raising the specific UK issue with Piketty, he throws the whole "rock-star French economist appears to have got his sums wrong" stuff at him with a day to respond. It is interesting you think this is good journalism. Have you tried the Daily Mail?
Re-reading SW-Ls blog above a number of times I'm still searching for anything at all in which can possibly be (mis)interpreted as our host claiming "journalists have unacceptably high standards for accuracy?".
DeleteEspecially strange given that SWLs piece is so clearly attacking headline grabbing, misleading journalism.
Again, don't waste time and effort on trolls Simon.
Krugman was far harsher than you:
ReplyDelete"There were extensive critiques of both A-A and R-R as soon as they were circulated, raising almost all the points that finally got widely accepted in 2013 or so. No, we didn’t know about the coding error, or the peculiar weighting strategy that made R-R’s results sensitive to a single year in New Zealand. But the most important objection — that the causation ran largely from slow growth to debt, not the other way around — was out there right from the beginning. But A-A and R-R told the likes of Olli Rehn and Paul Ryan what they wanted to hear, and so their work became immediate gospel. The nakedness of the whole thing remains amazing" (Apr 2, 2014 The Will To Believe).
Krugman's October 7, 2013 'Credibility All The Way Down' puts his and the Wren-Lewis points as answered by Rogoff.
And of course there is the Krugman blog 2014 Feb 19 'Letting Lehman Fail' in which Vince Reinhart and Ken Rogoff saying that the collapse of Lehman Brothers wouldn't be too bad a thing:
Reinhart: "But one thing is clear: Lehman did not cast a long enough shadow over markets to warrant support. And Treasury Secretary Henry Paulson and his colleagues are to be congratulated for the courage to make that determination."
Rogoff: "This past weekend, the U.S. Treasury and the Federal Reserve finally made it abundantly clear that they won’t bail out every significant financial firm in America. Certainly this came as a rude shock to many financiers. In allowing the nation’s fourth-largest investment bank, Lehman Brothers, to file for bankruptcy, and by forcefully indicating that they are prepared to see even more bankruptcies, our financial regulators showed Wall Street that they are not such creampuffs after all."
October 7, 2013, 10:01 am
Credibility All The Way Down
My main problem with R and R was not the spreadsheet error. It was the notion that a country faces a bound on its debt of 90 per cent of GNP, irrespective of that country's size, industrial structure, terms of trade and trade structure, dependence on foreign imports and capital inflows, external position, institutional and social structures, political structures....
ReplyDeletePiketty's framework, however, is arguably a simple, but good starting point to start the debate on whether trends towards inequality is inherent in capitalist systems. Of course refinement has to be done and errors corrected, but let us not throw the baby out with the bath water.
Agreed, though I normally use far more brutal language than you when referring to R&R, because they basically have no grasp of what national debt is. That is, they fall for the error that non-economists (understandably) fall for, namely that because the phrase “national debt” includes the word “debt” therefor the national debt must have similarities to “debt” as the word is normally understood. Krugman said that R&R are motivated by a psychological hang up about national debts, or words to that effect. Krugman is spot on.
ReplyDeleteIndeed they were talking about government debt with little consideration about even whether a country has a large private debt or not. Sargent also said when asked about whether government debt levels are sustainable replied "the budget constraint makes it sustainable". This is a miserable example about what happens when you uncritically apply micro theory to macro. States are not households or an aggregation of such markets.
DeleteOne thing I think we all hope comes out of Piketty is that well informed historical analysis, both quantitative and qualitative makes its way back into the core of macro. Of course, ideally we want that without errors but as SWL points out and any of us who have done this know, with large data sets and historical data with difficult decisions to make with regarding the compatability of data sets etc, its going to happen.
From the abstract of R&R's paper:
ReplyDelete"Our main findings are: First, the relationship between government debt and real GDP growth is weak for debt/GDP ratios below a threshold of 90 percent of GDP. Above 90 percent, median growth rates fall by one percent, and average growth falls considerably more. We find that the threshold for public debt is similar in advanced and emerging economies."
In other words, the spreadsheet error was of great consequence. It's discovery showed that the first and presumably most important of the the reports three main findings was wrong.
I can sympathize with Professor Wren-Lewis' attempts to make the economic debate friendlier and I agree that we shouldn't heap scorn on someone just because they've made a mistake or two. I do belive however that people were right to make a big deal out the error since it appears to have had a big effect on policymaking over the last few years.
I still think the problem was making a big thing about the 90 per cent, which would have been a bad thing to do even if there was not a spreadsheet error.
ReplyDeleteThe FT behaviour looks sensationalist - this is tabloid territory - sending up the implications of allegations and picking out data without context. This is no way to critique a book or empirical investigation of this kind. You should google what the Daily Mail says about Piketty - pretty much the same as the FT. People usually rely on the FT as a serious source of information. I suspect though people can now find a lot of that online. Perhaps they are looking for a new market.
The reason why the 90% was so crucial is that such a threshold would indicate that debt really has a strong negative effect on growth. What other reason could there be for growth being slower above a certain level of debt?
DeleteIf on the other hand there is no debt threshold, it seems far more likely that the causation is reversed - that slow growth increases debt. Simple debt equations make it clear why slower growth leads to higher debt while there is no obvious theoretical reason why higher debt should lead to slower economic growth.
Moreover, without the coding error being revealed, serious work concerning the direction of causation might never have been published. See link for only the latest to show that R&R's assumption of high debt --> slower GDP was wrong.
Deletehttp://www.voxeu.org/article/growth-and-sovereign-debt-correlation
I would like to think this last point by 'unknown' is wrong. Suppose there had been no spreadsheet error. I think the Amherst academics would still have published their paper, showing the sensitivity of the results to other choices. I think this would have inspired the same blog debate. It would not have hit the headlines in the press, but I doubt that would have stopped academics doing additional research.
DeleteEqually Chris Giles could have written a very good piece pointing out the issues with the UK data. Instead he chose to sensationalise his article, playing the man rather than the ball.
So in both cases, what good has come from this media hype?
""What good has come from this media hype?"
DeleteSimon, for better or worse, Giles has completely achieved his objective - from now on, when Piketty is mentioned in policy circles, the words "debunked by the FT" will invariable follow.
That's a boost to Giles career (it was Giles wot did it!) and the political objectives of himself and the FT...
I fear the FT with its "gotcha" moment has successfully managed to trivialise a lot of substantial and useful work.
DeleteLook, splining, averaging, filtering and smoothing is fair game. Adding integers to a number because the data don't seem to fit on general picture one is trying to paint is not. I don't think Piketty acted in bad faith - else he wouldn't have let his data online. But scrutiny and robustness is essential to scientific progress and the FT report is a great piece of journalism.
ReplyDeleteWhile the spreadsheet error in Reinhard-Rogoff (2010) was certainly egregious, it was not the main factor driving their erroneous results, as Herndon-Ash-Pollin (2013) were at pains to emphasize. Instead, it was their cherrypicking of the data for New Zealand and their country weighting scheme and binning procedures that were the culprits, facts that were also not apparent until their spreadsheets could be examined (see http://silverberg-on-meltdown-economics.blogspot.co.at/2013/05/reinhart-rogoff-vs-new-zealand-final.html).
ReplyDeleteIn Piketty's case, he has made all of his procedures public, and he is sufficiently on top of the data problems to be able to identify why Giles' main point for the UK is flawed: Giles combines survey data for the later period, which tend to underestimate the upper tail of the wealth distribution (by as much as 10%), with tax and estate data for the earlier period, which are more accurate. That said, incomplete and inconsistent data sources like for wealth will always leave room for errors and disagreements about proper treatment, but Piketty (like Atkinson, Saez, Zucman et al.) have at least made a good faith effort to address the issues.
Where Piketty fails is in his r>g argument, which strikes me as deeply flawed from a macro point of view.