Winner of the New Statesman SPERI Prize in Political Economy 2016

Saturday 3 December 2016

Hitting back

Not a post about a certain byelection, but a reaction to reading this:
“A more serious incident was the forecast by the Office for Budget Responsibility in the UK, which said last week that Brexit would have severe economic consequences. Coming only a few months after the economics profession discredited itself with a doomy forecast about the consequences of Brexit, this is an astonishing reminder of the inadequacy of economic forecasting models.

The truth about the impact of Brexit is that it is uncertain, beyond the ability of any human being to forecast and almost entirely dependent on how the process will be managed. “Don’t know” is the technically correct answer. Before the referendum, Project Fear was merely a monumental tactical miscalculation. Today it is stupidity. One of the debates was whether people should be listening to experts. We have moved beyond that. Because of a tendency to exaggerate, macroeconomists are no longer considered experts on the macroeconomy.”

Shrug your shoulders and move on? If it had appeared in the partisan press that would be a sensible reaction, but this was written by a widely respected journalist in the UK’s internationally renown financial newspaper. Furthermore - lest my motives be misunderstood - written by someone whose knowledge on the Eurozone is beyond dispute and whose views I often agree with. Well on this occasion this particular member of a discredited profession who is no longer apparently considered an expert on macroeconomics is not prepared to take this kind of stuff anymore, whoever it may come from.

It is difficult to know where to start with such apparent and complete ignorance. Nonsense expressed as platitudes. You can only make sense of “beyond the ability of any human to forecast” if you either think we know nothing about the impact of trade restrictions, which is false, or that forecasts are non-probabilistic. No journalist has any excuse nowadays for misunderstanding the probabilistic nature of forecasts (Bank of England fan charts), and any academic economist who knows anything about forecasting will tell you that unconditional macro forecasts are only slightly better than intelligent guesswork. They exist because it is worth being slightly better than guesswork when the stakes are so high.

You can also only make sense of these two paragraph if the writer is unaware or is just choosing to ignore the difference between conditional and unconditional forecasts. These are long words for a very simple concept. You would not dream of asking your doctor to forecast the number of times you would catch a cold over the next year (an unconditional forecast), but if you gave them all your relevant data they could probably make a better guess than your own. Their forecast would be probabilistic, but if you took the mean as ‘the forecast’ then in any particular year your doctor would generally be wrong. It would be absurd for you to then say that, having ‘discredited the profession with this inaccuracy’ you were now going to ignore their advice about how to avoid catching colds (advice based on conditional forecasts). But this is the logic of these two paragraphs.

As for a tendency to exaggerate, the simplest response involves a black kettle. But on this particular occasion I think there is a more honest response. In the Brexit campaign I felt the temptation to exaggerate (I don’t think I ever succumbed), because the media was failing to get the message from economists across. Our collective knowledge about the impact of trade restrictions was treated as just one more opinion, or described as Project Fear. When you are effectively being ignored you tend to shout louder.

But this is all defensive. Trying to explain yet again some basic economic ideas, and to be honest about what you can or cannot do and any failings you have. I’m just tired of doing this stuff over and over again, so it is time not just to defend. There are many good journalists out there, who when they write about macroeconomics do try to check with academics that what they are writing makes sense. (It was one of those journalists who drew my attention to the article I quote above.) It simply lets them down when others think they can write this sort of stuff without any of the kind of basic fact checking that journalists are supposed to do. It brings the profession of journalism into disrepute.

And they can only get away with it because academic economists only get a media voice by the grace and favour of journalists. If anyone should be doing some serious introspection after the Brexit result it should be journalists and the media. Warning of the dangers of trade restrictions was not a ‘tactical mistake’. What was a mistake was for journalists to allow those warnings, that knowledge, to be characterised as Project Fear, all in the name of ‘balance’ or cheap copy. But this was not a temporary lapse in an otherwise good record, but just another example of a growing tendency for the media to allow politicians to define economic facts and truths, a record I described in my lecture.

To have the nerve to blame economists for the Brexit result, to suggest that using their knowledge was a ‘tactical mistake’, to imply that the OBR should pretend they know nothing about Brexit, all that is itself amazing malevolent chutzpah. But it goes beyond audacity to criticise a profession and subject matter you appear not to understand when it is this lack of understanding that has contributed so much to the damage over the last few years.


  1. I didn't understand his (gratuitous?) criticism of macroeconomists either. Whatever the outcomes of Brexit we won't know them for sure till after the Brexit event, should it ever happen. To complain that the predictions proved wrong before the event has happened is bizarre.

    However I do agree with his final comments about the failure of anybody to
    "...solve the problems of an out-of-control financial sector, uncontrolled flows of people and capital, and unequal income distribution...".
    Hard to impossible problems - but until we address them (or events change the game for us) the how's and why's of Brexit seem almost a collective displacement activity. Let alone climate change.

    Maybe that's the point.

  2. No one questions the science in the medical profession (you make an analogy between economists and doctors), but they do macro-economists. Would you like a cancer diagnosis based on something like a rational expectations model? Where did the Treasury get the 4000 pounds a year from? From something like this.

    Manchau is right, the answer is we simply don't know. These models do not help us understand issues, they greatly mislead us. I agree, there is likely to be an adverse long term impact from Brexit. But it is unquantifiable and macro-economic models will not help us understand much. And there was indeed a tendency to exaggerate the effect of Brexit, particularly in the short term. In the end macro-economists and Project Fear simply strengthened the Eurosceptics and the Far Right.


    1. Even if it were right to say that "[simply] don't know" is the right economics-informed answer to the "likely impact of Brexit" question it would be wrong to say that it is unquantifiable.

  3. Simon,

    Aren't you taking this very personally? Your constant Quixotic flaying of the media is not very edifying.

    Why would you expect the media to believe anything an economist or group of economists said. Forget about whether economists exaggerate or not, the fact is there are groups of economists saying perversely different things. So why would anyone expect the media and the public to take head of advice from any one group of economists when they all contradict each other.

    All any economist can do is offer his/her opinion and take what comes back even if it is indifference. Brexit was about more than economics. Economists don't have a monopoly on effective and relevant policy.


    1. You do not ignore the views of 22 economists because 1 says something different (Brexit). And you do not present the views of a minority as if it is undisputed fact (austerity). Of course I take this personally - what is the point of good economic research if policy ends up being what sells best to an uninformed media.

    2. There is an entire continuum of possibilities between a perfect consensus and a perfect disagreement where all experts have clearly distinct opinions. As such, diversity of opinions in a field does not justify ignoring the dominant views -- especially when it is very close to a perfect consensus.

      Another issue here is how results are obtained. Some conclusions are very robust -- that is, they don't change a lot when you change your modeling choices or the estimators used. When you have a majority of experts believing that A is true and a buttload of very different papers pointing in that direction, you clearly have enough information to beat the hypothesis of "it's anyone's guess."

      If you have to manage ANYTHING, your decision should be based on your best guess. It is always going to be short of certainty -- but that doesn't justify jumping from listening to experts and ignoring them. You know, doctors sometimes disagree about a diagnosis. Maybe you should then just ask your neighbor about their opinion... After all, there is no perfect consensus and mistakes are possible. You'd never do this because going with expert advice is playing the odds. The same goes with economists: even if we just provide a fraction of a percent in edge over guesswork, it is justified to prefer our advice. And, the thing is, we outperform guesswork by very large margins in many fields -- which is why private businesses often seek advice from economists.

      Your basic alternative besides asking for economic exertise is to let politicians sell their snake oil.

    3. I am not British - I was not subjected to the day to day argy bargy of the Brexit debate so I am not sure who said what. Was there really only one economist who argued the Brexit case?

      Anyway, economists have a clear and long history of contradicting themselves on just about every issue. Why would the public/media, in a general sense, have any confidence in what an economist says or group of economists say, having been conditioned by economists arguing amongst themselves since time immemorial. Look at the internecine mauling that economists have engaged in since 2008. Let's forget about predicting the GFC, economists could not form a consensus view on how to deal with the crisis. I think public/media cynicism regarding the advice of economists is understandable.

      And the almost plaintive reliance on a "best guess" is hardly inspiring.

      Economists have to get their own house into order.

    4. According to the IPSOS MORI Veracity Index, hairdressers are considerably more trusted than economists to tell the truth![m]/0/


    5. And economists have double the rating of journalists, yet the public, according to Simon, took heed of journalists over economists.


  4. Alexander Harvey4 December 2016 at 09:36

    I don't have access to the FT but I believe that the following was the immediate preamble to the quote you give.


    At a time when Britain has voted to leave the EU, when Donald Trump has been elected US president, and Marine Le Pen is marching towards the Elysée Palace, we — the gatekeepers of the global liberal order — keep on doubling down.

    The campaign by Tony Blair, former UK prime minister, to undo Brexit is probably the quaintest example of all. A more serious incident was the forecast by the Office for Budget Responsibility in the UK, which said last week that Brexit would have severe economic consequences.


    An obvious inference is that he thinks the OBR report was an intervention in support of a "global liberal order". Surely that's nuts.

    Also I get a sense that he's become an economincs agnostic, no longer believing that the field informs us as to the likelihood of various outcomes.

    Because he believes we are failing to address his list of root causes to his view of the problems of his bigger picture, he asserts that there is no point listening to economic projections based on the assumption that his problems are not being addressed.

    Perhaps when he has cured our deeper ills, he will get back to believing in economics.

  5. I do sympathise! but you're now going through the same propaganda that miners & workers in general went through in the 80's & since.because the economics,politics & journalism is all aimed at destroying the truth of the fact, that flat earth economics is completely useless and dangerous for humanity.
    They must crush academia "Experts" and turn them into quacks,whilst in truth the real quacks are those doing it,welcome to the Orwellian world that most people have been living in for 40 yrs.
    Off course what you're experiencing is the closing down of descent moral values,but fear not for the illiberal ignorance of these callous people is that they in trying to divide & rule will & have taken it through there own paranoia & madness to the point where actually they are uniting all those they have disenfranchised against them,fear not who rises out of the ashes first for they're is the poison chalice.
    whilst like in Germany nothing changes nothing improves it only hastens the day real change & if not exterminated from the world the voices of truth will once again rise & even if they are,the logic of truth will one day return,these kind of people were always cast out of good societies & if humanity is to survive & flourish once more all truth Sayers must like you have here make themselves heard,to once again force them out of society that's why they hate society, we really are back to a world were protecting falsehoods to protect power is the norm,just very Orwellian but then Orwell was just showing how humanity works when it is all about oppressing humanity & not setting it free

  6. Will Douglas-Mann4 December 2016 at 12:42

    I see Donald Trump is to redirect NASSA funding away from research into climate change.
    The way things are going I would t be surprised to see other areas of academic research deprived of funding if politicians don't like the conclusions.

  7. This is the angriest post I have read by you. Perhaps it is worth considering that anger for a moment.

    The language and concepts that much of the media and politicians use to talk about economics (what you have called "media-macro") seems designed to spread misconceptions with the simplest means possible. When the woodgrain of public debate is defined by such terms, it takes longer and more complicated posts to work against it, no matter how much more coherent and simple such counter-arguments may be. The media-macro ideas like "Project Fear" and "the national credit card" therefore looks more simple, and the simpler argument is not only the one easier to communicate in the mainstream press, but also the more convincing (a la Occam's razor). After fighting against this for years, it is no wonder that you slowly beginning to tear your hair out.

    Perhaps it is time to fight back not with reasoned argument, but with a counter bullshit tailor-made for the liberal left. But that is a activity which, I imagine, you do not want to get into. It is a job for politicians rather than academic economists.

    Andy Murray.

    1. I would put it slightly differently. Here we have a journalist who is both clever and not in the business of pulling the wool over peoples eyes. The fact that he could think of writing such uninformed nonsense without realising that it lets down the standards that he would apply to himself shows there is a structural problem. And that, as an academic, is what I have tried to analyse.

    2. Good point.
      Andy Murray.

    3. A lesson for me from last year's General Election and the Brexit vote is that, sadly, a simple lie often bests a complex truth. But it would be very dangerous for society were the Left to respond with a counter brand of bullshit (tempting as it might be). The solution is to coin simple, compelling slogans that promote the truth. For example, austerity doesn't work because in an economy my spending is your income and your spending is my income - if we both cut back we'd both be poorer. (Ok, simpler still would be better!)

      In general, I find slogans that link to business and have a smidgen of patriotism do well. An example is SWL's excellent hypothetical Q&A between an MP and a reporter, which should be in every Labour MP's notebook. It included comments along the lines of "sensible companies borrow to invest when interest rates are low and they see a high return opportunity - the government is being short-sighted for not doing the same. Don't they believe in Britain?"


  8. I'm not an expert in macroeconomics, but it can be argued there's a logical fallacy in your thinking. Three times in the piece you mention "trade restrictions" suggesting they are bad, and you seem clear that macroeconomics has demonstrated that. Fair enough. From this I infer that the assumption in these probabilistic models is that Brexit leads to greater trade restrictions, but this seems unfounded, perhaps based on (unintended) political bias.

    Those on the Brexit side would argue that it is this approach that stands for free trade, away from the restrictions of the EU (which is seeing the slowest economic growth of any economic block). By distancing ourselves from the EU we are far more able to do separate unrestrictive deals with the major and growing economies of the world. For instance, it's fairly clear Trump wants a free trade deal with the UK quickly, and many of the Asian economies will also want to deal simply with the fifth largest economy in the world rather than interminable complex negotiations with a 28 country block (and their subdivisions).

    Then, what does the EU decide post-Brexit (because it is surely their decision)? Economically it's clearly in their interest to also do a free trade deal with the UK, but politically they might decide that this is untenable. But it's a tough call to stay in government if you sacrifice voters' jobs for a point of political principle.

    It does indeed seem that the OBR has decided to double-down on its prediction that a Brexit outcome in the referendum vote itself would have immediate very bad consequences for the economy (a prediction that strongly suggests their model was wrong). What is your and their justification that membership of the EU is better in a world where trade restrictions are bad, when to an outsider in this debate that seems an unfounded assumption? Thanks.

    1. I think this comment shows the complete inadequacy of the media's Brexit debate. The media did not want to discuss the Treasury's gravity equations - indeed it made fun of them in its usual way. But these are all about the fact that it is still less costly to trade with your neighbours. That is why the 'we can trade more with others' argument is wrong.

      And it is trade economics, not macroeconomics, by the way.

    2. Keithmansfield i realise that your question was aimed at Mr Wren-Lewis,but the fact is we are going to have to pay to have access to the EU,which is a tariff in itself!Yet know we have no access to influence future regulations which can be a good thing,the British empire was built on regulations of coinage weights & measures that gave confidence that the coin you was receiving was of value &would buy goods to that equivalent,which forced others to raise standards,regulations stopped arsenic being used for medicine,sawdust in bread etc etc,regulations reduce when policed right fraudulent practises that can be outright dangerous,hence we see has standards fall more fraudulent practises not less this in effects reduces confidence in products,security & in general economic outlook,we only have to remember salmonella in eggs for proof of that being just one many,so we will pay to have access yet have no say because all regulation & tariffs are not good ones & arguing whether their good or bad is totally different to having to abide by them for access.
      The model may well be wrong,but i time delay ii we haven't left or even attempted to leave iii the uncertainty that the process deliberately creates also hasn't kicked in,in full,once we start on the path to actually leaving,with world factors the models may prove to be right(or even understated),since it was based on leaving & many including the OBR may well have expected article 50 to be invoked immediately!or based their predictions on that so immediately could even should e interrupted has the point of exit,but in not doing so mischievous & malicious political gain is being used in the vacuum,including yourself!

    3. Perhaps I may add a few thoughts additional to those of the learned Professor.

      When considering trade questions nowadays, one needs to focus on non-tariff barriers as well as tariffs, as these are arguably often more important at a time when tariff levels are really quite low by the standards of previous years.

      The argument for the UK staying in the single market and customs union rests to a considerable extent on the fact that within them there are neither tariff barriers nor to very considerable degree many of the non-tariff barriers which could otherwise be created (paperwork, delays at borders lasting days, inspection requirements -with costs charged to the trader etc.). Establishing free-trade agreements with non-EU blocs would dispose of tariff barriers but not non-tariff barriers. Leaving the single market and customs union would re-establish both sorts of barriers with all EU members. The probability that the net effect would be damaging to UK trade is so close to 100% as to constitute virtual certainty

      Real world considerations need to be taken into account, which are generally glossed over when these matters are discussed in the UK. Thus, the US is already a relatively easy market for most to export to, so signing a free-trade agreement with the US will probably do little to improve formal UK access to US markets but with a disproportionately larger improvement in US access to UK markets. Why do you think Trump is keen to do a deal quickly? He can see when a potential negotiating counterparty is in a weak position, which the UK will be if it leaves the single market and the customs union, for HMG will probably be desperate to be seen to be doing deals if only for PR purposes.

      Deals with Asian countries could be signed but it is very doubtful that these would result in very much larger trade. These countries are in truth, many of them, very resistant to buying from abroad so that a piece of paper will have little practical effect in reality.

      Most British exporters have made it clear that they want the UK to stay in the single market and the customs union. One of the depressing features about the recent discussions and the Brexit debate has been the extent to which armchair pundits have chosen to ignore the clearly expressed views of those who actually have to go and generate the UK's foreign earnings so that we can afford to import the vital goods we need.

    4. So, if it is 'less costly to trade with your neighbours', the EU will want a free trade deal with the UK - won't it? It's surely in the interests of the EU's citizens to have one. And given that the UK would like one, what's the problem?
      Or is it that the EU leaders want to punish UK citizens for leaving even if it means punishing their own citizens at the same time?
      And if that's the case, why would we want to be in such a despicable club? Sounds a bit like a protection racket to me.
      I've read numerous times about the need to punish the UK because if leaving is seen to be good for UK citizens, others will want to leave. Well surely if leaving would be good for a particular country, then everyone should want them to leave because we all want what's good for people, right?
      Apparently not. The club seems to have become an end in itself. Or more likely, it's in the interests of the club's elite hierarchy to maintain it.

  9. I was a financial journalist for over 40 years. As I watch TV news or read quality papers, a couple of things really annoy me:
    1) unasked questions. Best seen in the current soccer scandal. The obvious question is whether the same problems occur in other sports such as gymnastics, swimming, tennis etc in which ambitious youngsters are in close contact with powerful coaches for prolonged periods. During the Brexit debate there were many examples.
    2) phony "balance." Best seen in debates about climate chance. Typically one scientist representing the views of over 90% of specialists is pitted against one "denier," and their views are given equal weight. Or 20 macroeconomists vs 1, for that matter.

  10. The author appears to be referring to the effects of brexit pre-referendum short term forecast by the treasury and pointing out it was wrong so why should we believe the longer term forecast. He has a point - the forecast correctly predicted the fall in sterling but was dead wrong on the immediate impact on consumer and business spending and consequent recession. Wouldn't be so bad had they not claimed it to be 'cautious' and a 'rigorous analysis'. They clearly overstated the case and so is it any wonder that there is scepticism?

  11. You’re missing the point. None of the macroeconomic analysis carried out prior to the referendum actually assessed the impact of Brexit. All they did was to compare two (or more) cases differing in the degree of openness to trade and conclude that a less open economy will tend to grow more slowly than a more open one. As the UK outside the EU was assumed to be less open, the conclusion naturally followed that the UK would then grow more slowly.

    It is now obvious that Brexit entails far more than just changing trading relationships. The referendum result has initiated a sequence of political consequences that are shifting the assumptions on which the economic models were based.

    You state ‘You can only make sense of “beyond the ability of any human to forecast” if you either think we know nothing about the impact of trade restrictions, which is false, or that forecasts are non-probabilistic.’ But we are now in a period of radical uncertainty in global politics in which “don’t know” is a reasonable position on where the world might be in a few years’ time. There is no mention of Donald Trump in the Treasury analysis.

    The economic argument that a less open economy will tend to grow more slowly remains valid but the consequences of Brexit go far beyond that. Not in a positive way. The immediate dangers are more evident than any potential benefits and the prospect of a Le Pen presidency worries me more than a few percentage points of GDP.

    But economists claim too much when they stretch their models to predict outcomes which will ultimately depend on politics and history. Right now, ceteris paribus is not a sensible assumption.

    1. I think you're over-stating Simon's aim. He's primarily trying to get across what you yourself say:

      "The economic argument that a less open economy will tend to grow more slowly"

      Judging by comments in the media and some above, that simple point is proving devilishly difficult to get across.


  12. There was an important point hidden behind the hyperbole. No-one knows what the possible political models might be, let alone how likely they are.
    You might be able to predict the economic consequences of a Norway-type Brexit, of hard Brexit, of various hybrids, at least somewhat better than random guessing. But how do you combine these predictions? If you have to take the average of three completely different number and two "Don't Knows", then the correct answer IS "Don't Know".

  13. Simon,

    I read him as saying something slightly different and (I think) making a point that economists need to take seriously.

    Suppose you are asked to forecast the roll of two dice.

    the correct answer is 7.

    the dice are rolled and we get 3.

    you have made the wrong forecast, and are discredited.

    if you re-read Münchau, don't you think he could be talking about what did happen, and how economists / elites are behaving, not how things *ought* to be?

    fact is, we live in a world where if you forecast 7 and the dice come up 3, you *are* discredited because, well quickest version: people are stupid. To give people more credit, we live in a world where we are constantly faced with situations where you would need N=1000 to learn who is, and who isn't, making good forecasts, but we only ever have N=1 (or 3 or whatever) and we only ever see the realization and the point forecast, and in that world maybe judging forecasters on that basis isn't so dumb (perhaps it is constrained rational or some such thing).

    he talks about economists being over confident, I think what he could be getting at is that the message that people hear (and understand) is not "here is our best point forecast but the probability density is very flat" - that if economists had real message discipline they would not allow themselves to be quoted predicting X or Y, but we be saying "we don't know" much more prominently and talking about how the timing of effects is uncertain, and how things that happen very slowly in a world full of noise are hard to perceive. Etc. Because that is how you avoid being discredited. Because, if you take this as a description of popular perception and no more, economists have been discredited. And whilst journalists shoulder some of the blame, well even the best of them suffer from the deformation professionnelle of looking for the *story* which means painting in bright colours, not fan charts.

  14. This was written by a widely respected journalist in the UK’s internationally renown financial newspaper... furthermore written by someone whose knowledge on the Eurozone is beyond dispute and whose views I often agree with.

    If that was Wolfgang Münchau, you only prove your lack of judgement. It has been possible for years to realize that he was continually writing nonsense.

  15. “You would not dream of asking your doctor to forecast the number of times you would catch a cold over the next year (an unconditional forecast), but if you gave them all your relevant data they could probably make a better guess than your own.” This sounds reasonable but the problem is the phrase “all your relevant data”.
    The trading and macroeconomic models used to predict the effect of Brexit did not use all the relevant data, either because the data were (and are) not available, or because some of the available data were not usable within the models. Examples include election results, resignations, legal rulings, negotiating stances, etc.

    It was (just about) conceivable that the political fallout from Brexit could have led to radical policy changes (such as abandoning austerity across Europe) which would have had larger positive effects on output than the negative effects from trade restrictions. That has not happened and the political consequences seem far more likely to have negative than positive economic impact, even excluding possible outcomes such as military conflict over the south China sea or Korea, or further disintegration of the EU.

    This is not to assert that the economic models are useless, just to recognise that, on an issue of the magnitude of Brexit, they contribute to our overall understanding but do not wholly encompass it.

  16. Of course you are right, ceteris paribus. The point in the article, as I read it, was that there are so many binary factors influencing the outcome, and many of these are cross roads with true uncertainty (will lePen become president) that only the great prof Hari Seldon at the Streeling University could give any meaningful estimate of the outcome.

  17. "unconditional macro forecasts are only slightly better than intelligent guesswork."

    You're going to have to produce some evidence for that because I don't see any, at least over the last 20 years.

    It's a fact of life that economics is a branch of politics and your take of trade restrictions is a political stance which ignores, among other things, questions of slavery and exploitation which are part and parcel of free trade. The idea that it is cheaper to trade with neighbours is also somewhat antiquated, at least in the sense that it is so much so as to be an important factor compared to, say, relative wages.

    Economists frequently talk about "better" and "worse" but they rarely concern themselves with who is better or worse off, at least when the "better" part includes their own social class. We (the non-economists) would be better off if you (the economists) would stop pretending that what they do is a science or even a branch of mathematics so that we could have honest debates about the assumptions that underlie all economic models, forecasts, and judgements.

  18. Simon,
    The problem is your economics did not fit some people's political narrative with the added defect of being wildly exagerated. Of course we don't know exactly how Brexit will play out but the only question is how much damage will it do in the medium term rather than how much benefit is there.

    We'll never know anyway because Brexit will never happen. bill40


Unfortunately because of spam with embedded links (which then flag up warnings about the whole site on some browsers), I have to personally moderate all comments. As a result, your comment may not appear for some time. In addition, I cannot publish comments with links to websites because it takes too much time to check whether these sites are legitimate.