A few days ago, I had a back and forth with Narayana Kocherlakota on Twitter where he called economic forecasting of inflation a "success":
AIC refers to the Akaike information criterion (as a proxy for various information criteria) which is a maximum likelihood metric that takes into account the number of parameters in the model (penalizing a model with more parameters). EPOP refers to the employment-population ratio Kocherlakota refers to in his blog post (and for which there is a pretty good dynamic equilibrium model available, by the way). I also used π to refer to inflation as is common in DSGE models in my last tweet.
The data Kocherlakota compares to the Fed inflation forecast is much better described by a constant 1.6% inflation. Not only would that model have a better AIC (since the Fed forecasts undoubtedly contain more than one parameter), but a forecast of 1.6% wouldn't have the negative bias of the Fed forecast. This is basically saying that an AR process would outperform the Fed model (as we've seen before).
My main point was that on its own, the inflation forecast is not evidence of success because it is beaten by a constant. In physics, if a model was beaten by a constant that model would be rejected and it's likely the entire approach would be abandoned unless some extremely compelling reasons to keep going were found. But Kocherlakota's post was part of a short series wherein forecasts of other variables from the same framework were completely wrong. If the IT framework got GDP and EPOP completely wrong and then underperformed a constant for inflation, that would be more than sufficient evidence to reject it. One's Bayesian prior for the entire framework should be given a lower probability because it gets two variables wrong and doesn't do as well as a constant. This is to say a scientist would abandon pretense of knowledge of EPOP and GDP (and the framework that produced the forecasts) and resort to a constant model of inflation. Kocherlakota not only doesn't reject the framework but calls the inflation forecast a success!
The next line was Kocherlakota saying "This the point - [inflation] evolves almost completely separately." which I thought was a good enough stopping point because I am not sure Kocherlakota understood what I was saying. And on its own, that is an interesting point coming from a former Fed official ‒ that inflation seems independent of other macroeconomic variables. However, my point was that despite the fact that Kocherlakota was staring at evidence that he should probably be starting over from scratch, he wasn't seeing it.
It's almost amazing if that went over his head and he missed the implications of his last statement. Have you considered emailing him? I find that destroying someone's perspective is usually better done in private when it comes to getting them to understand a simple point.
ReplyDeleteI tend to follow Max Planck's view in cases like these:
Delete"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."
Jason: "I tend to follow Max Planck's view"
ReplyDeleteI agree with this but it is inconsistent with your view that peer review is an effective quality control for science. As far as I can see, peer review often leads to group think, particularly in immature subjects. That's definitely a problem in economics.
We need a better way of discussing what we should and should not consider to be "successful" science. This discussion should not be based on physics which is already a successful science. It should be based on more controversial subjects like economics and climate change, with major social implications, where we need to de-politicise the discussions; determine, as a society, what is the best approximation of the truth; and understand why intelligent people hold different views.
If economist A says we need policy X because policy Y won't work, and economist B says the opposite, how should the rest of society decide whom, if anyone, to believe without resorting to prior political bias and accepting the view of the economist who appears to agree with what we already thought?
That was mostly a fatalistic joke.
DeleteIt's not just peer review, but "Feynman integrity". Without Feynman integrity (leaning over backwards to think of how you can be wrong) the whole scientific process breaks down.
Kocherlakota should have been leaning over backwards to address that he might be wrong. He didn't and there's really no way to make him. He just has to be left out of future discussion because he isn't credible.
Your economists A and B are both not scientists unless they both acknowledge what could be wrong with policy X or Y, respectively.
But that's the problem. People seem to accept that kind of behavior from economist A. And really, it is just economist A. Economist A writes Op-Eds in favor of policies that benefit the wealthy while working for a think tank funded by wealthy donors. Meanwhile economist B tries to show what economist A is neglecting in their analysis and give evidence in favor of policy Y.
Yet most people seem to take both economist A and economist B as just "both sides" of a debate because they don't have a lot of experience dealing with the scientific process. They don't recognize the signs of failing to adhere to Feynman integrity.
For example, it just strikes me as completely strange that people don't immediately see Paul Krugman as a credible economic scientist and Ed Prescott or Greg Mankiw as not. At least in terms of Feynman integrity. In terms of politics, people just agree with people that agree with their positions.
I actually wrote a bit about this in a comment on David Glasner's blog. I took issue with something Glasner said:
DeleteGlasner: “Climate-change deniers don’t reject science; they reject or disagree with the conclusions of most climate scientists. They may have lousy reasons for their views … but holding wrong or biased views doesn’t make someone an opponent of science.”
JS: This doesn’t capture the real problem. The real problem is that these people lack what Paul Romer called “Feynman integrity” in a blog post awhile back – leaning over backwards to consider how you could be wrong. It’s not just holding incorrect views, but holding them without attempting to confront the idea that they might be the ones in error. The scientists in the IPCC reports are constantly questioning their own results. Climate change deniers do not. I think they think being skeptical of the consensus is enough to show this integrity, but it isn’t. This lack of introspection kills science; if science was to become a set of opinions about data where no one ever considered that they might be wrong the whole edifice collapses. That’s why there is peer review: to force scientists of all disciplines to consider they might be wrong. If a reviewer tells you you’re wrong and you don’t entertain that possibility, your paper will be rejected. Op Eds don’t quite operate in the same way, and that’s why climate “skepticism” proceeds mainly through political journalism than scientific journals.
I would like to add that I think (like climate change skeptics) a lot of heterodox econobloggers/economists think their skepticism of mainstream econ is sufficient to show "Feynman integrity". Being skeptical of the field's consensus does not "rub off" on your own theories -- you have to maintain a certain amount of openness and skepticism of them as well.
DeleteI try my best to fully document everything that I do and provide downloadable code at github using publicly available data sets. I make sure to say things are the results of specific models that I document -- and note that there exist competing ones. I track my forecasts.
One thing I've realized is that this is considered highly unusual in economics. However it shouldn't be.
I agree with a lot of this although you don’t go nearly far enough. However, let me start with a disagreement.
ReplyDeleteJason: “a lot of heterodox econobloggers/economists think their skepticism of mainstream econ is sufficient to show ‘Feynman integrity’”
This is true. However, it is also true in reverse:
Jamie: “a lot of mainstream econobloggers/economists think their skepticism of heterodox econ is sufficient to show ‘Feynman integrity’”
That doesn’t get us anywhere. Your statement is part of the problem. The question I am asking, and the question everyone should be asking, is how do we recognise “successful” science with respect to a subject like economics?
I am not asking why some people reject scientific evidence e.g. climate change. That is a different question. I am asking how intelligent people, who are open to scientific knowledge, should decide whether to believe a claim made by someone who says they are a scientist e.g. the claim that economic inflation forecasting is a success.
It seems to me that, to agree what is successful science in this context, we would need to agree some criteria and we would need to agree who should apply the criteria to determine the appropriate answer.
Regarding the criteria, I think Feynman integrity is one criterion but it is not the only one, or even the most important one.
However, let’s leave that for a minute and assume we have a set of criteria. Who should apply the criteria to determine the answer?
In practice, we each decide what we accept as scientific knowledge. I think that is the only realistic answer in an open society i.e. we agree as a society what is or is not science. For example, in the case of the Feynman integrity of your work, what matters is your readers’ view of whether you meet Feynman integrity – not your own view (although your readers are more likely to recognise your integrity if you demonstrate it in your words and actions – one of the reasons I read your blog is that you have an intellectual integrity that is missing in economists). More generally, physics passes my ‘society decides’ test, partly because it meets various criteria and partly because we are all educated in basic physics at school. However, economics fails – and not just because of Feynman integrity.
Scientists seem to think that they are the people who decide what is or isn’t science. However, many scientists throughout the ages have been delusional, so it makes no sense for society to accept something as science just because it is said by someone who self-identifies as a scientist. That’s just not good enough.
Also, many famous scientists have been ridiculed by other scientists (sometimes in peer reviews). For example, I saw a TV programme recently which said that Ludwig Boltzmann committed suicide in part because some of his peers didn’t accept his work on the existence of atoms. I seem to recall that Newton got bad peer reviews too. Also, Newton believed in alchemy and other nonsense so even the smartest people in history can be wrong about lots of things.
The problem in economics is that economists claim to be scientists and suggest that we should accept their policy advice on that basis. They reject criticism from others (including you) on the basis that they are scientists and others are not. In other words, the term “science” protects them from criticism regardless of the quality of their work.
Where I disagree with you is that you (as a physicist) want scientists to decide what is science by themselves when you are the scientist. However, you (as an ordinary member of society) want to be able to say that economics is not science when others claim that they are scientists. That’s not consistent.
Further, you have studied some aspects of economics whereas most people have not. Yet, everyone must make the judgement of whether to view economists as scientists, and which economists to believe when economists disagree.
I will write another comment later with some further criteria on recognising science where physics passes and economics fails.
Further to my previous comment, the best example of my conception of modern science is drug R&D in the pharmaceuticals industry. The R&D is carried out by large teams of very intelligent scientists. Nevertheless, society does not just agree to accept the findings of the scientists. Instead, drug companies are forced to carry out controlled trials and to submit their results to the FDA before they can sell their drugs to the public. Many drugs fall by the wayside during these trials. If drug companies wanted to do R&D without ever commercialising any products, scientists could adopt whatever standards they wanted and demonstrate whatever levels of integrity they wanted. However, if they want to commercialise their products, they must follow society’s approval rules.
ReplyDeleteIn contrast, academic macroeconomists seem to be under the misconception that society should just accept what they say, without any challenge, if they tell us that they are scientists. Society reacts mostly by ignoring macroeconomists (except when they can be used to justify whatever politicians want to do anyway), and will continue to do so until economists reform themselves, or until they are replaced by people with much greater professionalism and humility.
Regarding the criteria we should adopt for acknowledging economics as science, here are a few initial thoughts. However, there are many other aspects.
ReplyDeleteHistory is not science. There are too many disparate facts, at different scopes and scales, and too many interacting causes and effects, to develop a single scientific model of history. This is not because historians are dumb. It is because the subject is too complex and any scientific model, particularly a mathematical model, would have to over-simplify the story into a few arbitrary and measurable facts. There is an argument that macroeconomics is more like a specific slice of history than science. JM Keynes involved himself in key historical events such as advocating against German war reparations. These reparations led directly to hyperinflation, the rise of fascism and WW2. That seems to me to be more useful than over-analysing minor trends in inflation.
Any subject which involves morals and values is not science. For example, everyone recognises that physicists are specialists in the components of the atom and how to split them apart. However, no-one, including physicists, suggests that physicists are entitled to influence social policy decisions relating to nuclear power or nuclear weapons. Such decisions are based on social benefits, costs, risks and morals as well as on the scientific / technical feasibility. There is a very strong argument that economists cross this line all the time while claiming to be scientific.
In fact, any social decision that is, to some extent, based on weighing up disparate facts, is not science. That is because the weightings are always subjective even if the facts are objective. I would argue that all policy-making can be methodical but not scientific. Historical precedent can be used to promote what works and what doesn’t work without calling it science e.g. austerity doesn’t work because one man’s spending is another man’s income.
(Cont'd)
In as much as economics can be considered scientific, we might ask what type of science it is, and what are the relevant success criteria? A lot of the mainstream versus heterodox disagreement is in this area. The mainstream (and you) think that economics is like a social version of physics and the success criteria are mostly about forecasting accuracy. Many of the heterodox people think of economics as a social version of medicine and the success criteria relate to diagnosing, curing and ideally preventing economic ‘diseases’. The state of the art of mainstream modelling is that it works only when nothing much happens. When diseases strike, these models contribute nothing. Hence, the mainstream distrusts the heterodox because they don’t do the sort of forecasting that the mainstream likes to do. The heterodox distrusts the mainstream because they ignore many of the factors most likely to cause disruption e.g. the representative agent precludes any discussion of profit, private debt, inequality, herding behaviour. That doesn’t mean that these things always cause disruptions. However, it does question the usefulness of models which assume that these things do not exist. The first economist I heard making this type of argument against representative agent models was Joseph Stiglitz – Nobel Prize winner – not some random heterodox dude on the internet.
ReplyDeleteA prime role of academic scientists in all disciplines is to educate the rest of society. We might measure the success of this endeavour in economics by measuring economic literacy amongst policy-makers and the public. However, academic economists have no interest in this. In fact, they see their own complete failure to educate the public as a failure of the public, and many economists see it as further justification for moving even more control of economic policy from elected governments to unelected economists.
At a more mundane level, even beginner-level science involves a vocabulary which defines terms unambiguously and specifically. In economics, many terms are defined ambiguously (money, saving, the economy, investment, productivity) or generically. Monetary policy isn’t a policy – it is a generic group of disparate policies. It makes no scientific sense to talk about the impact of monetary policy when monetary policy means different things to different people at different times. The monetary / fiscal debates really concern political biases. Right-wing economists prefer monetary policies as they don’t redistribute money from the rich to the poor. Left-wingers prefer the opposite. It has nothing to do with science.
I don’t think anything I have said here would be controversial outside the economics profession. However, each point paints economics as failing to meet a standard that would warrant the label of a coherent science.