Barriers to entry in the quantitative parable industry

John Cochrane writes about his view of Ricardo Reis' paper defending macroeconomics. Noah Smith previously wrote about it here, and I commented on that post here. I also based my short play/Socratic dialogue in part on Reis' paper.

Cochrane provides another data point lending credence to my thesis that economists don't quite know what mathematical theory is for (see e.g. here, here, here, here, or here). The specific version that Cochrane starts off with is similar to Dani Rodrik's view (that I discuss here):

Many [critics of macroeconomics] bemoan the simplifications of economic models, not recognizing that good economic models are quantitative parables, or abstract art. Models are best when they isolate a specific mechanism in a transparent way.

Nope. 

I put the proper scientific process in picture form in this post:

Cochrane's quantitative parables go on the right hand side of this. The basic idea is that the "simple clean theoretical model" (quantitative parable, toy model) is something that comes after you've had some empirical success. In words, I'd describe the process like this:

1. Observe one or more empirical regularities [1]
2. Describe one or more empirical regularities with models (built using realistic or unrealistic assumptions ‒ your choice!)
3. Observe any errors in theoretical descriptions
4. Revise theoretical descriptions
5. Repeat/continue process
6. Collect empirical regularities and their successful models into theoretical framework
7. Test scope of theoretical framework with data, derive novel results from framework, teach the framework using toy models demonstrating framework principles, isolate mechanisms theoretically using framework for empirical study, support unrealistic assumptions using scope of the framework, and/or revise framework
8. Collect issues with theoretical framework and new empirical discoveries into new framework
9. Repeat/continue process

And this process is followed by all sciences. At any time, people can be working on steps 1 through 4. Steps 6 and 7 come along in a more mature science like physics, chemistry and biology. As far as I know, only physics has gone through step 8 multiple times (with relativity, quantum mechanics, and quantum field theory), but happy to be wrong about that. Economics has gotten to step 5 (Noah Smith has a list of some of the successes, and makes an excellent case for only getting to step 5 in this post [2]).

The key point is that the Cochrane's quantitative parables and models that theoretically isolate mechanisms come much later in the process than economics has progressed. It usually takes a genius or otherwise seminal figure to do step 6. Newton, Einstein, Noether, and Heisenberg in physics. Darwin in evolutionary biology. Hutton and Wegener in geology. Snow in epidemiology. Mendeleev in chemistry. However, we cannot use the converse: the existence of famous/seminal figures does not imply they developed a theoretical framework. And it is also important to stress the empirical piece. Wegener wasn't the first person to posit continental drift, but was the first to include e.g. fossil evidence. Economics in contrast is rife with theoretical frameworks posited by famous economists without comparison to empirical data. Even Keynes and Adam Smith appeal to philosophy and argument rather than data ‒ Keynes famously saying "But it is of the essence of a model that one does not fill in real values for the variable functions."

As an aside, I think I might have finally arrived at a really good way to describe what critics are saying when they say economists have "physics envy": economists think they have a theoretical framework. It explains why economists have papers with overly-mathematical symbols given how poorly the theory describes the data. It explains why they feel they can make unrealistic assumptions even when the result doesn't describe any data. It explains why they think the words "toy model" should even be in their vocabulary. Economists think they are in step 7, but really they are still cycling through step 5.

That seems like a bad start for Cochrane's piece, but the next thing he says is right on:

Critics [of macroeconomics] usually conclude that we need to add the author's favorite ingredients ‒ psychology, sociology, autonomous agent models, heterogeneity, learning behavior, irrational expectations, and on and on ‒ stir the big pot, and somehow great insights will surely come.

There is far too much "we should include X" in economics (including heterodox and non-economists). The only scientific way to say "we should include X" is to say:

APPROPRIATE: We included X and it improved the theoretical description of empirical data, therefore we should include X.

The unfortunate thing that some scientists do however is this:

INAPPROPRIATE: "We included X and it improved the theoretical description of empirical data in our field of science, therefore we should include X in economics.

It would be fine if it improved the theoretical description of empirical data in economics, but theory by analogy only goes so far. I try to call this out as much as possible when I can (biology, evolutionary biology, complexity theory).

I think this is the unfortunate consequence of the history of theoretical frameworks without reference to empirical data in economics. If you don't discipline theory in your field with data, anyone thinks they can come up with a theory because they reckon they know a bit about how humans think about money being a human who has thought about money.

And that's really a bigger takeaway. Many macroeconomists are frustrated with the criticism and the economic ideas coming from people without PhDs in economics. But they set up their field to essentially be armchair mathematical philosophy, and the barriers to entry for armchair mathematical philosophy are extremely low. A high school education is probably more than sufficient. I think that explains the existence of the econoblogosphere. It's really not the same in physics, mathematics, or signal processing from my experience (using examples of my favorites) ‒ those fields all tend to have experts and PhD students sharing ideas.

I think I'll leave it there because this forms a nice single thesis. There is a process to science and mathematical theory has a specific place. However, theory without comparison data or without an empirically successful framework is just armchair mathematical philosophy. There are few barriers to entry in armchair philosophy. For toy models isolating mechanisms however, empirical success is the barrier to entry. 

...

PS I did get a chuckle out of this:

Others bemoan "too much math" in economics, a feeling that seldom comes from people who understand the math.

I think that is sometimes true. 

However, I personally do understand the math. My opinion is that 1) the level of mathematical complexity of macroeconomic models far outstrips the limited amount of empirical data, and 2) the level of mathematical rigor far outstrips the accuracy of those models.

PPS I imagine some people will call me out for hypocrisy regarding the "INAPPROPRIATE" statement above. Aren't you, physicist, saying things from physics should be included in macroeconomic theories?

I would remind those people that I am not forgetting the clause about comparing to empirical data. I am not saying just "economists should use information theory", I am saying the information theory nicely encapsulates several empirical successes. I also test the theory with forecasts.

...

Footnotes:

[1] Yes, you always need to have some sort of underlying model in order to collect and understand data. However this is not that onerous of a requirement as even philosophy can frequently fill this role. An example is Hume's uniformity of nature. The implicit model behind the unemployment rate is that there are some people who do things in exchange for money and some people looking for opportunities to do things in exchange for money. 

Sometimes people make this observation out to be much more consequential than it actually is. Sure, it's important in the case of understanding what fluctuations of NGDP are important (i.e. what is a recession). However a lot of empirical data in economics is "counting things" where the implicit theory is not much more complex than the one that governs "Which recycling bin does this go in?"

[2] Added 4pm. In contrast to Noah's opinion, though, I wouldn't say that makes economics "not quite" science. It's true that most of the "hard" sciences made it to step 7, but they once were in steps 1 through 4. Physics didn't suddenly become "science" after Newton. It just became a science with a theoretical framework. Biology didn't become a science after Darwin, it was just more about empirical data before.

I think macroeconomics and microeconomics prematurely thought it had a framework with rational utility maximization and some ideas from Keynes (the neoclassical synthesis). Economists started to evade the discipline of data. In a sense, what Noah is calling the "credibility revolution" is the slow realization that the purported frameworks like rational utility maximization and DSGE failed some of the tests in step 7 and so macro is falling back to steps 1-4. This is a good development.