I've been reading Noah Smith's latest post over and over and I'm not quite sure I get the point. My summary:
Economics wasn't very empirical, so theories used to be believed for theoretical reasons. Then along came data in the form of natural experiments, but these ruled out theories. Natural experiments have limited scope and don't tell you what the right theory is. This creates a philosophical crisis that manifests as an adversarial relationship between theory and data that will work itself out.
Sounds like an irrational three year old with fingers in ears saying La-la-la ... I can't hear you! when it's time to go to bed. That's pretty funny because the rational agent stuff tends to be what is being killed.
Actually, physics is dealing with the exact same problem (with dignity and grace). We have the standard model and general relativity (aka "the core theory"). There have been several natural experiments (supernovas telling us the expansion of the universe is accelerating, solar neutrino oscillations) that have 'proven' the core theory 'wrong' in ways that don't tell you what the right theory is. But there is no philosophical crisis and no adversarial relationship.
Noah chalks that up to a long tradition of empiricism in physics, but I disagree. It is the existence of a framework that says that even though the core theory is wrong about neutrino oscillations and the accelerating universe, it's still right about the things that it is right about. That's because of what Noah (in the prior post) says physicists call 'scope conditions' (that is fine although the first google reference is to sociology, and domain of validity and scale of the theory were terms more commonly used by this physicist). It's actually the funniest line of that prior post:
I have not seen economists spend much time thinking about domains of applicability (what physicists usually call "scope conditions"). But it's an important topic to think about.
Yes. That does sound like an important thing to think about. I have this theory. Under what conditions does it apply? Maybe we should look into it ...
Ya think?
At least "we should look into it" is better than Dani Rodrik's assertion that the scope conditions are just whatever the model assumed in order to model a specific effect. The IS-LM model is limited to the Great Depression. DSGE models are limited to the period of the Great Moderation in the US. A model of the lack of impact on unemployment of that minimum wage increase in New Jersey from 4.25 to 5.05 in 1992 Noah mentions in his post is restricted to that minimum wage increase from 4.25 to 5.05. In New Jersey. In 1992.
Anyway, physicists' so-called scope conditions mean that discovering neutrino oscillations or a positive cosmological constant doesn't burn through your theory like a building without firewalls or fire doors.
The econ 101 model of a minimum wage rise causing unemployment doesn't actually have any scope conditions. So that minimum wage increase in NJ burns down the econ 101 model of minimum wages. To the ground. Rodrik tries to put in a firewall and say that the natural experiment should only burn down the econ 101 theory when you go from 4.25 to 5.05 in NJ in 1992.
But that brings us to an even more important point. You can't interpret a natural experiment without a framework that produces scope conditions. How do you know if you've isolated an external factor if you don't know what the scale of the impact of that external factor is? The real answer is 'you can't', but economists have been trying to get around it with instrumental variables and structural estimation.
Structural estimation is the idea that you could make up a plausible argument for X to depend on Y but not Z. Instrumental variables is the idea that ... you can make up a plausible argument for X to depend on Y but not Z.
Anyway, those plausibility arguments are basically hand-waving scope conditions, but without a framework you have no idea what the size of the domain of validity is. As Noah says: "you have an epsilon-sized ball of knowledge, and no one tells you how large epsilon is."
The other way to get around the issue of data rejecting your theory and lack of scope conditions (thus the data burning your entire theory down) is to relax your definition of rejection. One way of doing that is called calibration. And all of these ran into each other on twitter today:
Basically we have this ...
Problem: Data rejects our theory and without the firewalls of scope conditions, it burns the entire theory down
Solution 1: Scope conditions are limited to original purpose of theory (Rodrik)
Solution 2: Hand-waving about scope conditions with instrumental variables
Solution 3: Relax definition of "rejects" with calibration
Let it burn was apparently not an option.
...
PS I'm sure you want to ask about the scope conditions (domain of validity) of the information equilibrium models. Well, the scope of any particular model consists of its equilibrium relationships between its process variables. If data rejects the market information equilibrium relationship $A \rightleftarrows B$, then that relationship is rejected. If the model is made up of more than one relationship, but depends on the rejected relationship, then the model is rejected. That should make intuitive sense: either information flows between $A$ and $B$ or it doesn't. And if part of your model requires information to flow between $A$ and $B$ and it doesn't, then your model is wrong.
PPS This post started out with my personal opinion that if a particular statistical method matters in rejecting or accepting your model, then your model probably doesn't tell us much.
This is very interesting. I spent many years in management consulting. One of my observations relates to the difference between novice consultants e.g. recent graduates and experienced consultants.
ReplyDeleteNovice consultants tend to focus on themselves and the techniques in which they are experts. As a result, they tend to see every assignment as the same and they often jump quickly to conclusions.
Experienced consultants, on the other hand, tend to focus on each client and his problem. They start by assuming that the current problem is unique. After some investigation, they begin to recognise patterns with previous assignments. The current assignment X has a lot in common with assignment C but with some flavour of assignment F, some of the people problems of assignment P and the technical computer problems of assignment T.
In the terms of your post, novices start by assuming a broad domain of applicability of a specific technique or solution and require evidence (normally from someone else bashing them over the head) to change that assumption. Experienced consultants start by assuming a unique situation and then using evidence to widen the relevant domain.
The difference between the novices and the experienced consultants is that the experienced consultants have developed a complex decision tree (mostly undocumented) to assess the scope conditions that apply to each problem. They focus on what is different about a problem at least as much as what is the same. They look at the problem from several different perspectives and they listen keenly to the views of people who have those different perspectives. They don’t talk about theories which are universally true. Rather, they talk about rules of thumb which apply in certain situations. As a result, they have a mental agility which the novices lack.
In this respect at least, most economists behave like novice consultants. However, economists have an additional problem in that they don’t listen to anyone else apart from other economists who think in the same way, so they never change.
Hi Jamie,
DeleteI see some similarities here with the "fox and hedgehog" theory (your novices are like hedgehog's who know one big thing well, and your managers are like foxes who are more generalists).
This actually tends to be pretty typical in fields of research from my experience. For example: I am applying information theory (important in signal processing) to economics :)
https://en.m.wikipedia.org/wiki/The_Fox_and_the_Cat_(fable)
DeleteI don’t think that the fox and hedgehog analogy is relevant to my point. Experienced consultants know the one big thing that novices know. However, they also know many other things. They are not generalist managers. The point is more that ‘if your only tool is a hammer you tend to see everything in terms of nails’. The novices have only one tool. The experienced consultants have many tools and the wisdom about how to use them.
DeleteRegarding your information transfer techniques, you are looking to replace one forecasting technique with another. That’s fine but is a different point. There is always room to improve specific techniques. I read your blog because there is not enough innovative thinking in economics. It’s also interesting to read your perspective on mainstream economics and compare it with mine. However, the analogy with my point would be that macroeconomic forecasting is only one part of economics.
One of the oddest aspects of economics is that physicists, chemists, engineers, entrepreneurs and even politicians have contributed much more to our economic prosperity than economists. For example, the inventors of the washing machine and the pill freed women from the home and doubled the effective workforce. Future economic prosperity will probably depend on further disruptive innovation. However, I doubt that such innovation is forecastable via any mathematical technique. It will arise from the randomness in your models rather than from the parameters you are modelling.
Other changes may arise from factors such as demographics, climate change and wars. The biggest economic issue in Europe at the moment is the mass migration of people from the Middle East and North Africa. That wasn’t in any economic forecast even a year ago.
Finally, mathematical forecasting is limited by our ability to measure things. When I buy a book over the internet I no longer have to spend an hour travelling to and from town to visit a physical bookshop. That hour of time is a benefit of internet shopping but it is not measured anywhere. Similarly, if I buy a faster computer, a more reliable car or better quality food, but pay the same price as before, this quality improvement is not measured either. These innovative changes take place at the same time as economists solemnly tell us that we have a ‘productivity problem’ due to a lack of innovation. Maybe economists have a measurement problem?
Hi Jamie,
DeleteYou said I'm trying to replace one forecasting method with another -- that is not exactly what I'm doing. There are a few methods of determining whether a model is correct. One way involves statistical tests (instrumental variables, Granger causality, etc). Another way is a precise retro-diction with a fairly convincing theoretical model (Einstein's calculation of the precession of Mercury falls in this category). However another way is to predict future data (conditional on model inputs). This last one is the reason I do the forecasts. I'm trying to replace (or maybe just augment) a set of analytical tools with a new one.
Another reason for the forecasts is that they embody one of the key differences from the mainstream approach. Many economists think you can't predict recessions and the like because it boils down to predicting human behavior ... which people tend to believe can't be predicted. I want to show that the law of large numbers leads to the averaging out of human behavior and a fairly predictable macroeconomic system.
The forecasts I make are attempts to show macro is predictable. Plus the forecasts I make are actually pretty boring. They all look like linear extrapolations!
Two other things:
The migration into Europe is a big deal in the news, but tends to be somewhat of a wash in terms of economic impact because it's only a few million people into a region with 800 million people. That's less only on the order of 0.1%, which is on the order of the error in the measurement of NGDP.
The fall in prices is measured and so-called 'hedonic' adjustments are made to CPI. See e.g. here for computers the US:
https://research.stlouisfed.org/fred2/series/CUSR0000SEEE01
Ordering the book online is measured in higher productivity of book producers, and having you travel to your bookstore is actually less efficient than the economies of scale available from moving the books to you in a "book bus" (delivery van). That's one less car on the road (reducing other's wait time in traffic), and you've saved a bit of fuel.
Is all of this 'good'? Not really -- the bookshop is put out of business, and we use a lot more packaging that fills up landfills.
Or, laws in economics are few in number, empirically verifiable yet probabilistic, and should be based off of long time series. Economics needs better and longer empirical time series, and these will arrive "over time".
ReplyDeleteThough, if you can't get the profession to accept that 3 month t-bill rates are mostly a function of monetary base / NGDP back to the formation of the Fed, the problem may be non-tractable, i.e political in nature -- economics may simply be politics by other means.
I agree that it is entirely possible there are very few empirical regularities -- and such a view would mean you should treat economics like history instead of a science.
DeleteJason, a very interesting post.
ReplyDeleteThanks Tom!
DeleteJason, I'm curious: I'm not really a Twitterer, nor do I have a desire to be one; I know basically how to navigate around and see the whole conversation, but that's about it. If I'm not mistaken, it appears in your screen capture above that you were about to respond to all of them, yet I didn't see a tweet from you in this conversation when I looked it up in Noah's conversations.
ReplyDelete1. Did you chime in on this?
2. Do you generally chime in? Does Noah (in particular) ever respond? I'm curious since he so often erases your comments on his blog.
3. Would we see your tweets there if you did?... do any of them block you from their twitter conversations?
It always shows a reply space if you expand a conversation if you're on twitter. I didn't chime in there and I'm sure Noah has muted me already (you can't block people on Twitter as far as I know, you can only mute them so they don't show up in your own feed).
DeleteAs there were already 5 people in the conversation, adding another would probably take up all of the 140 characters anyway.
It'd be nice to hear Weird Al do a song based on this post to the tune of "We built this city."
ReplyDeleteIt would be and that was the theory behind the title. On Twitter I used my second choice "burning down the theory".
DeleteIn a strange confluence of events, Noah put Weird Al at the top of his post that bore striking similarities to my post that references this post ...
Deletehttp://noahpinionblog.blogspot.com/2016/01/situationalism-in-economic-policymaking.html
The lattice of coincidences ...