A starry-eyed aside on methodology

Mark Thoma and Simon Wren-Lewis recently had some discussion of economic methodology, and during the course of writing the past several posts, methodology has been on my mind. I hinted at it here, here, here and here.

Now I'm not here to say "you're doing it wrong" (which would anger Chris House), but I do think economists, econobloggers and econoblogocommenters should have far more skepticism of their conclusions than they seem to have -- especially when it comes to "natural experiments" in macroeconomics (such and such country did X and the result was Y). I also think Stephen Dubner should be legally required to say "This may well be bullshit, but ... " before everything he says.

I am going to illustrate this with an elaborate analogy. Macroeconomics has a difficult time performing reproducible experiments which means that it is primarily an observational science. Just like astronomy -- and that will be the source of the analogy.

The astronomy story

In astronomy, there is an empirical diagram called an HR diagram. It plots stars' luminosity (L) versus their color index (B-V). It makes somewhat of a line (for main sequence stars):

The theoretical framework at the time believed stars converted gravitational energy into thermal radiation as the star shrank in size. However this led to the sun being only millions of years old instead of billions.

Some stars do fluctuate in luminosity and you might think that would be an excellent place to look for a natural experiment. However, the correlation of log L with the color index is -0.86. These variables are highly correlated so "natural experiments" (e.g. watching a star fluctuate in luminosity) are practically useless (you can't be certain of effects you don't know about or haven't controlled for -- you have to assume your model is right). The only resort is to accumulate data from a bunch of different stars that are in different places on the diagram and try to come up with a theory that encompasses the data based on something besides astronomy.

Even though astronomers didn't know where stars got their energy, Eddington managed to explain the basic physics of how the diagram comes about in the 1920s. Thermodynamics says that stellar temperature is related to luminosity, but the required energy is far beyond gravitational or chemical energy. Thermonuclear reactions are required.

The economics story

In macroeconomics, there is an observed empirical relationship between the money supply and the price level. There is a basic theory that says that growth in the price level in the long run is equal to the growth in base money. However this had a problem explaining how Japan's base could grow without causing inflation.

Economic fluctuations do occur, so many economists try to use them as natural experiment. However, the correlation of the price level with the base (or even M2) is 0.95. These variables are highly correlated so "natural experiments" are practically useless (you can't be certain of effects you don't know about or haven't controlled for -- you have to assume your model is right). The only resort is to accumulate data from a bunch of economies and try to come up with a theory that encompasses the data based on something besides macroeconomics. How about this one?

The key to figuring out the HR diagram was applying the right theoretical framework (thermodynamics). But the interpretation of the data is highly framework dependent -- remember the gravitational theory of stellar radiation? Supply and demand, expectations-based theories, DSGE models. These are the frameworks. Looking at macro data and saying fluctuation X is due to cause Y is dependent on those frameworks. Fiscal policy was ineffective in the US because of monetary offset ... in a monetarist AD/AS model. The US is currently stuck in a liquidity trap so monetary policy is ineffective ... in an IS-LM framework. Macro data isn't going to resolve the ambiguities -- it will just add to existing correlations on which those models are based.

As a side note: if money is analogous to the energy source of stars, economics is still in a state of not exactly knowing what that source is, just like Eddington. Not knowing what money is shouldn't be an impasse!

Mildly related update (5/24/2014): There's a big dust up on the political economy side of things with this comment on the data in Piketty's recent book. Overall, I'd like to say that if you look at the blue lines in the later graphs, they're broadly consistent with the red lines (see e.g. Matt Yglesias) so the first several points don't seem to matter that much. But I would like to address one comment made by Giles:

As I have noted, even with heroic assumptions, it is not possible to say anything much about the top 10 per cent share between 1870 and 1960, as the data for the US simply does not exist.

This is where the HR diagram in the post above comes in. Part of the reason we know about the our sun's evolution is that there are hundreds of stars out there in that are similar to our sun at different times in their life cycle. We have several countries in the world out there in economic conditions broadly similar to the US in those earlier times and in most cases, pre-welfare state nations today tend to have Gini coefficients of 0.5-0.6 or higher. That is not an heroic assumption, and Piketty's results are consistent with that back of the envelope estimate.