Forecasting: IT versus DSGE

I was reading Simon Wren-Lewis's recent post, which took me to another one of his posts, which then linked to Noah Smith's May 2013 post on how poor DSGE models are at forecasting. Noah cites a paper by Edge and Gurkaynak (2011) [pdf] that gave some quantitative results comparing three different models (well, two models and a judgement call). Noah presents their conclusion:

But as Rochelle Edge and Refet Gurkaynak show in their seminal 2010 paper, even the best DSGE models have very low forecasting power.

Since this gave a relatively simple procedure (regressing forecasts with actual results), I thought I'd subject the IT model (the information equilibrium monetary model [1]) to an apples-to-apples comparison with the three models. The data is from the time period 1992 to 2006, and the models are the Smets-Wouters model (the best in class DSGE model), a Bayesian Vector Autoregression (BVAR), and the Fed "greenbook" forecast (i.e. the "judgement call" of the Fed).

Apples-to-apples is a relative term here for a couple reasons. First is the number of parameters. The DSGE model has 36 (17 of which are in the ARMA process for the stochastic inputs) and the BVAR model would have over 200 (7 variables with 4 lags). The greenbook "model" doesn't really have parameters per se (or possibly the individual forecasts from the FOMC are based on different models with different numbers of parameters). The IT model has 9, 4 of which are in the two AR(1) processes. It is vastly simpler [1].

The second reason is that the IT model's forecasting sweet spot is in the 5-15 year time frame. The IT model is about the long run trends. Edge and Gurkaynak are looking at the 1-6 quarter time frame -- i.e. the short term fluctuations.

However, despite the fact that it shouldn't hold up against these other forecasts of short run fluctuations, the scrappy IT model -- if I was to add a stochastic component to the interest rate (which I will show in a subsequent post [Ed. actually below]) -- is pretty much top dog in the average case. The DSGE model does a bit better than the IT model on RGDP growth further out, but is the worst at inflation. The Greenbook forecast does better at Q1 inflation, but that performance falls off rather quickly.

Note that even if the slope or intercept are off, a good R² indicates that the forecast could be adjusted with a linear transformation (much like many of us turn our oven to e.g. 360 °F instead of 350 °F to adjust for a constant bias of 10 °F) -- meaning the model is still useful.

Anyway, here are the table summaries of the performance for inflation, RGDP growth, and interest rates:

[Ed. this final one is updated with the one appearing at the bottom of this post.]

I also collected the R² values for stoplight charts for inflation and RGDP growth [Ed. plus the new interest rate graph from the update]:

Here is what that R² = 0.21 result looks like for RGDP growth for 1 quarter out

This is still a work in progress, but the simplest form of the IT model holds up against the best in class Smets-Wouters DSGE model. The IT model neglects the short run variation of the interest rate [Ed. update now includes] as well as the nominal shocks described by the labor supply. Anyway, I plan to update the interest rate model [Ed. added and updated] as well as do this comparison using the quantity theory of labor and capital IT model.

...

Update + 40 min

Here is that interest rate model comparison adding in an ARIMA process:

And the stoplight chart (added it above as well):

I do want to add that the FOMC Greenbook forecast is a bit unfair here -- because the FOMC sets the interest rates. They should be able to forecast the interest rate they set slightly more often than quarterly out 1Q no problem, right?

...

Footnotes

[1] The IT model I used here is

(d/dt) log NGDP = AR(1)
(d/dt) log MB = AR(1)

log r = c log NGDP/MB + b

log P = (k[NGDP, MB] - 1) log MB/m0 + log k[NGDP, MB] + log p0

k[NGDP, MB] = log(NGDP/c0)/log(MB/c0)

The parameters are c0, p0, m0, c, and b while two AR(1) processes have two parameters each for a total of nine. In a future post, I will add a third AR process to the interest rate model [Ed. added and updated].