The (Lack of) Usefulness of Empirical Evidence in Economics

My argument with Jason Smith about the philosophy of economics has become rather spread out and hard to follow, so I thought I would compile my argument into one blog post.

First, let's start where I did with my previous post. Suppose there is some effect $y$ that is caused by $x,z,...$. This can be written as

$$y = f(x,z,...)$$

I hypothesize that $y = g(x,u)$. How do I test this hypothesis? I must isolate all other variables but $x$ or $u$ at one time and then change one of the independent variables ($x$ or $u$). If I do this, then I will be able to determine if $g(\bullet) = f(\bullet)$ and whether or not $x$ or $u$ causes $y$.

This can't [easily] be done in economics; it is simply not possible [probable that it would be possible] to conduct a generalizable experiment in a closed system -- i.e., only one independent variable can change at once. Take my previous example of the minimum wage. There are two issues with concluding that the econ 101 partial equilibrium model is wrong in the face of apparently conflicting empirical evidence: there is no way to determine whether or not the minimum wage increase caused employment to be lower than it otherwise would have been, since there can be no control experiment, and, if employment is $y$ and the minimum wage is $x$ from the example above, empirical evidence can only prove that the theory is incomplete; i.e. that it doesn't capture all of the possible causes of $y$. 

So, in this sense, it is possible to invalidate an economic model with data, but it doesn't help with anything; it doesn't even say whether the model is actually wrong, or simply lacking the full number of causes that effect $y$. The econ 101 model may be completely right about what happens to employment given a minimum wage increase in a closed system, but we can never know. It is for this reason that empirical evidence is not very valuable in economics, not because people have too strong of priors (even though this is frequently the case).

This also means, as Jason rightly acknowledges, that empirical accuracy is not something worth praising very highly in economics:

[*] You'll never nail down correlation vs causation 

Due to external factors (and lack of controlled experiments), this may be true. However, this is a reason not to praise an empirically successful model.

The problem is that the flip-side of this is that you can never actually determine whether or not the supposed cause and effect relationships in models are correct. This is why Popperian rejection of hypotheses is not [rarely] possible in economics; why empirical evidence cannot "falsify economic models."

Jason also challenges my claim that DSGE models are [should be] qualitative models:

John makes the case that it is the latter: DSGE models are qualitative models. I don't buy this. For one, they are way too complex to be a qualitative model.

I agree with this when it comes to, e.g., the NY Fed DSGE, but not for DSGE in general. Basic DSGE models, without all the bells and whistles that try to make them empirical, are indeed, in my opinion qualitative. They are simply internally consistent ways of diagnosing a single problem in economics. If I want to come up with a theory about whether it is better to have PAYGO pensions or American-style social security, I don't bother modeling monopolistic competition or sticky prices because all I want is a qualitative analysis.

I disapprove of even using the supposedly structural DSGE models of the economy such as Smets-Wouters or the NY Fed DSGE on the grounds that it should be apparent that the assumptions of these models deviate far from reality; they are in no way structural, so expecting them to be accurate models is almost ridiculous. Because of this, they should just be thrown out for their complexity.

When it comes down to it, Jason seems to immediately associate DSGE with big, clunky models with tons of dubious assumptions that somehow approximate reality whereas I associate DSGE with utility maximization, budget constraints, and rational expectations. Every DSGE model I bother using is, in my opinion, qualitative and that's how it should be.

So, to make one of my previously misunderstood arguments slightly more clear, does the great recession invalidate the basic 3-equation reduced form New Keynesian model? No, it only shows that the model is incomplete, since there is no way to disprove any of the supposed causal relationships in the model, especially since rational expectations are unobservable. The lack of empirical support for New Keynesian DSGE (as I define it) does not inherently mean that any of the hypothesized cause and effect relationships in the model are wrong. It could be this, or that the model lacks enough complexity to explain the data.

It's Not Over Yet

Jason Smith today in a blog post:

New Keynesian economics = Ignore empirical data

...

Information transfer economics = Use empirical data

I'd like to venture to explain why exactly this might be true. Start with basic Humean epistemology in which knowledge is the relationship between cause and effect and, for a given effect, there might be multiple causes. This can be expressed as
$$y = f(x,z,...)$$
where $y$ is the effect, and $x,z,...$ are the causes. How do you go about determining whether or not $x$ causes $y$? You create a closed system in which only $x$ and $y$ can change and then change $x$. If you then observe $y$, you may then (and only then) conclude that $x$ causes $y$. If, however, you cannot create a closed system, you cannot conclude whether or not your hypothesis that $x$ causes $y$ is true.

Of course, this is all really basic, but what does it have to do with New Keynesian economics (or, more generally, mainstream economics). I'll start with a general example. Consider the minimum wage. Basic partial equilibrium analysis suggests that increasing minimum wage will cause an increase in unemployment, but empirical research seems to not confirm this prediction. Case closed, Econ 101 is wrong and Info Econ 101 is right!

No. The only way for any empirical research to invalidate the partial equilibrium analysis with the minimum wage is if it were done in a closed system; i.e., the only two variables were employment and the minimum wage, which has not been the case in any study that I have seen referenced. In this sense, the causal relationship between the minimum wage and employment has not been falsified -- the data have only proven that the partial equilibrium model of labor markets is incomplete (i.e., it doesn't capture all of the causes of changes in unemployment), which we all already knew.

This same problem is captured by the failure of NK DSGE models to explain the Great Recession; the only conclusion that can be drawn from the empirical evidence is that the model is incomplete, which everyone already knew anyway, and therefore failed to capture all the possible causes of the Great Recession -- this has no bearing on the correctness of the rest of the model.

Since it is obvious that closed systems can't be dealt with very often in social science, especially if the hypothesis being tested (e.g., NK DSGE) is rather complex, it should be obvious that empirical evidence is not very useful for social science. That said, the appropriate conclusion is probably that all economic models are doomed to be quantitatively unsuccessful (and even if they are, that is not necessarily an indication that the model in question is correct in the sense that it correctly matches causes with effects), even if they do capture some of the correct cause and effect relationships, and we should only judge models based on their qualitative predictions, not their quantitative ones.

Choosing the Best Model For Each Context

In spite of perhaps attracting the wrath of Jason Smith, I think it is safe to say that economics is too complicated for there to be one generally applicable model of everything. Because of this, there is a veritable plethora of economic models available to the economic theorist. This simply leaves the question of which one to use in which circumstance.

Simon Wren-Lewis seems to think that economists should select between models in an ex-post manner -- that is, we should seen which model better represents the data and use that model from then on:

How do we know if most economic cycles are described by Real Business Cycles (RBC) or Keynesian dynamics. One big clue is layoffs: if employment is fall because workers are choosing not to work, we could have an RBC mechanism, but if workers are being laid off (and are deeply unhappy about is) this is more characteristic of a Keynesian downturn.

 The issue here is that we can only diagnose events after the fact, we cannot reasonably make predictions because of the impossibility of ex ante empirical validation: it is impossible to determine whether or not a recession is New Keynesian or if it is a Real Business Cycle before data are released.

This is why context-based validation of theory is superior to empirical validation in the case of economics. The context -- i.e. the sub-field of economics that is being studied -- should inform model choice almost entirely. If the field is business cycles, then the relevant model is a New Keynesian DSGE model and if the field is growth theory, then New Keynesian models are superfluous and should be tabled in favor of neoclassical models -- whose only difference from their New Keynesian counterparts is nominal rigidity, which is irrelevant over a time scale longer than a decade.

Predictions about the economy can now be made based on currently available information: it is possible to determine whether or not, e.g. financial frictions should be present in our business cycle model based on the current state of the economy: we knew by Q3 2008 that financial frictions were relevant, so we should have put them in a model if we were trying to predict the next few years.

Alternatively, the model I should choose to use depends on the kind of thought experiment I choose to embark on. Am I trying to compare PAYGO pensions with Social Security? If so, the obvious model to use is a simple OLG model without a labor-leisure trade-off or sticky prices. Choice of models is equivalent to choice of assumptions, at least when it comes to the DGE approach currently dominant in economics, and assumption choice depends entirely on the question being asked. Nominal rigidity is obviously relevant for business cycle theory, but completely useless when it comes to determining the level effect of a tax increase.

Hopefully this selection mechanism is specific enough to not be "basically feelings," as Jason Smith would suggest is the case for most of economics.