Frequentist inference and process tracing: an oxymoron?

The methods literature on process tracing took a Bayesian turn in recent years. Bayesian inference, whereby you condition on evidence in order to update your prior confidence in a hypothesis, is presented as a mode of inference one should follow in process tracing (Bennett, Andrew (2008): Process-Tracing: A Bayesian Perspective. Box-Steffensmeier, Janet M., Henry Brady and David Collier (ed.):  Oxford Handbook of Political Methodology.  Oxford: Oxford University Press: 702-721), or that one does follow (e.g. Beach, Derek and Rasmus Brun Pedersen (2013): Process-Tracing Methods. Ann Arbor: University of Michigan Press).

Bayesianism makes a great deal of sense in general and in relation to process tracing because we constantly adjust our trust in a hypothesis in light of new evidence or (causal) process observations, as the now established term for data in qualitative research (Collier, David, Henry E. Brady and Jason Seawright (2004): Sources of Leverage in Causal Inference: Toward an Alternative View of Methodology. Brady, Henry E. and David Collier (ed.):  Rethinking Social Inquiry. Diverse Tools, Shared Standards.  Lanham: Rowman & Littlefield: 229-266); see here why I now prefer the term process observation to causal process observation). While Collier (Collier, David (2011): Understanding Process Tracing. PS: Political Science & Politics 44 (4): 823-830) does not refer to Bayesian reasoning in his discussion of process tracing, his Sherlock Holmes illustration in both the main text and the online appendix nicely illustrates how we/Holmes revise our/his confidence in hypotheses as we gather new evidence.

With regard to the development of process tracing methods, I think the Bayesian turn must be understood in light of King, Keohane, and Verba’s (1994: Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton: Princeton University Press) implicit but strong plea for frequentist reasoning in the social sciences. While I have much sympathy with Bayesian reasoning and notwithstanding that DSI sometimes misses the point of qualitative research, we should not lose sight of frequentism as an alternative mode of inference. I discuss this in chapter 1, 3 and 8 of my case study book, but I also turn a “Bayesian turn” in chapter in 8 in particular and focus more on this mode of inference at the expense of frequentism.

One might consider frequentism in inference to process tracing an oxymoron because it is so closely tied to causal inference in standard quantitative research (at this point, let’s leave aside the usefulness of frequentism for causal inference in quantitative research). However, one should clearly distinguish between the basic logic of both modes of inference – Bayesianism and frequentism – and the methods that rely on these modes. (For quantitative research, the dominant mode happened to be frequentism.)

But how should frequentism in process tracing work? Actually, I think it is very simple. Consider the Kennedy murder and the official line that Oswald was the only shooter (I am not implying anything here about the official line). If we follow this argument, Oswald must have made three shots within seven seconds (if I remember correctly, there is a scene on this in Oliver Stone’s JFK movie). The weapon that Oswald used was rather old-fashioned and required him to load bullets manually into the barrel of the rifle. Moreover, it is known that Oswald had to shoot through the branches of a tree; further, the car in which Kennedy was sitting was moving away from Oswald. Now, you can ask yourself: how likely is it that a not-so-well-trained shooter hits Kennedy with two bullets out of three within seven seconds when he has to load bullets into the rifle manually; there is a tree between him and his target; and the car is moving away from him? I would guess we would all agree it is very unlikely. It could have been like this, for sure, but it is pretty unlikely.

Congratulations, you have made a frequentist inference. Why is that? The null hypothesis is Oswald made all three shots. Given this hypothesis and all the facts we know about the shooting (type of rifle, unexperienced shooter, etc.), we deem it very unlikely that the null hypothesis is true and we would reject it. The important insight is: we are talking about qualitative inference and a handful of process observations. In contrast to what is sometimes argued, frequentist inference thus does not require quantitative methods or large samples and is not necessarily about mean causal effects or some other parameter. The actual realization of frequentist inference in process tracing shares elements with Bayesianism because we need to informally assess a conditional likelihood – p(E|H0). We do not specify the prior and do not calculate a posterior in frequentist inference, which points to the well-known weak spot of this mode: we discard a hypothesis with a certain level of confidence and without another substantively meaningful hypothesis stepping into its place. This implication of frequentist inference is all what you need when you aim to invalidate an argument, but it is not enough when you want to substantiate another hypothesis as the same time. But to repeat, the purpose of this post is not to make a case for or against frequentism or Bayesianism, but to point out that frequentism is possible and frequentist inference in process tracing is not an oxymoron.

Advertisements

About ingorohlfing

I am Professor for Political Science, Qualitative Methods at the Bremen International Graduate School of Social Sciences (BIGSSS, co-hosted by University of Bremen and Jacobs University) and Associate Editor of the American Political Science Review. My research interests are social science methods with an emphasis on case studies, multi-method research, and philosophy of science concerned with causation and causal inference. Substantively, I am working on party competition and parties as organizations.
This entry was posted in Bayesianism, case study, causal inference, causal mechanism, evidence, frequentism, qualitative and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s