More often than one might expect, television series and films offer excellent illustrations of methodological and methods-related arguments (which is worth a blog post of its own). When I was working on my paper on comparative hypothesis testing in process tracing, I was watching the first season of the terrific TV series, Homeland. As it turned out, a very important element of episode 1.7 (The Weekend) exemplified the deficiency of a central argument in the process-tracing literature that is at the heart of my paper.
For those readers who haven’t watched Homeland, some contextual information is necessary for understanding the point (spoiler alert, I should add). One central figure in Homeland is Adrian Brody, a Marine who was captured during the Iraq war and held captive by al-Qaeda. Brody had been part of a sniper team with Tom Walker, who was also captured and was apparently killed while being held prisoner.
After eight years of captivity, Brody is rescued, returns to the United States as a hero and starts a political career in Washington. Shortly before Brody is rescued, Carrie Matthews, a CIA agent working in the Middle East, receives information that an American prisoner of war was turned while being held hostage. She doesn’t know who it is, but believes that it must be Brody. Large parts of the first season circle around the question of whether she is right and her attempts to show that Brody is planning a terrorist attack in the US.
Now, at the end of episode 1.7, the CIA has gathered conclusive information that Walker is alive and seeks to carry out a terrorist attack in the US. The CIA concludes that Walker must be the one that was turned and not Brody. The following episodes show their conclusions are incorrect; in fact, Walker and Brody were turned by Al-Qaeda and a planned attack on the vice president of the USA involves them both.
This is the part of the Homeland plot that meets with a deficient argument in the process-tracing literature on comparative hypothesis-testing. The argument is tied to the 2×2 typology of hypothesis tests that became central in recent years and which I do not consider in detail here (e.g., Collier has an article (ungated) on this in PS&P and an appendix that includes a discussion of the typology and illustration with a Sherlock Holmes case.) One type of test is the so-called doubly-decisive test, one that is marked by high uniqueness and high certainty. This means we are testing an observable implication of a working hypothesis that only follows from it (high uniqueness) and for which theory tells us that it is highly likely that we find confirming evidence (high certainty). Such a test is argued to be doubly decisive because confirming evidence for the working hypothesis automatically invalidates competing hypotheses.
In my paper, I show that this is not necessarily true because an implication can be unique for one hypothesis and unrelated to another. As a matter of fact, this is the scenario in Homeland. Let’s formulate the two hypotheses “Brody was turned and is the assassin” and “Walker was turned and is the assassin”. At the end of episode 1.7, the CIA has information that leaves little doubt that Walker is planning a terrorist attack. Does this mean that Brody cannot be an assassin as well? No, it does not.
When simply examining the two hypotheses, it becomes clear that one does not rule out the other. Furthermore, the CIA does not have credible information suggesting that there is only one assassin. Carrie receives information about a turned prisoner in episode 1.1, but it is of unknown quality and does not strongly suggest that it is only one (or that any prisoner was turned in the first place). Sure, the CIA also does not have information that there are two assassins, but ruling this out at the end of episode 1.7 is premature. Confirming evidence on the Walker-hypothesis does not invalidate the Brody-hypothesis because the absence of evidence pointing to Brody does not mean that he might not be an assassin as well. (Whether the absence of evidence for a hypothesis is evidence of the absence of what the hypothesis stipulates, i.e., that it is wrong, is another interesting topic.)
For process-tracing researchers and people engaged in comparative hypothesis testing in general, the take-home lesson that I develop in my paper is that one should distinguish between unique and contradictory implications. A unique implication is attached to one hypothesis and unrelated to others. A contradictory implication is at stake when two hypotheses yield exactly the opposite expectation. When one person predicts the world is ending tomorrow and another one says it will not, only one of the two can be correct.
I haven’t done a systematic review of hypotheses in the social sciences, but there is good reason to believe that most hypotheses are not contradictory but instead yield unique and complementary implications (or non-unique implications). In order to incorporate the distinction between unique and contradictory implications in a typology of hypothesis tests, I develop a 2x2x2 typology at the end of my paper. It certainly is less handy than the 2×2 typology, but it does disentangle the two dimensions that are lumped together under the rubric of ‘uniqueness’ in the 2×2 version. The expanded typology would also have helped the CIA avoid drawing the wrong conclusion at the end of episode 1.7, but it also would have taken some of the edge away from Homeland’s fascinating plot.