How (not) to justify running a QCA

Qualitative Comparative Analysis (QCA) is a method utilized by different disciplines in the social sciences and beyond, e.g., business economics and management. However, QCA users must still justify their choice of method more frequently than the users of other methods. Whatever the reason, it is actually not a bad thing to reflect upon the choice of a method because it should be suitable for answering our research question. (Although it can be annoying to explain the basics of QCA over and over again; nobody expects you to explain what OLS stands for and how an OLS regression works).

Among the different justifications for running a QCA, two are popular, but inappropriate: the medium-N argument and the look-the-results-are-complex argument. Both arguments refer to the basics and the soul of QCA. Originally, QCA was portrayed as a medium-N that allowed one to link case knowledge with the processing of more cases than can usually be handled in case study research. It is true that QCA can handle a medium number of cases. (I leave aside here that nobody knows where medium N starts and ends in terms of the number of cases.) However, one should not choose QCA because it can process a medium number of cases of, say, eight that are insufficient for a conventional statistical analysis (even bootstrapping reaches its limits here). If the application of a method is justified by the number of cases, the causal inferences that one can make are made dependent on how many cases are available. Put otherwise, saying you use QCA because the N is too small for a statistical analysis implies you would rely on statistics if you had more cases.

The causal inferences derived from a QCA and a statistical analysis are not the same, not even close (at least, that’s my view on this issue). Because of this discrepancy between QCA and a statistical analysis, one should carefully consider what method is able to deliver the answers that one needs given the research question and hypotheses at hand. For example, when a hypothesis reads “X is making Y more likely”, then QCA would be a bad choice regardless of the number of cases because this is not a set-relational statement. Similarly, when your hypothesis is set-relational, such as “If X, then Y” or you hold the ontological belief that causal relations are set-relational, then go with a QCA and disregard how many cases you have.

A second fallacious argument attempts to justify the choice of QCA via the results it produces. A QCA is done because of the belief that the causal relation of interest is complex (equifinality and conjunctural causation) and this belief is thought to be found confirmed when the empirical analysis includes multiple conjunctions. The method is justified by the results it produces. This is a bad argument for running a QCA because the best method to apply is the one that correctly captures the data-generating process (DGP, with the major problem, of course, being that we often don’t know what the DGP is). What is equally important is that, even when a method completely misses the actual DGP, we might still obtain intelligible results. In fact, this is behind the popular criticism “correlation is not causation” that is raised against statistical techniques because we can obtain significant estimates when the statistical model does not match the DGP. The same problem holds for QCA, meaning that one’s choice of QCA can never be validated by the results it produces.

So how should we justify the application of QCA? It is quite simple and by the textbook on research design: you should either hold a fundamental, ontological belief in causal relations being set-relational, or, on a lower level, follow theory and run a QCA when a hypotheses is couched in set-relational terms. For QCA, it thus holds what holds for any other method: ontological commitments and theory should reign supreme.


About ingorohlfing

I am Professor for Methods of Comparative Political Research at the Cologne Center for Comparative Politics at the University of Cologne ( My research interests are social science methods with an emphasis on case studies, multi-method research, and philosophy of science concerned with causation and causal inference. Substantively, I am working on party competition and parties as organizations.
This entry was posted in causal inference, QCA, qualitative, set relation, set theory, theory and tagged , , , . Bookmark the permalink.

6 Responses to How (not) to justify running a QCA

  1. Okay, I’ll agree on the first one. The second one, however, is very widespread in the literature (and I use it myself). Can you point me to any treatments in the QCA lit that elaborates this argument (or that explains in more detail what a DGP is)?

    • ingorohlfing says:

      The only one I know in the QCA domain who talks about DGPs is Baumgartner (and Epple):
      Baumgartner, Michael (2009): Inferring Causal Complexity. Sociological Methods & Research 38 (1): 71-101.
      Baumgartner, Michael (2013): Detecting Causal Chains in Small-N Data. Field Methods 25 (1): 3-24.
      Baumgartner, Michael and Ruedi Epple (2014): A Coincidence Analysis of a Causal Chain: The Swiss Minaret Vote. Sociological Methods & Research doi:10.1177/0049124113502948
      He does not talk specifically talk about DGPs, but, in my reading, he speaks about DGPs in fact. I am skeptical, though, that one needs his algorithm (Coincidence Analysis) for studying temporality because the Quine-McCluskey algorithm should work as well.
      If you want to understand the general problem with DGPs in QCA: Just imagine the common-cause problem and that QCA cannot distinguish between spurious and causal conditions in an observational setting (i.e., we confront the set-relational equivalent to the “correlation is not causation” charge).

  2. rickjdavies says:

    Incidentally, re ” the popular criticism “correlation is not causation”” , is this not tiresome in some respects? My preferred counter-phrasing these days is something along the lines of “Associations are a necessary but insufficient basis for a strong claim of causation” The other part that is needed is a casual mechanism that explains the association. When you are talking of multiple cases a causal mechanism on its own is not sufficient either, you also need evidence of consistent association. Yes?

    • ingorohlfing says:

      I agree with you, but I also do not argue to the contrary, do I? Actually, the valuable point you raise is worth a separate blog post.

  3. rickjdavies says:

    Hi Ingo
    No, you did not suggest the contrary. Mine was just an incidental comment about something that bugs me, I.e. a cliche recited all too often without any further thought.
    Yes, please do write a blog posting about it!

  4. Jim Robins says:

    Thank you. Simple statement that highlights the most neglected feature of almost all research in areas of management and business: the data generation process and data structure. This is the most difficult point to get across to students, colleagues, reviewers and editors, yet it should be the simplest.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.