About two weeks ago, COMPASSS issued a Statement on Rejecting Article Submissions because of QCA Solution Type. In short, the reasoning was that methodological work on QCA is developing and that reviewers and editors should not judge empirical work based on whether one particular solution type is interpreted as causal. (Disclosure: I am a member of the COMPASSS advisory board, but was not involved in this statement in any way.) Dimiter Toshkov picked up on the statement and wrote an interesting blog post on this. Dimiter’s blog then hosted a response by Eva Thomann that led to a say, heated, exchange of arguments in the comments section.
With some delay, I want to raise two points about the debate that largely abstracts from the question of which type of QCA solution (conservative, intermediate, parsimonious) is the “correct” one and can be interpreted as causal. I have my viewpoint on this, but my remarks are more general.
The bar for public statements should be high
It is not without precedent that professional associations, which COMPASSS is for me, publish statements on various matters (such as the ASA statement on p-values). I believe that associations have the right to issue statements and should do so. However, the bar should be set high because, otherwise, there may be interference with the scientific process, with the association tending to take sides in a debate (not necessarily, though). Most of the time, the association should be a bystander and not an involved party.
The question of how to review QCA submissions to journals certainly is an important topic that potentially qualifies for a statement. In light of Eva Thomann’s post, however, my impression is that the bar has been set too low. She writes that the COMPASSS Steering Committee (SC) had no information about how often papers have been rejected based on arguments that a specific solution type can(not) be interpreted as causal. One might read this such that the issue is so important to the SC that the frequency of its occurrence does not matter. However, I am far from convinced that frequency should be irrelevant.
If we take this as the precedent of COMPASSS statements, it would mean that the SC could put out statements at a high rate because I am sure that reviews of QCA articles often contain arguments the members of the SC consider dubious or wrong, some of which strike me as being worse than the issue at stake here. As many QCA researchers surely can tell, one still has to manage reviews that flatly deny any value of QCA. One might respond that the SC saw a problem emerging and wanted to intervene early on, but I do not find this convincing. As a professional association, COMPASSS includes respected QCA researchers and experts, but it would still mean that the only professional association in the field would intervene and, further, would need to frequently intervene in the future because reviewer arguments that it deems ill-suited will continue to be made.
Most studies interpret one solution type as causal
The second point concerns the internal coherence of the argument, which dips into the question of which of the three solution types is causal. The COMPASSS statement does not say anything about a specific type, but readers with some knowledge of the field of QCA can reasonably guess that it is about Baumgartner’s and Thiem’s work and argument that only the parsimonious solution should be interpreted as causal[1, 2]. If you follow their reasoning, the question is: why should reviewers not be allowed to criticize a study if it interprets another type as causal? I agree that the Baumgartner/Thiem article is “only” one study and that more work is needed on this. However, this is not a sufficient reason to discount the work that has been done and it should mean something. If this was the standard, we could pretty much stop doing research because one could always say “I don’t follow this reasoning because there is not enough research about it.” (We would have to define what “enough” means, but this is probably even more subjective than the question of QCA solution types.)
Now to the internal consistency of the argument: There seems to be consensus that only one of the three solution types can be interpreted as causal (this should be consensus). This means that as soon as an empirical researcher interprets any solution type as causal (as many do), they are implicitly saying that the other two types are not causal. You might, like Eva, say other types are insightful or useful, but they cannot be causal (another point on which Eva seems to agree). The Baumgartner/Thiem argument that only one solution type is causal is therefore not new at all (the content is, I’d say, because most empirical researchers now seem to prefer the intermediate solution).
This means you can criticize reviewers for rejecting papers because they interpret the wrong type as causal, “wrong” from the reviewer’s perspective. For the sake of consistency, you should then also criticize empirical researchers if they interpret only one solution type as causal because they are designating the other solution types as non-causal. The liberal attitude that shines through the COMPASSS statement, stating that any solution could be interpreted as causal, is not sustainable because the causal interpretation of any solution type is exclusive. Either reviewers should be allowed to settle on the causal interpretation of one solution type, or reviewers and authors should not interpret any type as causal.
A way out
Avoiding causal terminology might not be the worst of ideas, but I think there is a middle way between rejecting papers because of the wrong solution type and not using any causal terminology per se. I agree with Dimiter and Eva that rejecting a paper based solely on the question of solution types should be discouraged. This has nothing to do with the topic, but rather the understanding of what constitutes good reviewer practice. If you think the paper is good, but also that the wrong solution type is interpreted as causal, then ask the author to clarify their position on why this solution type is taken as causal. Perhaps the author has a good answer for why this type and not another and the issue can be settled. Or the author starts thinking, accepts that this solution might not be interpreted as causal, and switches to non-causal terminology. I believe that most QCA researchers want to make causal claims, but we should keep in mind that good and interesting research can also be non-causal.