Gap-filling, puzzle-solving and replication in empirical research

A common justification of empirical research is that a gap in the literature needs to be filled. I have seen this reasoning in all kinds of papers and presentations by Master students, PhD researchers, Postdocs and professors (probably the less, the more advanced the academic career is). It is not wrong to justify a study with gap-filling, but it is also does not seem to be the most exciting argument because the natural question is: Is the gap worth filling? In my limited experience, parts of the audience, at least those who ask this question, tend to be skeptical that the question can be answered affirmatively.

A common response to gap-filling justifications is to focus on a different question and to solve a puzzle. Puzzle-solving has always been a popular research strategy (such as here), but my feeling is that it has grown more popular over the last years (Day and Koivu give an excellent teaching-focused intro). Identiying and explaining a puzzle certainly has merit and I do not aim to argue against it here. However, I think there is a downside to the overemphasis of puzzle-solving and to discouring researchers from filling gaps (to be reframed in a second as something else).

The issue with puzzle-solving is that one has to build on existing theoretical arguments that (supposedly) work in general, but not for all cases that then constitute the puzzle. Theoretically, we learn something new from puzzle-solving, which is certainly positive. However, this also means that puzzle-solving puts a premium on theoretical originality and tries to push a theory and its hypotheses forward by improving then and explaining the anomalies.

Puzzle-solving is fine as one element in the researcher’s toolbox if it does not crowd out alternative research goals and strategies. One alternative, which is still centered on the idea of theoretical originality, is generic exploratory research of new empirical phenomena such as the decarbonization of energy production.

A second alternative is gap-filling reframed as replication. Gap-filling studies are not all the same, but in my experience they are commonly justified with the argument that a selected hypothesis hasn’t been tested yet for a specific country, or region, or institution, or policy field, or period of time or what other dimension with variation across cases one can think of. I do not remember a study that framed gap-filling as a replication attempt, but the two are equivalent if gap-filling means taking an existing argument and testing it in a different empirical setting and with different data than the original study.

For example, imagine a study tested a hypothesis of lobby group influence on policy making by using cases of distributive policy making. A gap-filling replication study could test the same hypotheses using cases of redistributive policy-making (assuming the hypothesis is supposed to explain lobby influence on redistributive policy-making). The example indicates that gap-filling is not verification in the narrow sense where one exactly repeats the original study. The gap-filling analysis varies the type of policy compared to the original analysis, which means that gap-filling also has an element of theoretical originality. In comparison with puzzle-solving, it might be less original because one “only” tests an existing argument in a different empirical setting instead of developing the theoretical argument further.

The reframing of gap-filling research as replication helps in clarifying what and how much gap-filling adds to the existing state of research. Regarding the “what”-part, replications are valuable in themselves because no single original study is decisive and needs to be verified. The degree to which a gap-filling study has added value depends on the strength of evidence that has been collected in previous research. The more existing evidence speaks for (or against) a hypothesis, the less value I see in adding one more gap-filling/replication study. The more ambiguous the existing evidence is and the fewer studies have been done on a research question, the more a gap-filling study adds to our knowledge.

Altogether, gap-filling research understood as replication is neither interesting nor uninteresting per se. Some gaps are worth filling, while others are not and can be left as they are. Whether a gap-filling study is of the first type or the second can only be determined by extending the perspective and discussing how much it adds to the existing body of research on the same topic.

About ingorohlfing

I am Professor for Methods of Comparative Political Research at the Cologne Center for Comparative Politics at the University of Cologne (http://cccp.uni-koeln.de). My research interests are social science methods with an emphasis on case studies, multi-method research, and philosophy of science concerned with causation and causal inference. Substantively, I am working on party competition and parties as organizations.
This entry was posted in generalization, replication, research question, theory and tagged , , . Bookmark the permalink.