On September 27, the German Science Foundation (DFG) announced its decision to award the status of a research cluster of excellence (Exzellenzcluster) to 57 cluster proposals from all disciplines. This was the first step of its so-called excellence strategy, (Exzellenzstrategie/ExStra, formerly known as the Exzellenzinitiative/ExIni). Each cluster receives about 7 to 12 million Euro per year, which is a huge financial boost in the context of the German academic system.
Since the beginning of the debate about ExStra/ExIni, there have been supportive and critical voices, with the debate continuing and reemerging when an important decision is made, such as the one last week. This time, some of the critical voices referred to a recent study purportedly showing that an increase in the share of competitive research funding does not increase research quality, but rather decreases it. I wholeheartedly endorse the goal of evaluating science policy and academia, but I firmly believe that this study does not allow one to draw any causal inferences about any element of the political and academic system and research quality (there is no point in calling out anyone who made this claim on Twitter, so I am not including such tweets here). The published study is available here; a ungated summary with the main findings is available here.
First, the study uses as the outcome a field-normalized measure of highly-cited publications (top 10%). This is not research quality. Neither the rate of output nor the number of citations are indicators of quality. For judging the quality of articles, one would have to read them or, at a minimum, derive some quality indicators to code them, such as having a clearly formulated research question, interpreting the results correctly, pointing out the limitations of the study, etc. One might counter what many researchers and, possibly, funders want are a great many citations and the use of this measure is justified. This might be, but what matters is that the authors and critical voices of ExStra interpret citations as a measure of quality and that’s fallacious.
Second, the data is aggregated to the national level. This is understandable because collecting data about the share of competitive funding on the departmental or individual level would be very demanding. However, lots of insightful variation can go lost in aggregating data to the national level and the problem of ecological inference comes into play. Within each country, more competitive funding might lead to more publications of top-cited articles, but the trend is negative across countries on the aggregate level.
Third, the analysis is based on bivariate correlations. This is fine for exploratory purposes, but, in this case, not for causal inference. The theoretical model that is presented in the study suggests that the total effect of competitive funding is identified, but I highly doubt it (see the model in the ungated report). One can think of common causes such as ‘competitive pressure imposed by science policy makers’ that point into ‘autonomy’, ‘competitive project’ funding and ‘national evaluation research system’ or just into ‘competitive project funding’ and the outcome (belief in the benefits of competition leads to more competitive funding, but increased internal pressure to be productive makes researchers submit papers too early that are of lower quality, leading to fewer citations). At the least, one would need to argue in much more detail that this causal model is complete (if we take it as a causal model) because it does not look complete as it stands.
The proper interpretation of this study is much more limited: a higher share of competitive funding is weakly negatively correlated with the publication of highly-cited articles. This is an interesting insight, but not a causal one. I am not saying that ExStra is terribly good at improving research quality because I do not know how ExStra/ExIni influenced the German academic system. The point is that none of us know the causal effect of competitive funding, even after having read this study.