The peer review process as a preregistration device

Preregistration of an analysis requires the prior unobservability of data. One publicly declares what one intends to do before one can access data. One goal of preregistration is to diffuse potential concerns that a publication misrepresents the order in which an empirical analysis was implemented. Reasons for prior unobservability of data could be that the data has yet to be produced (as in experiments and some observational designs), or that access to the data is restricted (such as sensitive microdata that can only be accessed locally and by certified approval of the data host). Based on a peer review experience a friend of mine and I had, it occurred to me that there is a second way of preregistration that works regardless of the prior observability of data.

In our peer review process, a reviewer suggested to additionally test our argument with different data than we had used. We had not thought about using the data before and did it at the reviewer’s suggestion. (I don’t remember whether we had known it; I don’t think so.) The good news for us was that an analysis with the alternative data produced evidence that confirmed our original results. This confirmation is qualitatively different from the results that we reported in the first submission and it should count more, in my opinion.

The reason is that the peer review process and the reviewer comments in particular worked as a preregistration device. The data the reviewer recommended to use was available to us, but we weren’t aware of it and if we had been, we did not use it. What was unavailable to us was the idea of using the data in the way proposed by the reviewer. In analogue to the idea of ‘prior unobservability of data’ that underlies standard preregistration, one could call the experience that we made prior unavailability of an idea (or prior lack of imagination or so).

The advantage of preregistration through peer review is that it is available regardless of the type of design – experimental or observational – and the type of method – quantitative or qualitative. For example, imagine a researcher has done a qualitative content analysis of publicly available speeches of party leaders. The original analysis cannot be preregistered because the speeches have been publicly available from the start of the project. Now, the peer review process would work as a preregistration device if a reviewer (or an editor) has an idea for a coding category that the researcher had not thought of. When the researcher goes back to the speeches and codes them based on the new category, this part of the analysis would be preregistered. The peer review process allows one to timestamp the origin of the idea in terms of the time when it was proposed and the person who proposed it (the originator).

There are several requirements that have to be met for effective preregistration through peer review, where ‘effective’ means that the researcher’s degrees of freedom in misrepresenting an analysis are minimized. First, the peer review process needs to be made transparent. The reader of an article needs to know who the originator of an idea is – the author or a reviewer – and when the idea was introduced by the originator. A published manuscript would need to specify that an element of the empirical analysis was proposed by a reviewer and that this part of the analysis was only implemented after the reviewer proposed to do it.

Second, this information would not be enough because the reader needs to know how specific the reviewer’s proposal was. The more specific the reviewer’s comment, the smaller the researcher’s degrees of freedom when implementing it and the fewer opportunities to misrepresent the analysis and “manufacture” results such that they support a preferred theoretical interpretation.

Third, the reader needs to know what a reviewer proposed to do and what an author declined to implement. With this information, the reader would not be able to tell whether an author refused to follow the reviewer’s recommendation because of the belief that it is superfluous or flawed in terms of substance, theory or method; or because the author did not want to include it because the proposed analysis did produce results that challenge the conclusion in the manuscript and were withheld as a consequence of this. Still, it is useful to make transparent where reviewers and authors disagreed; keeping in mind that even many declined reviewer requests are not a strong indication of selective reporting because the reviews could make ill guided recommendations. The degree of disagreement would be an indication of how selective the author was in following the reviewer’s comments and could be a guide to a follow-up technical replication of the analysis.

Altogether, this shows that preregistration through the peer review process requires transparent peer review that makes public all versions of the manuscript – the first submission plus all revisions – and all reviewer reports, which could be anonymized for preregistration purposes, and editorial letters.

About ingorohlfing

I am Professor for Methods of Comparative Political Research at the Cologne Center for Comparative Politics at the University of Cologne (http://cccp.uni-koeln.de). My research interests are social science methods with an emphasis on case studies, multi-method research, and philosophy of science concerned with causation and causal inference. Substantively, I am working on party competition and parties as organizations.
This entry was posted in meta science, preregistration, publication bias, publishing, replication, transparency, Uncategorized and tagged , , , . Bookmark the permalink.