Is Open Science passé? is the question asked by Xenia Schmalz in this blogpost. I recommend reading it before I share brief thoughts on some points that are raised.
I wish an open science movement was not needed anymore, but I agree this is most likely not the answer to the leading question. Neither has the open science movement failed; progress toward more transparent and credible science is simply slow. When one looks at how many articles share code and data (or, maybe, how many do not); what the quality of the shared material is (README file, code annotation etc.); how often substantive significance is not discussed or misinterpreted in quantitative research etc., I feel there is still much room for improvement. The question is what this means for open science and meta science.
Does it make sense to write another article on the misinterpretation of the p-value in a given fieldor subfield? Probably yes, it depends on the field. It may be less needed in a field like psychology because there seems to be a larger degree of awareness to matters of research quality and credibility. In other fields, probably including political science, it may be interesting to review the interpretation of p-values and substantive significance because fewer graduate students may have received training on this. Do we need another many-analysts study showing that results differ across analysts? I am less sure about that, but this is a topic for a separate blog post.
I think the reviewer comment that motivates the blog post by Xenia Schmaltz illustrates that for many papers the study and results may be obvious to a researcher from the field, leading to a comment like “What’s the news here?“. For many other researchers, however, the study may be interesting and important to read because they do not follow closely work on open science and meta science. It is fine when an empirical researchers pays most attention to theories and empirical issues that are pertinent to the own subfield because this is where the primary interest lies. For open science and meta science reserchers, this means that there is still much work to do, including research, teaching and publishing on topics that may seem like old news to experts from these two fields.
Politics, Science, Political Science 