Is the scientific reform movement headed in the right direction?
Call for proposals
Call for contributions
4000 (excl. references & abstract)
May 18, 2022
September 30, 2022
The year 2011 ushered in a challenging era for the behavioral and social sciences (Wagenmakers, 2012). High-profile fraud cases, wide scale failures to reproduce some of psychology’s keystone studies, and reports of many researchers admitting to having engaged in research misconduct (John, Loewenstein & Prelec, 2012) all painted a grim picture of science.
In response to these concerns, a movement has arisen to reform science (Munafo et al., 2017; Spellman, Gilbert & Corker, 2018). This movement has led to initiatives that intend to improve research methods, decrease misconduct, and make science more transparent. Studies that conduct direct replications of previous findings and articles reporting null results have become easier to publish. Preregistration (in which authors register their study plans in advance of conducting a study) and registered reports (in which a research proposal is peer-reviewed and accepted for publication before a study is conducted) have grown exponentially in popularity. And more parts of science are now “open,” such as preprints, open-access publications, open peer review, and openly available data and code.
These initiatives generate clear and direct benefits for both science and scientists. At the same time, the scientific reform movement is still in its infancy and the long-term effects of many interventions have not been evaluated. The situation is made even more difficult by the fact that academic science is a complex system, and the downstream consequences of modifying complex systems are notoriously difficult to anticipate.
Several scholars have expressed concerns about these issues. In a 2021 article on scientific reform methodology, Devezer, Navarro, Vandekerckhove and Buzbas argue that the reform policies have “little evidentiary backing” and are “based on methods which are suggested with no framework for assessing their validity or evaluating their efficacy’’ (p. 1). Others (Ioannidis, 2014; Tiokhin et al., 2021) have expressed related sentiments, particularly regarding the potential for unintended consequences of otherwise well-intentioned reforms
To have a better chance of generating interventions that improve scientific practice, to mitigate the effects of unintended consequences, and to maintain its credibility as a movement open to self-criticism, the scientific reform movement can benefit by continually interrogating its practices and proposed initiatives.
In this call, we aim to inspire scholars to take heed and critically reflect on the direction in which the scientific reform movement is heading. Is scientific thought and practice being steered in a good direction? What are the known unknowns and unknown unknowns that challenge the field? Are there lessons that can be learned from other fields that have struggled with similar issues?
With the goal of stimulating a critical discussion in the field of scientific reform, we invite submissions that focus on “second-order” effects or “second-generation” challenges: issues that arise during or as a result of the resolution of a primary issue. We encourage a diversity of contributions, but all contributions should discuss a second-generation challenge for the scientific-reform movement. Examples of potential contributions include:
Historical and philosophical articles
Articles communicating insights from other fields
Formal theoretical models
Potential topics include:
Tipping points in complex systems
Second-order responses to changing incentive structures
Insights from innovation sciences and cultural evolution about cumulative knowledge accumulation
Excessive use of scientific resources (on e.g., open access in high-profile journals)
Gaming open science
Improved methodological rigor being circumvented novel Questionable Research Practices
Mindless check-box scientific practices
Issues with measuring scientific progress
The effects of reforms on scientific creativity
Sarahanne M. Field (email@example.com)
Leonid Tiokhin (firstname.lastname@example.org)
Noah van Dongen (email@example.com)
Please submit a proposal for consideration prior to beginning work on your contribution. The proposal should contain a brief outline (200 words max) of your potential contribution to this special issue. We will respond to proposals once the due date has passed.
Proposals are due by 1700h UTC+01:00 (Amsterdam), May 18th, 2022.
Final contributions are limited to 4000 words, excluding references and abstract. Please ensure that you submit a proposal early enough to have time to complete your full contribution by the deadline.
Full contributions are due by 1700h UTC+01:00 (Amsterdam), September 30, 2022.
Other general guidelines for submission to JOTE, and the submission portal itself can be found at: https://submit.jtrialerror.com/index.php/jote/about/submissions
Note that JOTE is a diamond open access journal. You will not be charged article processing fees.
Proposals and inquiries should be sent to Sarahanne M. Field at firstname.lastname@example.org
Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2020). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), 200805.
Ioannidis JPA (2014) How to Make More Published Research True. PLOS Medicine 11(10): e1001747.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Percie du Sert, N., ... & Ioannidis, J. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 1-9.
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018). Open Science. Stevens' Handbook of Experimental Psychology and Cognitive Neuroscience, 5, 1-47.
Tiokhin, L., Panchanathan, K., Lakens, D., Vazire, S., Morgan, T., & Zollman, K. (2021). Honest signaling in academic publishing. PloS one, 16(2), e0246675.
Wagenmakers, E. J. (2012). A year of horrors. De Psychonoom, 27, 12-13.