Is scientific reform heading in the right direction? A call for contributions.

Preregistration. Registered Reports. Open sharing of data and materials. Preprints. Getting rid of journals. Stopping the practice of evaluating scientists based on the Impact Factor of the journal in which they publish. Getting rid of p-values. Less competition & more collaboration. More team science. More replications. Less emphasis on novelty. More rigor. Larger sample sizes. More post-publication peer review. Audits of scientists. Lotteries for allocating grant funding.

These are just a subset of scientific reforms that have been proposed & implemented in the last decade. As with most policy changes, scientific reforms are put forth with the best intentions. Many also have an intuitive appeal. Less competition & more collaboration. More rigor. Free and rapid sharing of information with other scientists. Who could possibly be against those?

But the road to scientific salvation is dark and full of terrors. Many reforms do not yet have a solid evidence base. And there are typically no agreed-upon criteria for deciding whether reforms “worked.” What is the appropriate timescale on which to measure the effectiveness of interventions? What is the relevant outcome variable? Are there other variables that are also affected by the intervention, such that the system-level outcome is some combination of these things?

These are hard questions. And often times, it’s tempting to substitute a hard question — is incentivizing sharing of data and materials good for science? — with an easier one — were more errors caught in papers published in journal xyz after the journal modified its data-sharing policy? This is better than nothing, but it isn’t really what we want to know.

Science is a complex system, and interventions to complex systems are…complicated. Changing one part of the system can cause unexpected changes in other parts. There are feedback loops, delays, and unanticipated nonlinearities. Interventions interact with one other. And the actors whose behavior you are trying to modify are constantly coming up with ways to circumvent your interventions and continue to pursue their own goals.

It’s because of such complications that interventions often produce unintended consequences. And it’s also why systems theorists worry so much about policy makers who confidently modify complex systems, only to push systems in the wrong direction.

A call for contributions on the consequences of the scientific reform movement

Sarahanne Field, Noah van Dongen and myself are co-editing a special issue of the Journal of Trial and Error, focusing on the consequences of the scientific reform movement. Our goal is to inspire scholars to critically reflect on existing scientific reforms and the directions in which the reform movement may be headed.

Are current reforms pushing science in the right direction? What unintended consequences should we worry about, and do we have tools for anticipating and mitigating these? Are there reforms that have gone wrong, and what lessons can we learn from these? What are the known unknowns and unknown unknowns that challenge the field?

We are inviting submissions from anyone who can provide insights on such “second-generation” challenges — issues that arise during or as a result of the resolution of a primary issue. We encourage a diversity of contributions, but all contributions should discuss a second-generation challenge for the scientific-reform movement.

Examples of potential contributions include:

• Historical and philsophical articles
• Opinion pieces
• Articles communicating insights from other fields
• New data
• Empirical syntheses
• Formal theoretical models

Potential topics include:

• Unintended consequences
• Tipping points in complex systems
• Second-order responses to changing incentive structures
• Insights from innovation sciences and cultural evolution about cumulative knowledge accumulation
• Excessive use of scientific resources (on e.g., open access in high-profile journals)
• Gaming open science
• Improved methodological rigor being circumvented novel Questionable Research Practices
• Mindless check-box scientific practices
• Issues with measuring scientific progress
• The effects of reforms on scientific creativity

Practical Information

Special Issue Editor-in-Chief

Sarahanne M. Field (field@centeroftrialanderror.com)

Guest Editors

Leo Tiokhin (leotiokhin@gmail.com)

Noah van Dongen (nnnvandongen@gmail.com)

Proposal

Please submit a proposal for consideration prior to beginning work on your contribution. The proposal should contain a brief outline (200 words max) of your potential contribution to this special issue. We will respond to your proposal within 7 working days of receiving it.

Proposals are due by 1700h UTC+01:00 (Amsterdam), May 16th, 2022.

Final contributions are limited to 4000 words, excluding references and abstract. Please ensure that you submit a proposal early enough to have time to complete your full contribution by the deadline.

Full contributions are due by 1700h UTC+01:00 (Amsterdam), September 30, 2022. Original call for contriutions is available at https://archive.jtrialerror.com/pub/callscientificreform/release/2

Other general guidelines for submission to JOTE, and the submission portal itself can be found at: https://submit.jtrialerror.com/index.php/jote/about/submissions

Note that JOTE is a diamond open access journal. You will not be charged article processing fees.

Proposals and inquiries should be sent to Sarahanne M. Field at field@centeroftrialanderror.com