ERЯOR: A Bug Bounty Program for Science

Speaker: Malte Elson, University of Bern

Co-Authors:

Abstract

The scientific enterprise, as a human profession not immune to error, has developed several failsafe mechanisms (e.g., peer review) that also reflect one of its basic tenets: Science is self-correcting. However, these mechanisms are not purposely designed to catch errors, and often only do so long after errors have already proliferated in the literature. Most errors are discovered coincidentally, and error detection as a scientific activity is rarely incentivised.

The project Examining the Reproducibility and Robustness of Research (ERЯOR) is a comprehensive program to systematically detect, report, and prevent errors in scientific publications modelled after bug bounty programs in the technology industry with two major goals: 1) Estimating a benefit-cost ratio of funding error detection, and 2) Obtaining robust empirical estimates of types of errors and their prevalence in the literature.

In ERЯOR, Investigators examine highly-cited published works (including study materials, data, and code) for errors and receive monetary compensation depending on the severity of their findings: the more impactful the errors, the greater the payout. Similarly, authors who agree to having their work examined this way receive compensation if their work proves to be reliable.

A cost-benefit analysis of implementing ERЯOR on a larger scale will need to consider (1) running costs, (2) counterfactual costs (e.g., papers existing in the literature without having been purposely reviewed for errors), (3) consequential costs (e.g., further grants awarded building on flawed research), and (4) opportunity costs (other ideas not pursued instead of flawed research).

ERЯOR joins a range of proposals and measures to foster a culture that is open to the possibility of error in science, and that embraces a new discourse norm of constructive criticism.

Back to top