An overview of tools for detecting irregularities in reported summary statistics

Speaker: Lukas Jung, University of Bern

Co-Authors:

Abstract

The need to check published research for accuracy has been increasingly recognized (Elson, 2024; Nosek et al., 2022; Wilkinson et al., 2025). With the impending Cochrane requirement to apply the forensic INSPECT-SR checks in all systematic health reviews, error detection has gained heightened relevance (Wilkinson et al., 2025). However, researchers often lack practical means to investigate articles for possible errors. This talk presents an emerging suite of such forensic metascience techniques, and the software packages that implement them, developed and implemented by our research group. In particular, this talk presents the R packages scrutiny, tides, and unsum, along with their web-apps written in Shiny, which implement the relatively prominent GRIM test (granularity-related inconsistency of means), but also newer techniques that are currently being developed, such as GRIM mapped to error repeats (GRIMMER), truncation-induced dependency in summary statistics (TIDES), sample parameter reconstruction via iterative techniques (SPRITE) and complete listing of original samples of underlying raw evidence (CLOSURE). Together, these tools enable researchers to check reported summary statistics such as means and standard deviations for numeric consistency, evaluate their credibility by assessing the relative size of standard deviations in context of the other statistics, and reconstruct all possible samples that are compatible with the reported statistics to assess the plausibility of their distributions. Using these packages, researchers can detect impossible or implausible configurations of reported statistics. I will discuss the current state of our research group’s development of these techniques, including their use cases and limitations, teaching experiences and outreach, findings from their large-scale application to the psychology literature, as well as future directions.

Back to top