Craig2024FAIRMEPR

Title

From Open Review to Reproducible Review: FAIR Metrics Analysis of Open Peer Reviews for Brain Informatics Literature
View PDF | Save PDF

Authors

Adam Craig, Carl Taswell

Affiliations

Brain Health Alliance Virtual Institute, Ladera Ranch, CA 92694 USA

Abstract

Brain informatics helps researchers discover and derive new insights from existing data and metadata in brain sciences, medicine, and healthcare, making the documentation of information methods, platforms, and data sources in scholarly meta-research especially important in this field. Evaluation of new reports by expert peer reviewers remains essential to maintaining the integrity of this published research, but determining the best way to assess the quality of these peer reviews has not been addressed adequately and poses an open question about what should be open peer review. Previously, we proposed the paradigm of reproducible peer review, in which a second reviewer should be able to draw on the same factual claims as the first reviewer in order to reach the same conclusion. We introduced a new family of metrics for peer reviews as an extension of the existing families of Fair Attribution to Indexed Reports (FAIR) Metrics to evaluate how well reviewers attributed the claims substantiating their recommendations to the original sources of that information. However, we only demonstrated this new family of FAIR Metrics on five example peer reviews. We report here the results of FAIR Metrics analyses of published open peer reviews on 14 brain informatics articles. These analyses demonstrate the value of the FAIR Metrics by highlighting ways in which the brain informatics community can improve the reproducibility of the peer review process. We call for open peer review that emphasizes references to or quotes from the specific passages of the work under review, indication of which standards of the publication venue the work meets or fails to meet, and citation of the literature when drawing on prior knowledge of the problem domain.

Keyphrases

Brain informatics, open peer review, PORTAL-DOORS Project, PDP-DREAM Ontology, FAIR Metrics, NPDS Cyberinfrastructure.

Citation

Brainiacs Journal 2024 Volume 5 Issue 2 Edoc QA6A795A3
DOI: 10.48085/QA6A795A3
PDP: /Nexus/Brainiacs/Craig2024FAIRMEPR
URL: BrainiacsJournal.org/arc/pub/Craig2024FAIRMEPR

Dates

Created 2024-10-07, presented 2024-10-09, updated 2024-12-23, published 2024-12-23, revised 2025-03-30, endorsed 2025-03-31.