2026-05-07 –, Track 1
Citation metrics are still relied on when making science management decisions: from hiring and rewarding individual scientist, financing research groups, to budgeting for departments and whole universities. This goes on despite many primary studies showing the invalidity of citation metrics as indicators of scientific rigor or quality. Individual studies ranked on their citation count, and journals raking on their impact according to various metrics has been found to be negatively related to the statistical power of reported research ("N-pact factor") , research replicability , and instead positively related to the presence statistical reporting errors.
To increase the velocity of abandoning these metrics in science management decisions, we need a definitive, synthesized evidence base. This hackathon aims to lay the groundwork for a comprehensive Umbrella Review or large-scale meta-analysis assessing the validity of journal metrics as indicators of research quality, rigor, or impact.
A draft Preregistration for a systematic review/meta-analysis on the OSF.
A finalized Search Strategy and Coding Protocol ready for pilot testing.
The formation of a Collaborative Team to carry out the review post-conference.
How will the session's content foster diversity & inclusion (e.g., who will present, who will it serve), and how will it improve psychological science?:This session addresses a critical equity issue in academia. The reliance on citation-based metrics disproportionately disadvantages researchers from the Global South, those at under-resourced institutions, and marginalized groups who may not have access to the "prestige networks" that drive high citation counts. By rigorously debunking the link between JIF and quality, we aim to flatten the hierarchy and promote evaluation based on methodological rigor rather than journal brand.
We welcome participants from all career stages and backgrounds. We specifically encourage participation from researchers with experience in bibliometrics or systematic reviews, as well as students who want to learn how to design a meta-analysis. The resulting project will operate on a CRediT authorship model, ensuring that all contributors to the protocol and subsequent review are recognized.
Please note any pre-requisite knowledge/expertise you will expect from attendees (i.e., is the session most appropriate for someone who already has experience with a topic?).:No prior experience with meta-analysis is strictly required, though helpful. A basic understanding of research methods in psychology (e.g., understanding what statistical power or p-values are) will be beneficial for the discussion on coding schemes.
I am a PhD student of organizational psychology at Masaryk university in Czech republic. My broad interests range from philosophy of science and theory construction in psychology to research applications in public policy. I am active in univesity governance as a member of faculty and university academic senates, internal board of quality evaluation, and faculty committee on PhD programme quality.