SIPS 2026 Online

Pilot studies, typically conducted to refine methods and procedures ahead of planned data collection, are common in psychological research, but little guidance exists on how to report such studies in publications. Therefore, we designed a study that aims to examine whether 1) pilot studies are reported in published psychology literature and 2) how they are reported when mentioned. To answer these questions, we conduct a retrospective, observational assessment of recent articles in psychology journals. In the current presentation, we will present and discuss the study methodology and preliminary findings from two pilot studies where we sampled and coded over 300 articles published in 2022 and 2023. We report the main descriptive findings (e.g., proportion of pilots mentioned and related details such as pilot sample size, purpose, or results) alongside qualitative observations on how pilot studies were reported.

Piloting—the preliminary testing of a study before full data collection—is integral to psychological research, yet these practices are rarely discussed and lack concrete guidance. Research methods textbooks seldom cover piloting, and when they do, it is often reduced to a brief footnote. Consequently, current techniques are tied to individual training and perceived field norms.

Piloting varies considerably within and across workflows (Pilot Reporting Task Force, 2024): researchers use different approaches, may understand "piloting" differently, and some are unaware that others pre-test their methods at all. This variability is not problematic in itself; rather, because so little attention is paid to these methods, we cannot fully learn from or develop them.

We therefore propose creating a living, open-access handbook to make piloting more visible, learnable, improvable, and valuable across psychological subfields. In this unconference, we aim to brainstorm how such a resource could be developed.