A many-analysts approach to assessing heterogeneity in eye-tracking analysis pipelines
Eye tracking has gained popularity in psychology, computer science, and neuroscience (Rayner, 2009). However, concerns remain that preprocessing choices may influence results (Godwin et al., 2025; Landes et al., 2025). Prior research highlights potential decision points but does not account for analysts’ decisions. Many-analyst studies (Silberzahn et al., 2018), in which multiple teams independently analyse the same data to test hypotheses, demonstrate the impact of the "garden of forking paths” (Gelman & Loken, 2014) on conclusions drawn from the same dataset. We propose a many-analyst study using a visual attention and reading dataset to investigate heterogeneity in the preprocessing and analysis of eye-tracking data. The Stage 1 report has been submitted, and we hope to invite further colleagues to join the project. The lightning talk will cover procedures based on research methods and the challenges they entail. Feedback on operationalising and facilitating the data collection will be appreciated.
