2025-06-25 –, Second floor 217 - WS
Researchers’ degrees of freedom in data analysis presents significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose a framework of an exploratory multiverse simulation to empirically compare various decision pathways to identify how arbitrary analytical decisions affect the conclusions of a study. The framework is demonstrated on the Congruency Sequence Effect (CSE), a well-studied phenomenon in cognitive control research. We reviewed existing literature to identify common non-theory-specific analytical decisions, such as outlier exclusion criteria and hypothesis testing methods and incorporated these into our simulation framework. Using a large number of simulated datasets, we compared the True Positive Rates (TPR) and False Positive Rates (FPR), and observed effect sizes across different decision pathways. We recommend this framework as a tool to quantify the impact of different analytical decisions on study conclusions in notable well-studied effects.
