LT4: An exploratory multiverse simulation framework to assess the impact of arbitrary analitical decisions
Researchers’ degrees of freedom in data analysis presents significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose a framework of an exploratory multiverse simulation to empirically compare various decision pathways to identify how arbitrary analytical decisions affect the conclusions of a study. The framework is demonstrated on the Congruency Sequence Effect (CSE), a well-studied phenomenon in cognitive control research. We reviewed existing literature to identify common non-theory-specific analytical decisions, such as outlier exclusion criteria and hypothesis testing methods and incorporated these into our simulation framework. Using a large number of simulated datasets, we compared the True Positive Rates (TPR) and False Positive Rates (FPR), and observed effect sizes across different decision pathways. We recommend this framework as a tool to quantify the impact of different analytical decisions on study conclusions in notable well-studied effects.