Don Moore
Don Moore holds the Lorraine Tyson Mitchell Chair in Leadership at the Haas School of Business at the University of California at Berkeley. He received his Ph.D. in Organization Behavior from Northwestern University. His research interests include overconfidence, including when people think they are better than they actually are, when people think they are better than others, and when they are too sure they know the truth. He is only occasionally overconfident.
UC Berkeley
@donandrewmoore.bsky.social
Session
We conducted a study each year 2021-2024 comparing data quality across online platforms (e.g., Prolific, CloudResearch) and university laboratory pools (UC Berkeley, UChicago). Participants completed identical surveys with multiple quality assessments: “attention checks” (e.g., embedded instructions), “accuracy checks” (e.g., reading comprehension, writing quality metrics), and self-reported attention and engagement measures. Three sets of results challenge common beliefs about data quality. First, across years and platforms the quality measures correlated only weakly with each other (often r<.17), suggesting quality is a multi-dimensional construct. Second, online participants outperformed lab participants on traditional attention checks and self-reported attention, but lab participants performed higher on accuracy checks, suggesting these pools have systematically different quality. Finally, despite numerous changes over the years we studied (e.g., AI), quality metrics remained relatively stable. These findings suggest different platforms yield different types of quality suited to different research needs.