2026-06-09 –, AUDITORIUM
We conducted a study each year 2021-2024 comparing data quality across online platforms (e.g., Prolific, CloudResearch) and university laboratory pools (UC Berkeley, UChicago). Participants completed identical surveys with multiple quality assessments: “attention checks” (e.g., embedded instructions), “accuracy checks” (e.g., reading comprehension, writing quality metrics), and self-reported attention and engagement measures. Three sets of results challenge common beliefs about data quality. First, across years and platforms the quality measures correlated only weakly with each other (often r<.17), suggesting quality is a multi-dimensional construct. Second, online participants outperformed lab participants on traditional attention checks and self-reported attention, but lab participants performed higher on accuracy checks, suggesting these pools have systematically different quality. Finally, despite numerous changes over the years we studied (e.g., AI), quality metrics remained relatively stable. These findings suggest different platforms yield different types of quality suited to different research needs.
Liman Wang, Weishan Zhang, Chenyu Wang, Donald Lyons, Aastha Mittal, Rachel Tran, Juliana Schroeder
Could you give us a rough idea of what your lightning talk / poster presentation will cover? Such as:: A brief presentation of past research (a completed study)
Don Moore holds the Lorraine Tyson Mitchell Chair in Leadership at the Haas School of Business at the University of California at Berkeley. He received his Ph.D. in Organization Behavior from Northwestern University. His research interests include overconfidence, including when people think they are better than they actually are, when people think they are better than others, and when they are too sure they know the truth. He is only occasionally overconfident.