SIPS 2025 Budapest

Concerns about the construct validity of psychological measurements are increasing. In this lightening talk, I propose a distinction between the validity potential of psychological measurement procedures (e.g., personality tests) and the realised validity of the scores they produce in specific instance (e.g., personality test scores collected in a study) and describe how this distinction can facilitate better construct validation practices and encourage more frequent validity evidence reporting.

Calls to improve the construct validity of psychological measurements are increasing. However, there is less information available about how to improve, or even evaluate, construct validity. One potential barrier to developing clear construct validation guidelines is the undifferentiated inclusion of measurement and non-measurement (e.g., prediction, diagnosis) uses of test scores under the umbrella of construct validity. This can be problematic because the characteristics that make a test a good measure of a construct can differ from the characteristics that make a test good for other uses like prediction or diagnosis. In this hackathon, we will develop a framework for evaluating psychological tests that distinguishes between key test uses and summarises the most relevant sources of validity evidence based on each test’s type and use.