Equivalent Selective Attention Scores From Different Digital Devices: On the Fairness of Assessments Designed for Smartphones, Tablets, and Desktop Computers

Organizations are increasingly offering pre-employment assessments on different digital devices to evaluate candidates. However, in most cases, it remains untested whether the psychometric properties of those assessments are equivalent when different devices are used. Thus, for most assessments, it is unclear whether scores of candidates differing in their choice of device can be compared fairly. The aim of this study is to investigate whether employing a mobile first based cognitive assessment yields equivalent scores of selective attention across different devices. Measurement equivalence across device types was tested using data collected from 296 matched participants. Participants completed the assessment on either a desktop computer or a smartphone. The equivalence of selective attention test scores was investigated using confirmatory multigroup factor analysis. Measurement invariance ensures that test-takers with the same latent trait level have equal probabilities of solving each item and achieving the same scores. Employing a mobile first design approach resulted in equivalent psychometric properties of the assessment in both groups, as indicated by measurement invariance on all levels of investigation. Thus, measurement invariance was achieved unconditionally for both test groups. The results of this study provide convincing evidence that adhering to mobile first principles can yield a valid and reliable assessment for selective attention that can be used on different devices equivalently. The study highlights the importance of considering different devices when designing digital assessments to avoid systematically disadvantaging candidates due to their choice of device.