Mode system effects in an online panel study: Comparing a probability-based online panel with two face-to-face reference surveys

Bella Struminskaya, Edith De Leeuw, Lars Kaczmirek

Veröffentlichungen: Beitrag in FachzeitschriftArtikelPeer Reviewed


One of the methods for evaluating online panels in terms of data quality is comparing the estimates that the panels provide with benchmark sources. For probability-based online panels, high-quality surveys or government statistics can be used as references. If differences among the benchmark and the online panel estimates are found, these can have several causes. First, the question wordings can differ between the sources, which can lead to differences in measurement. Second, the reference and the online panel may not be comparable in terms of sample composition. Finally, since the reference estimates are usually collected face-to-face or by telephone, mode effects might be expected. In this article, we investigate mode system effects, an alternative to mode effects that does not focus solely on measurement differences between the modes, but also incorporates survey design features into the comparison. The data from a probability-based offline-recruited online panel is compared to the data from two face-to-face surveys with almost identical recruitment protocols. In the analysis, the distinction is made between factual and attitudinal questions. We report both effect sizes of the differences and significances. The results show that the online panel differs from face-to-face surveys in both attitudinal and factual measures. However, the reference surveys only differ in attitudinal measures and show no significant differences for factual questions. We attribute this to the instability of attitudes and thus show the importance of triangulation and using two surveys of the same mode for comparison.
Seiten (von - bis)3-56
Fachzeitschriftmethods, data, analyses (mda)
PublikationsstatusVeröffentlicht - 2015
Extern publiziertJa

ÖFOS 2012

  • 102019 Machine Learning
  • 504007 Empirische Sozialforschung
  • 501006 Experimentalpsychologie
  • 501018 Psychologische Diagnostik