z-logo
open-access-imgOpen Access
When numbers fail: do researchers agree on operationalization of published research?
Author(s) -
Matthias Haucke,
Rink Hoekstra,
Don van Ravenzwaaij
Publication year - 2021
Publication title -
royal society open science
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.84
H-Index - 51
ISSN - 2054-5703
DOI - 10.1098/rsos.191354
Subject(s) - operationalization , rigour , quality (philosophy) , task (project management) , psychology , empirical research , computer science , cognitive psychology , applied psychology , management science , statistics , epistemology , mathematics , philosophy , management , economics
Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalization of phenomena. Operationalization is the process of translating theoretical constructs into measurable laboratory quantities. Thus, the validity of operationalization is central for the quality of empirical studies. But do differences in the validity of operationalization affect the way scientists evaluate scientific literature? To investigate this, we manipulated the strength of operationalization of three published studies and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalization is more valid, but (2) the different validity is only to some extent reflected in a judgement of the study's quality. These results combined give partial corroboration to the notion that researchers’ evaluations of research results are not affected by operationalization validity.

The content you want is available to Zendy users.

Already have an account? Click here to sign in.
Having issues? You can contact us here