It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks

Instructional manipulation checks (IMCs) have become popular toolsfor identifying inattentive participants in online studies. IMCs function by attemptingto trick inattentive participants into responding incorrectly. However, from aconversational perspective, question characteristics are part of the researcher’scontribution to the conversation, and IMCs may teach participants that there is “morethan meets the eye,” prompting systematic thinking on subsequent tricky-seemingquestions in an attempt to avoid being tricked. In two online studies, participantsresponded to a simple task either before or after completing an IMC. As expected,answering an IMC prior to the task improved performance on items that benefit fromincreased systematic thinking—namely, the Cognitive Reflection Test (Study 1), and aprobabilistic reasoning task (Study 2). We conclude that IMCs change attention ratherthan merely measure attention and discuss implications for their use in onlinestudies