Sequencing Effects in the Analysis of Complex Experiments in Business Research: Mechanisms, Biases, and Recommendations

Experiments in business research became more complex overtime, yielding complex sequences of stimuli and measurements. Thisraises the issue of sequence effects, where effects are found onlyin specific sequences of the experiment. One case in point isfactorial surveys. Here, presenting the stimulus is followed byasking subjects to evaluate several vignettes presented in a certainsequence. The researcher is interested in the effect of the stimuluson responses to vignettes with certain features. As sequence andstimulus can be made uncorrelated by construction, holding thesequence constant or excluding the sequence from the analysis seemsto be justified when researchers are only interested in effects ofvignette features or the stimulus. In both cases, even if thesequence is relevant for the dependent variable, correlation betweensequence and stimulus, the necessary condition for an omittedvariable bias, is absent. The effect estimated for the stimulusshould thus be unbiased. We show that even in the case wherestimulus and sequence are uncorrelated or the sequence is heldconstant, an omitted variable bias occurs when the effect of thestimulus in a vignette is in its magnitude dependent on the sequencein which the vignettes were presented. Such an effect would bemodeled by including a sequence‑stimulus‑interaction term and theomitted variable is this interaction term, which is, byconstruction, always correlated with each of the constitutivevariables. A simulation is presented to illustrate the problem.Implications for experimental research are discussed.