Premium
Question order bias revisited: A split‐ballot experiment on satisfaction with public services among experienced and professional users
Author(s) -
Thau Mads,
Mikkelsen Maria Falk,
Hjortskov Morten,
Pedersen Mogens Jin
Publication year - 2021
Publication title -
public administration
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.313
H-Index - 93
eISSN - 1467-9299
pISSN - 0033-3298
DOI - 10.1111/padm.12688
Subject(s) - ballot , government (linguistics) , order (exchange) , agency (philosophy) , scale (ratio) , service (business) , psychology , survey data collection , data collection , public relations , social psychology , business , marketing , political science , statistics , sociology , politics , finance , law , mathematics , social science , linguistics , philosophy , physics , quantum mechanics , voting
Abstract Public decision‐makers increasingly rely on satisfaction surveys to inform budget and policy decisions. Yet, our knowledge of whether, and under what conditions, this input from public service users provides valid performance information remains incomplete. Using a preregistered split‐ballot experiment among government grant recipients in Denmark, this article shows that the ordering of survey questions can bias satisfaction measures even for highly experienced and professional respondents. We find that asking about overall satisfaction before any specific service ratings lowers overall user satisfaction, compared to the reverse order, while the correlations between specific ratings and overall satisfaction are relatively stable. Also, the question order effect outweighs that of a large‐scale embezzlement scandal, which unexpectedly hit the investigated government agency during the data collection. Our results support rising concerns that subjective performance indicators are susceptible to bias. We discuss how practitioners should approach satisfaction surveys to account for the risk of question order bias.