Usefulness of Internet Surveys to Identify People with Disabilities: A Cautionary Tale

Disability is an important characteristic to consider in survey research. However, people with disabilities are a hard-to-reach population. Internet survey methods offer tremendous potential to expand researchers’ ability to reach and learn about people with disabilities. The goal of this study is to examine potential bias when using nonprobability Internet samples to investigate demographics and socioeconomic outcomes of people with disabilities. We compare the findings based on a national employment and disability survey instrument fielded to four samples: (1) a random-digit dial (RDD) sample, (2) a prescreened sample from a nonprobability Internet access panel, for which screening was based on the presence of 139 previously reported health conditions, (3) an unscreened sample from another nonprobability Internet access panel (without previously prescreened health conditions), and (4) a mixed nonprobability self-recruited (river and snowball) sample. Each sample was weighted on four demographic variables (gender, age, race/ethnicity, and region) using benchmarks from the American Community Survey (ACS). Three dichotomous outcome variables of interest (level of education, household income, and current employment status) were contrasted with weighted population estimates from the ACS. Results showed that the sample resulting from the RDD and all three nonprobability Internet samples differed significantly from ACS population estimates on all three outcome variables. Reweighting to include type of functional disability did not significantly reduce dissimilarities with ACS for any of the four samples. Nonprobability Internet survey methods offer relatively low-cost, easy-to-use avenues for disability-related research. Yet, researchers must proceed with caution to reduce or avoid known sources of bias in both the methodology and the interpretation of results.