Accuracy in Student Placement Data

Placement of engineering students at the end of their undergraduate studies is one measure of the success of the educational program. For the measure to be helpful in providing feedback about educational programs, the reported data must present an accurate picture of placement. Accuracy is limited by low response rate, non-response bias, and restrictions on employment eligibility faced by international students. Simulations in this paper demonstrate that, even in the absence of non-response bias, low response rates can lead to inaccurate estimates of the fraction of graduates headed to industry (vs. going to graduate or professional school or pursuing other activities) and inaccurate estimates of job placement. Non-response bias can greatly overestimate job placement rate, even in the setting of otherwise very good response rates, also illustrated by simulation. High enrollment of international students in undergraduate engineering programs coupled with restrictions on their legal right to work in the U.S. can lead to lower overall placement rates that are not necessarily indicative of the quality of the students, education, or placement efforts of that institution. Strategies for increasing response rate and thus eliminating non-response bias as well as a recommendation for separate analysis of the placement of domestic vs. international students are presented. Introduction Most engineering schools gather student placement data, which is then used to provide feedback about educational programs, to guide allocation of resources and effort related to placement, and to inform prospective students of what might be in store for them. For placement measures to be helpful for these purposes, the reported data must present an accurate picture of placement. Three significant issues negatively affect the accuracy of these data: low response rate, nonresponse bias, and restrictions on employment eligibility faced by international students. Impact of low response rate on placement results In a limited survey of ten highly ranked U.S. engineering programs where a Ph.D. is the highest degree offered, undergraduate placement survey response rates for the Class of 2011 ranged from 44 to 95%, with a median of 75%. Placement data and response rates were determined from Page 24131.2 online postings by the given institution or obtained in response to a personal telephone or e-mail request of the appropriate administrator. Most engineering programs aim to obtain survey data from an entire graduation cohort rather than sampling a subset of the cohort. Thus, sampling bias, that is non-random selection of which graduates to survey or interview, is generally not an issue. The greater concern is response rate. The American Association for Public Opinion Research (AAPOR) provides six different definitions for response rate. Of most relevance to placement reports are the first and second definitions: RR1 = I (I + P) + (R + NC + O) + (UH + UO) and RR2 = I + P (I + P) + (R + NC + O) + (UH + UO) where I = # of complete responses P = # of partial responses R = # of refusals to respond NC = # of non-contacts O = # of other non-responses UH + UO = # of unknown eligibility For purposes of computing accurate placement results, a partial response may be sufficient if placement information is provided even though other survey items such as starting salary or forwarding address may not be provided. Thus, a qualified use of RR2 can be appropriate. As the cohort of graduating students is a clearly defined population, UH + UO is zero in this