Development and Testing of an Instrument to Understand Engineering Doctoral Students’ Identities and Motivations

This research paper explores the formation and psychometrics of a survey developed to understand the identities and motivations of engineering graduate students. Within STEM, almost 50% of students leave before completing their PhD, and many interventions suggest strengthening students’ identities to prevent dropout. However, engineering identity and identitybased motivation (IBM) within the engineering graduate population is not well-explored. To remedy this, two questionnaires were created, with items derived from qualitative research with graduate students. Data were collected from 227 engineering graduate students, and were analyzed for patterns in missingness (i.e., individual items skipped by participants despite overall survey completion), effects on participants’ emotional state (using the Positive and Negative Affect Schedule, PANAS), and validity and reliability. MANOVA analysis indicated there was no significant effect of survey focus on positive or negative affect, F(4,646) = 1.075, p = .368, but that data was not missing at random in the IBM survey, χ(503) = 580.80, p = .009). With Exploratory Factor Analysis, the latent constructs were tested and the list of items refined. The implications of these findings for the full survey and for future studies will be discussed. Introduction: the GRADS Project Amidst calls for an increase in STEM graduates in the U.S., attrition among engineering graduate students remains a serious issue [1]. Previous studies have indicated that the attrition rate among STEM doctoral students is as high as 50% [2], and retention of students from traditionally marginalized groups continues to be of special concern [3]. These studies also indicate that strong engineering identities and clear future goals are critical to student success [4]–[6], but often fail to include graduate students as a population distinct from undergraduate students [7]–[9]. To begin remedying this gap, the GRADS project was proposed, a qualitative and quantitative investigation of engineering doctoral students’ (EDS) experiences, identities, and motivation [10]. As the first step in this process, three qualitative studies were conducted with an EDS sample [11]–[13]. This was done both to investigate whether EDS framed their experiences and identities similarly to previous undergraduate populations, and to begin exploring questions for the planned quantitative phase. Results indicated that engineering identity, identity fit, and future goals were as important for doctoral students as they were for undergraduates, but that the constructs were sometimes articulated and prioritized differently. For instance, graduate students who strongly identified as engineers took a more agentic role in defining what ‘engineering’ meant, recognizing that the field is broad and that the stereotypical definitions can be limiting [11]. Similarly, graduate students used past experiences to clarify goals and describe their future place in the field, unlike undergraduates who often only considered present performance [12]. The second step in the GRADS project entailed the development and administration of a nationally representative survey of EDS. To do this, the research team gathered previously developed scales and surveys, published literature about EDS identities and motivations (including the three qualitative studies mentioned previously), and combined them to fit the current survey’s goals while retaining previously established metrics of validity and reliability. This resulted in the development of three full-length surveys, each focusing on engineering identity, identity-based motivation, and future time perspective. To combine these three surveys into a single, smaller survey that could be used with a national population, we piloted them with an engineering graduate student sample and the results were analyzed for validity and reliability using exploratory factor analysis (EFA), ANOVAs, and Little’s MCAR test (missing completely-at-random test). This research paper explores the development of two of the three pilot surveys, specifically those focusing on issues of identity and identity-based motivation. The process by which previous research and interview data from graduate students were combined is explored, as well as the quantitative analyses of the surveys’ psychometrics. We conclude with recommendations for others seeking to extend and transfer findings from the undergraduate to the graduate space, as well those seeking to develop surveys for use with nationally representative populations of engineering students. Overall, the goal of this paper is to recount the creation, testing, and validation of the survey measure before it was used with a national population. We hope this paper will contribute to the discussion of survey development best practices within engineering education, as well as act as a future resource for the GRADS project as data and results are shared.