Using The Emergent Methodology Of Domain Analysis To Answer Complex Research Questions

As engineering education research matures, engineering education researchers seek to answer increasingly complex questions rooted in social situations, such as “What is engineering in various communities?” and “How does engineering work happen at various stages of professional development?” 1 . The desire to ask such questions leads the community to develop or incorporate diverse methods that help the community to answer the complex question. The purpose of this paper is to present to the engineering education community an introduction to domain analysis, an ethnographic method developed within anthropology 2 designed to answer these complex questions. Careful observation serves to identify productive routes to inquiry so as to move the researcher towards understanding relationships present within the social environment defined by the question. Because this methodology can call attention to both desirable and problematic relationships, results from this methodology can inform individual research agendas, program assessment, and policy creation by enabling researchers to construct a map of social situations. Specifically, this methodology builds on the tradition of flexible design characterized by question asking, good listening, adaptiveness and flexibility, grasp of the issues, and lack of bias 2-3 and relies on anthropological techniques of domain analysis. 2, 4 We present domain analysis as an iterative four-step method: 1. Locate a social environment to observe. 2. Decide what evidence already present in that environment helps you answer your question. 3. Identify inter-relationships between the evidence. 4. Organize these relationships according to a question tree. This paper explains these four steps within the context of engineering education research, with specific examples relating to our ongoing investigation of how engineering education researchers research gender. This paper is explicitly about our method; describing our data in detail is outside the scope of this paper. This research method provides important insights needed to design engineering education research agendas both at the individual and community level. Introduction As engineering education research matures, engineering education researchers seek to answer increasingly complex questions rooted in social situations. The desire to analyze social situations leads some engineering education researchers to pull in qualitative research methods from education, 5-6 marketing, 7 sociology, 8 history 9 and anthropology. 10-14 Qualitative methods from these disciplines provide researchers with an ability to explain situations in detail without necessarily having to make specific recommendations for future change. However, researchers who build on the tradition of applied anthropological research use their rich understanding of social situations to make specific recommendations for practice. 15 Applied anthropologists have investigated the culture of design firms, 10,16-17 global partnerships within high-tech industry, 18 and socialization of professional engineers 19 all with the goal of making recommendations for P ge 15343.2 practice. The purpose of this paper is to use techniques from applied anthropology to illustrate how domain analysis 2 can be used to advance research in engineering education. As a methodology, domain analysis is well suited to answer complex questions. Complex questions feature “a community” as a crucial element of study, and reflect activities within a social environment. Different people can answer these very open-ended questions differently. Moreover, complex questions connect together with other questions. Indeed, the benefit to conducting research to answer complex questions lies in creating a thick description that can guide further inquiry. 2 Examples of complex questions present in the engineering education community include questions such as “How is design understood within engineering education?” and “Why is innovation valued by engineering?” Domain analysis is one useful technique for mapping cultural spaces to show order, organization, omissions, and potential places to make change. 2,4 As a method, domain analysis provides a way to arrange this observational data as a set of connected questions. The purpose of this paper is to describe our method as a process so that other researchers can see how the process unfolds. In particular, we focus our example of using this method to offer recommendations to inform others’ future research regarding gender and engineering education. Domain analysis is one of the many tools of naturalistic inquiry. 3 Broadly speaking, domain analysis can be understood as an iterative four-step method: 2, 4 1. Locate a social environment to observe. 2. Decide what evidence already present in that environment helps you answer your question. 3. Identify inter-relationships between the evidence. 4. Organize these relationships according to a question tree. This paper explains these four steps within the context of engineering education research, with specific examples relating to our ongoing investigation of how engineering education researchers research gender. Specifically, we approach our example from the perspective of applied anthropology where we seek to identify how researchers can make specific recommendations for practice. This research method provides important insights needed to design engineering education research agendas both at the individual and community level. It is hoped readers will gain insight as to how to investigate the complex questions in their individual research agendas. 1. Locate a social environment to observe The identification of a social environment in which to situate the research study is a design choice. Because the utility of this method depends on appropriate site selection, we offer some suggestions as to how to select a suitable social environment. A social environment can be a literal site such as a particular laboratory or it can be a metaphorical site such as a loosely connected group of Internet blogs. To begin, social environments have three requisite components: place, actors, and activities. 2 However, the complex question already identifies both actors and activities. In considering the question of “How do engineering education researchers research gender?” we asked a question that had “engineering education researchers” as the actors and “research gender” as the activities. The social environment we choose to answer this question must include both engineering education researchers and people researching gender. However, a researcher has many options when it comes to deciding on a place to situate the P ge 15343.3 inquiry. In choosing a social situation, some researchers may find it helpful to make a list of everywhere they expect to see the targeted actors engaging in the specified activity. For instance, engineering education researchers who research gender might gather at a particular conference, work within certain research groups, or publish in particular journals. The choice of an appropriate place shapes the subsequent inquiry. Researchers should think deeply and meaningfully about the location of their observations. Spradley 2 recommends considering six factors when choosing a social situation: simplicity, accessibility, unobtrusiveness, permissions, recurrence, and participation. Table 1: Critical Factors Influencing Choice of a Social Situation Criterion Definition Why It Matters Simplicity The scope of your investigation of a single situation Naturalistic inquiry is unbounded, people move in and out of social situations according to complex networks Accessibility Ease of entry, ability to record observations IRB protocols, industrial competitive advantage Unobtrusiveness Avoid calling attention to yourself Social situations morph depending on people present Permissions Gatekeepers of the environment Some entries can be quite limited owing to features of the site Recurrence Frequency of the activities you want to see Detailed information requires many observations to validate analysis Participation Entering into the cultural environment Opportunities to make richer observations by participating in the situation you are observing The first factor, simplicity, involves a clear understand of why you want to conduct the study from the outset. Another way of thinking about simplicity is asking the question “What is the easiest way to get at the type of information I need to gather?” Deciding what you need requires a deep, working knowledge of your research question. Considering how your question might connect with other questions of interest within relevant research communities also helps you decide if you are asking a good question. Considering both the easiest and relevant social situations offer a means of defining the scope of your inquiry. We chose to situate our inquiry of understanding how engineering education researchers research gender in the Journal of Engineering Education because this journal seeks to catalyze rigorous engineering education research and is readily available to us as researchers. Ideal simplicity of a situation involves a single place. This definition of simplicity can free a researcher from undue concern about trying to capture every facet of inquiry in an open system. Inherently, human experiences and interactions are unbounded; what engineers call an open system, other researchers would call naturalistic, or real-world, inquiry. A researcher must spend considerable time and effort putting a boundary around the “place” of their inquiry, much akin to the need to scope engineering problems. P ge 15343.4 A place can only provide productive inquiry if the researcher has access to the place. A researcher with access can enter the environment and record observations. When a researcher enters an environment, he or she should be unobtrusive so as to not interfere unduly with the social interactions present. Many,