An Inclusive Process for Developing a Taxonomy of Keywords for Engineering Education Research

Engineering education research is a broad-based, rapidly-evolving, diverse, interdisciplinary, and international field. There is a clear need for a standardized terminology and organizational system – a set of keywords – to map the field and communicate research initiatives. Such a taxonomy could provide multiple benefits, including: better connections between research and researchers; more accessible research results, recognition for and identification of emerging research areas, a way to describe the diversity of engineering education research areas, and a common terminology with which researchers could frame their efforts. The goals of our NSF-funded project, then, are two fold: (1) to develop a comprehensive taxonomy of keywords that could be used to describe engineering education research projects, publications, authors, and peer reviewers, and (2) to establish an inclusive process for developing the taxonomy that invited the participation of the diverse global community. To achieve our project goals, we used an iterative process to develop and refine a taxonomy of keywords which involved a variety of stakeholders in multiple settings. We were assisted in our efforts by a professional taxonomist and supported by an advisory board that included ten engineering education leaders and editors of several engineering education journals. We intentionally designed an inclusive process, involving faculty, administrators, and graduate students from across the world, and we used this opportunity to develop a globally-relevant taxonomy for use in the United States. We issued open calls for participation and advertised broadly for key opportunities to be involved. We created transparent selection processes, criteria that encouraged diversity, and engaging activities that de-emphasized hierarchy and privilege of particular research areas. At workshops and conference sessions, we adapted activities to be relevant in the local context for engineering education research. Participants were engaged in creating initial keyword outlines and in refining draft versions of the taxonomy at a two-day conference and at a series of four subsequent conference workshops (American Society of Engineering Education Annual Conference, Research in Engineering Education Symposium, European Society for Engineering Education, and Frontiers in Education Conference). After the first two workshops, nine graduate students worked on three virtual teams to further refine the taxonomy. Over 220 individuals provided additional feedback that improved the taxonomy. Here, we describe participant selection, details of each activity, and outcomes for each phase of the project. The sections below detail the project stages in chronological order, including participants and selection procedures, details of the stage and its outcomes. Table 1 provides participant details, and Figure 1 maps the home institutions of participants. Page 24166.2 Table 1. Details and participant demographics for taxonomy activities Activity Location Date Number US participants Number non-US participants Total participants Initial keyword outlines N/A March 2013 – April 2013 3 3 6 Mapping the Field Conference Ann Arbor, Michigan, USA 05/20/13 – 05/21/13 39 9 48 Workshop at ASEE Conference Atlanta, Georgia, USA 6/25/2013 35 6 41 Workshop at REES Kuala Lumpur, Malaysia 7/6/2013 12 47 59 Virtual workshop N/A August 2013 8 1 9 Workshop at SEFI Conference Leuven, Belgium 9/20/2013 3 54 57 Workshop at FIE Conference Oklahoma City, Oklahoma, USA 10/25/13 29 6 35 Total unique participants* 107 117 223 * Some individuals participated in more than one activity, so the number of unique participants is not a summation of the overall participant count Figure 1. Home institution for 223 unique participants. P ge 24166.3 Initial keyword outlines Participant selection: The project began with an international, open call for initial keyword outlines. From the proposals submitted, we selected three teams (six individuals total) to provide keyword outlines as the basis for the initial taxonomy. Keyword outline details: The three outlines, each consisting of keywords arranged in a two-level outline, were developed differently. 1. Outline A was derived in two phases. First, the authors analyzed bibliographic data (i.e., title, abstract, and keyword list) for nearly 900 international journal articles and conference papers from 2005 to 2008. They iteratively created a set of first-level terms (each having a clearly identifiable research area) and second-level terms. Next, from over 500 relevant journal articles and conference proceedings, the authors expanded their analysis by creating a more comprehensive set of keywords for one of the 38 categories (diversity). Their resulting outline contains 1,242 keywords categorized into 38 first level terms. 2. Outline B was based on 2,216 peer-reviewed journal articles in five international engineering education journals from 1959 to 2012. The author studied the titles to identify an initial set of keywords then applied network analysis to identify underlying themes and relationships between them. The outline includes 256 keywords, arranged in 46 first-level terms and multiple secondlevel terms, as well as a time dimension and a level of connectedness (frequency of occurrence). 3. Outline C involved a keyword analyses of three separate sources: (1) the full text of research or discussion papers from journal articles during 2006-2012, conference papers from 2008-2012, and a pedagogic research statement for the field, (2) personal reflections on three related conferences and a study relevant PhD theses; and (3) meeting notes of an engineering education research special interest group from 2009 – 2013. The final outline includes 13 first-level terms and 43 second-level ones. Outcome: From the three commissioned keyword outlines, Access Innovations (the nation’s largest taxonomy creation firm) created a draft taxonomy. They integrated and refined the outlines, normalized terms, and worked to consolidate the two-level structure into a more hierarchical one. The resulting taxonomy (version 1) included 1,079 keywords arranged in 13 branches and seven levels. Mapping the Field Conference Participant selection: The Mapping the Field Conference was a two-day workshop to refine the taxonomy of keywords and the accompanying organizing structure. Email invitations were sent to several groups, including the Research in Engineering Education network, International Federation of Engineering Education Societies/Global Engineering Deans Council, and the Educational Research and Methods Division and the Engineering Libraries Division of the American Society of Engineering Education. The invitation included a link to a brief application consisting of an activity for individuals to list keywords to describe their main research area and two questions: (1) What areas of engineering education are in most pressing need of additional research and (2) How will you contribute to diversity of perspectives at this workshop? We selected 48 conference participants from 96 applicants. We aimed to be as inclusive of the community as possible, and we selected a group of researchers and other stakeholders based on a variety of factors including: P ge 24166.4 diversity of experience and history in the field (selecting a mix of participants ranging from graduate students studying engineering education to experienced practitioners interested in transition into the field to researchers with extensive experience in engineering education research), diversity of research area and understanding of the field based on application responses, and diversity of perspective, especially as related to international and national backgrounds. Invited participants included nine international colleagues and 39 individuals from the US, representing 40 distinct institutions from across the world. They have broad experience levels and ranged from current graduate students and faculty new to the field to seasoned veterans with significant experience. Conference details: The Mapping the Field Conference was held May 20-21, 2013, in Ann Arbor, Michigan. Participants reviewed sub-sections of the taxonomy in advance of the conference and worked to further refine version 1 of the draft taxonomy. At the conference, a professional taxonomist from Access Innovations presented an introduction to taxonomies and an overview about using and maintaining taxonomies. Participants engaged in multiple breakout sessions where they discussed subsections of the taxonomy, suggested term-by-term revisions, and made high-level revisions to the overall taxonomy. Outcome: Access Innovations worked with the documents produced at the conference, and through an iterative process, we produced version 2 of the taxonomy. It included 971 total terms, arranged in 14 branches and six levels. Annual Conference of the American Society of Engineering Education Workshop details: We held a pre-conference workshop at the 2013 American Society of Engineering Education Annual (ASEE) Conference in Atlanta, Georgia on June 25, 2013 [1]. Any individual attending the ASEE conference was invited to participate in the pre-conference workshop. During the three-hour session, 41 participants reviewed version 2 of the draft taxonomy, using and revising it during two separate activities. For the first activity, we compiled six sets of five abstracts, selected from ASEE 2012 abstracts and abstracts from recent manuscripts published in Advances in Engineering Education, International Journal of Engineering Education, and Journal of Engineering Education. Working individually, then in small groups, participants were asked to “apply the taxonomy” by identifying four to six keywords that described each abstract. They also identified missing terms and offered suggestions for making the taxonomy easier to use. After gaining familiarity with the taxonomy, participants