Utility of Post-Hoc Audio Reflection to Expose Metacognition and Strategy Use by First-Year Engineering Students for Different Problem Types

This work-in-progress paper identifies metacognitive activities and problem solving strategies utilized by first-year engineering students to solve different types of problems. Our research question is: What problem solving strategies and metacognitive activities are revealed by students’ post-hoc audio reflections on their solutions to three different types of engineering problems (story, open-ended, exercise)? This study was conducted with first-year engineering students at our institution over two semesters. Students solved problems using tablets and custom-designed software that recorded student written work and erasures. Students then would watch playback of their problem solution and insert verbal comments into their work (post-hoc audio reflection). Analysis of students’ written data did not reveal much about metacognition and only afforded minimal insight into strategy use. Audio data was then analyzed for strategy use and types of metacognitive activity for the three problem types. Our analysis suggests that students do employ different strategies for different problems; openended and story problems are more likely to elicit metacognition in general, and few students articulated their thinking for exercise problems. Most metacognition occurred in the form of monitoring and evaluating, with little evidence of planning. Several strategies are used by students for all problems, but some are unique to specific types of problems. These findings demonstrate the usefulness of post-hoc audio reflection in engineering education research to better assess and address students’ metacognition and problem solving strategies. Introduction This work-in-progress paper identifies metacognitive activities and problem solving strategies utilized by first-year engineering students to solve different types of problems. Our research question is: What problem solving strategies and metacognitive activities are revealed by students’ post-hoc audio reflections on their solutions to three different types of engineering problems (story, open-ended, exercise)? Post-hoc audio reflection is defined as verbal student commentary recorded as students watch a play-back of their work after solving a problem. It is difficult to discern cognitive problem-solving skills that are developed and those lacking in engineering students based only on their written work. Although studies have been done to explore the value of metacognition through think-alouds and post-hoc written reflection, little research exists exploring what can be gleaned about student strategy use and metacognition through post-hoc audio reflection. Moreover, existing research has largely overlooked the potential for different problem types to elicit diverse strategies and metacognitive activities from students. Research in this area could help address difficulties faced by first-year engineering students with diverse backgrounds and needs. Students who begin the program with weak mathematics preparation already find themselves at a disadvantage compared to their peers with more experience in mathematics. By identifying strategies and metacognitive activities that are most essential for success in solving engineering problems, we can advise ways to mediate these discrepancies between students and help retain engineering students who are still developing math proficiency. Prior research that focuses on differences between experts and novices based on the ability to know and apply core principles of a field asserts that engineering education often fails to provide students with opportunities to solve authentic problems. Consequently, students are not given practice utilizing metacognitive strategies that are necessary for solving ill-structured problems. The goal of this study is to differentiate metacognition in terms of planning, monitoring and evaluation during problem solving for different problem types. This study was conducted with first-year engineering students at our institution over two semesters. Three different problem types, story, open-ended, and exercise, were selected for this study, with the understanding that different problem types elicit different problem-solving processes. Students solved the problems using tablets and custom-designed software that recorded students’ written work and erasures. Students then would watch playback of their problem solution and record verbal comments at relevant points within their work (self-selected based on what they wanted to reflect on). Although analysis of students’ written data allowed detailed documentation of students’ problem solving processes, it did not reveal much about metacognition and only afforded minimal insight into strategy use. Audio data was then analyzed for strategy use and types of metacognitive activity for the three problem types. The implications of this research concern scaffolding for metacognition, strategy use and skill development in engineering education. Educators benefit from considering these findings when developing interventions for struggling students or when trying to develop approaches to solving problems that more closely align with those used by practicing engineers. Literature Review An issue of concern for post-secondary engineering educators is that of retention; only about half of the incoming freshmen in engineering graduate with a degree in the field. Particularly noteworthy is the attrition of students from nontraditional backgrounds. Studies aimed at understanding student characteristics that correlate with leaving the field produce mixed results, probably because student outcomes are the result of a confluence of factors, although students who leave are more likely to feel less prepared in math and science than their peers . For retention rates to improve, barriers to completion need to be identified and addressed. Among identified barriers is the lack of awareness young scholars have regarding what work in their selected field actually looks like and a paucity of resources for students who need help transitioning to the demands of engineering education. Because attitudes at the end of the first year in the program have strong predictive value in successful outcomes, it is valuable to find ways to make the college experience a more successful transition between high school and the workforce. One way to do that may be to focus on the self-management skills and strategies needed by engineers, identify those that are lacking in freshman engineering students, and develop classroom experiences that address this discrepancy. Problems faced by engineers are typically ill-structured, meaning that they do not have a clearcut answer or single approach to finding a solution. However, engineering education has failed to provide students with authentic experiences with these types of problems; instead, students see problems that are highly structured. To prepare students for what they will encounter after college, engineering educators need to emphasize deeper understanding of the core concepts underlying the field of engineering. A metacognitive basis for instruction may help students to develop problem solving approaches that will prepare them for success in dealing with illstructured problems. However, many students enter their post-secondary engineering studies with poorly-developed metacognitive skills, and in its current state, educational experiences do little to address this shortcoming. Explicit education on strategies addressing metacognition have been shown to be beneficial in improving outcomes for students , even in cases where content-specific knowledge is weak. Some work has already been done to test the idea of using metacognitive interventions in college science settings. For example, Sandi-Urena et al found that small-group discussions that prompted for metacognition helped chemistry students solve ill-structured problems. Remedial math students improve through explicit instruction on thinking strategies and frequent teacher feedback . Various forms of metacognitive intervention have been tried as well. Hanson and Williams saw gains in student performance through written metacognitive assignments, and McLaughlin suggested that the use of student-created podcasts fosters skill development. Existing research has left a few important questions about approaches to problem solving unanswered. Problem solving can be studied in the phases of planning, monitoring, and evaluating. Schraw defined planning as identifying resources needed to solve the problem, monitoring as assessing how well the task is understood and approached, and evaluating as determining the overall value of the work done. Ku and Ho studied activity in the three stages using general critical-thinking tasks, which they defined as tasks requiring “strategic use of cognitive skills that best suit a particular situation, as well as active control of one’s own thinking processes for well-justified conclusions” (p. 253). Using think-aloud to measure metacognitive activity, they found discernable differences between students with more critical thinking aptitude and activity in each of these stages. Students with high performance had more advanced planning strategies, more frequent monitoring activity, and a greater tendency to evaluate their work. However, similar work has not been done to examine activity in relation to problems specifically designed for engineers. Moreover, literature differentiating metacognitive activity in solving ill-structured versus well-structured problems in engineering classes is notably sparse; research largely neglects to differentiate the diversity of problem types and how they may elicit unique metacognitive activity. Research aimed at addressing these gaps could provide engineering educators with information to help them better serve students entering the program, particularly those with weak content foundations, to develop skills to become successful engineering students and to b