Can structured reflection enhance learning in a heat and mass transfer course?

This paper presents a quantitative and qualitative study of written reflective exercises and normalized gain scores from a concept inventory assessment in a junior-level heat and mass transfer course for chemical engineers. The primary purpose of this research is to determine to what extent written reflection activities are successful at adjusting commonly-held misconceptions students have about heat transfer. As described in a previous ASEE paper (Chenette and Ribera, 2016), the authors conducted a series of prediction activities in several course sections, each with approximately 25 junior-level chemical engineering students, with either a structured follow-up reflection assignment or no structured reflection assignment after each prediction activity. The Heat and Energy Concept Inventory (HECI) was administered to students of all sections at the start and the end of the course. The overall HECI score along with HECI subcategory (Temperature vs Energy, Temperature vs Feeling, Rate vs Amount, Radiation) scores were used to evaluate learning gains. Archived data from classes with no prediction activities and no reflection activities served as a control group. To explore if the quality of reflection is related to learning gains, student reflections were ranked according to a validated rubric and compared with quantitative data on learning gains. Qualitative contributions include student responses from focus groups and student surveys. Key highlights will be discussed to provide a better understanding how the students’ perception of learning is affected by these activities. Preliminary results showed a weak correlation between the normalized gain score of individual students in the Rate vs Amount subcategory and the quality of reflection displayed by each student in the follow-up reflection activity. This paper analysis of another HECI subcategory relevant to the in-class prediction activity on radiation. This analysis also includes an additional cohort of students in an effort to increase the sample size to address limitations of the preliminary results. The goal of this study is to better direct the role of prediction and reflection activities in fundamental engineering courses. Introduction As other scholars have shown, including prediction activities in university-level heat transfer courses enhances conceptual understanding of heat transfer concepts among chemical engineering students5,6,9. Additionally, guided reflection is being used in engineering education to elicit deeper understanding from an experience8. This study compares students among different classes to determine to what extent reflective activities in combination with prediction activities impact shifts in conceptual knowledge of heat and mass transfer. The theory upon which this study is founded includes literature on conceptual change3, inductive-learning as a form of active-learning4, and reflection7. A detailed literature review of these subjects can be found in a previously published work-in-progress ASEE paper (Chenette and Ribera, 2016)1. Methodology This study expands upon a preliminary study that aimed to uncover the extent to which structured written reflection activities play a role in facilitating conceptual change for students in a fundamental heat transfer course. Established instructional methods based on inductivelearning guided the prediction activities used in this study5,9. This experimental design is an extension of the preliminary work, and is considered quasiexperimental (students were not randomly assigned to different sections). It includes traditional instruction (X1), in-class prediction activities (X2), and written reflection activities (X3), across various cohorts, as shown in Table 1. A preand post-test HECI (O1) and an optional focus group (O2) provide quantitative and qualitative evidence to describe the effect of these activities on student experience. Students in the control group (Class A) did not participate in the prediction nor reflection activities. Class C all received the same experimental conditions, however distinctions are made for the following reasons: Class C2 and C3 were not a part of the preliminary study (REF omitted), and a different instructor taught Class C3. Three 25-minute prediction activities were spaced throughout the 10 week course. As mentioned earlier, established instructional methods guided the structure of these activities (REF omitted). Briefly, the three activities centered on the topics of conduction, convection, and radiation. Sample lesson plans can be found in the Appendix of this paper. Each prediction activity began with the instructor explaining the demonstration and asking a question about what would happen in the system. Students wrote down their predicted answer with some justification. A brief demonstration followed, allowing students to observe what actually happened. The instructor concluded the activity with a 5 minute explanation of the theory governing the system. Within a week of the in-class activity, students in Class C completed a brief set of structured questions, aimed to engage students in reflection. These follow-up activities are in the Appendix. Table 1: Experimental Design. The same HECI test was administered preand post-instruction as a quantitative form of observation (O1). In addition to traditional instruction (X1), the instruction received by Class B also included prediction activities (X2) and instruction in Class C also included reflection activities (X3). A voluntary focus group was assembled of students from Class B and Class C. The classes were all taught by the same instructor except for Class C3. Sample HECI test Traditional instruction Prediction activities (3) Reflection activities (3) HECI test Focus group (optional) Class A (control) O1 X1 O1 Class B O1 X1 X2 O1 O2 Class C (1,2,3) O1 X1 X2 X3 O1 O2 Assessment Assessment methods for this study are the same as those published previously (REF omitted). Here, a brief overview of these methods is presented, and the reader is encouraged to review the prior work for justification details and supporting references. Quantitative learning gains were assessed using the Heat and Energy Concept Inventory (HECI), which was selected for its relevant subject categories and its established internal consistency reliability and content validity 5,6,9. The HECI was administered preand postinstruction for all students in this study. Witten follow-up reflective exercises were ranked using a four-category rubric developed by Kember and colleagues that has since been validated2. The four levels of reflection identified in this rubric are: habitual action/non-reflection, understanding, reflection, and critical reflection. Each reflection activity (student name and course information removed) was independently reviewed and ranked by 2 or 4 faculty members. To facilitate this process, an online survey was created specifically for this study in the form of an online survey. Moodle, an online learning management system, hosted the questionnaire, which is the name given to an activity which allows teachers to create questions and receive feedback from students. The questionnaire was developed specifically for this study and provided reflection response text, a reminder of the rubric definitions, and fields to enter their numerical ranking and additional comments, if so desired. Faculty were trained to use the rubric and reviewed sample reflections to assist with inter-rater reliability. The developers of the rubric indicate that intermediate categories may be used. For this study, faculty reviewers did not assign intermediate categories, however average rankings across the multiple reviewers were rounded to fall into the four categories or intermediate categories (1.0, 1.5, ..., 3.5, 4.0). To complement quantitative assessment, qualitative responses from a survey administered by the Consortium to Promote Reflection in Engineering Education (CPREE) were gathered to provide supplementary qualitative data obtained in earlier work from a focus group. Nine students from Class C1 completed the survey, which asked about the impact of these activities on students. Results and Discussion One-way ANOVA (analysis of variance) tests were conducted to compare differences in the mean overall pre-test and post-test scores for the HECI among each student cohort. Differences were significant at a p < 0.001 level for every class except Class A (p > 0.05). Figure 1 displays mean preand post-instruction scores on the HECI for each class. Figure 1: Overall HECI Score Comparison. Mean HECI score is plotted for each class: Class A: X1 (traditional instruction); Class B: X1 and X2 (prediction activities); Class C1, C2, C3: X1, X2 and X3 (reflection activities). Error bars represent SD. Average normalized gain scores for this assessment were also calculated for each class for the overall HECI as well as individual subcategories, as shown in Table 2. Defined as the ratio of actual average gain to the highest average gain possible, student scores that increase significantly between the preand post-assessment show a high normalized gain. A decrease in an assessment score results in a negative normalized gain. Normalized gain scores may be further classified as “high-g” where () ≥ 0.7, “medium-g” where 0.7 > () ≥ 0.3, and “low-g” where () < 0.3. Table 2: Average Normalized Gain Scores by Category and Overall. Normalized gains for individual students were averaged among each class. Temp v Energy (10 items) Temp v Feeling (9 items) Rate v Amount (8 items) Radiation (11 items) Overall (36 items) Class A (n=34) 0.01 0.06 0.09 0.12 0.08 Class B (n=27) 0.21 0.40 0.23 0.44 0.33 Class C1 (n=18) 0.19 0.12 0.39 0.42 0.34 Class C2 (n=20) 0.32 0.41 0.36 0.54 0.44 Class C3 (n=14) 0.23 0.32 0.18 0.44 0.32 These data support the conclusions drawn from analysis of the raw HECI score, and show that no distinguishable difference is observed between clas