Process Analysis as a Feedback Tool for Development of Engineering Problem Solving Skills

The goal of advancing personalized learning through tailoring instruction to meet students’ individual needs requires an analysis of the cognitive processes that lead to discrepancies during learning. This cannot be adequately accomplished using outcome-based scoring methods. The purpose of this research was to test and refine an evidence-based method of assessing students’ problem solving processes, and establish a tool that is robust enough to assess problems from a variety of contexts and formats. The assessment tool was developed using three well-defined story problems of various contexts for first year engineering students enrolled in an introductory course. Process analysis was applied to identify cognitive tasks used in successful solutions and common errors made in students’ problem solving attempts. These tasks and errors were mapped to a six-category problem-solving cycle, enabling the evaluation of the problem solving process, identifying underutilization of key tasks and errors committed in each stage of the process. Results of testing and refinement of the assessment tool are presented in this study. Percent agreement and Cohen’s Kappa statistics were calculated to assess inter-rater reliability between instructors using the assessment tool and to assess convergent validity between process analysis using the coding scheme and the modified process analysis using the assessment tool. The major advantage to the assessment tool over traditional grading methods is that it can provide personalized feedback to students about their level of problem-solving proficiency as well as pinpoint skill deficiencies that need attention.