Evaluating Student Responses in Open-Ended Problems Involving Iterative Solution Development in Model-Eliciting Activities

Open-ended problems are an important part of the engineering curriculum because, when well designed, they closely resemble problem-solving situations students will encounter as professional engineers. However, valid and reliable evaluation of student performance on openended problems is a challenge given that numerous reasonable responses are likely to exist for a given problem and multiple instructors and peers may be evaluating student work. In previous work, evaluation tools for open-ended problems, specifically Model-Eliciting Activities (MEAs), were rigorously developed to ensure that the evaluation tools evolved with fidelity to characteristics of high performance and with increased reliability. As part of an on-going process of tool development, this study presents an expert evaluation of student work using the Spring 2009 version of assessment tools. The Just-in-Time Manufacturing MEA was implemented in Spring 2009 in a large first-year engineering course. A sample of 50 teams was selected for this study. Each of the teams‟ three iterative solutions was rigorously scored and coded by an engineering expert using the MEA Rubric and JIT MEA specific assessment supports. The expert scores were then compared across the three iterations, showing a positive trend of improvement in student performance across some dimensions, but little change in others. These findings have implications for instruction along each dimension. These findings also provide opportunities to investigate the nature of peer and GTA feedback that may (or may not) have resulted in change on a given iteration.