Successful Use Of Rubrics To Assess Student Performance In Capstone Projects

Capstone Experience, MTC 420, is a required course for all Mechanical Engineering Technology (MET) students during their senior year. The capstone projects are intended to be culminating experiences, drawing upon a wide range of knowledge from courses in the curriculum. Students are responsible for written project specifications, planning and milestone identification, implementation of the work, an oral presentation, and a final written report. Because these projects are so comprehensive, they provide an opportunity for faculty to assess a wide range of student learning that is directly related to program outcomes. For this reason, MET faculty developed a rubric for assessing capstone projects, as shown on the following page. The instructor rates each project in terms of ten attributes on a scale of one to four, providing a direct measure for assessment of program outcomes. The scale of one to four (1-Not Acceptable, 2Below Expectations, 3-Meets Expectations, 4-Exemplary) was intentionally chosen so that the instructor had to make a clear decision as to whether or not expectations were met. The rubrics were administered at the end of each semester, fall 2007 through spring 2009, and results were tabulated and analyzed to identify areas which were satisfactory and areas needing improvement. The measure of successful performance in each attribute was the percentage of students achieving a level of three or four. A benchmark of 70% was used to gage the level of success. If 70% of students scored a three or four, then the program was considered successful for that attribute. An interesting side-effect occurred when students were given copies of the rubrics at the beginning of the semester in fall 2008. This raised their awareness of expectations, especially in the non-technical areas such as teamwork and communication skills. A simpler rubric was also used to evaluate students’ mid-semester oral presentations. After the presentation, the instructor completed the rubric for each student using a binary scale (0unsuccessful, 1-successful). The measure of satisfactory performance in each category was the percentage of students scoring “1.” A benchmark of 70% was used to gage the overall level of success. If 70% of the students were successful, then the presentations were considered successful. Results from this rubric gave the instructor and students feedback, and the results also provided a basis for monitoring improvements that occurred during the second half of the semester. The levels of success in the capstone rubric were documented and correlated with levels of achievement for six program outcomes for four consecutive semesters. In fall 2009, these results were accepted by TAC/ABET as part of the continuous improvement program in MET. Note that the rubrics were not used directly to assign grades; however, results gave the instructor useful insight for evaluating the projects and assigning grades. Although the rubrics took some initial effort to develop, it is now fairly easy to complete and analyze the results each semester as a routine part of the ongoing assessment processes. P ge 15144.2 Introduction and Overview Rubrics have been used in general education courses for many years. They have proven to be very helpful for grading written papers and oral presentations, particularly for large classes or multiple sections. Recently, rubrics have been adapted for assessing student learning in technical courses. 1,2,3 Similar work has also been done on evaluating capstone projects using an industrial scale. 4 Because these projects are so comprehensive, they provide an opportunity for faculty to assess a wide range of student learning that is directly related to program outcomes. For this reason, MET faculty developed a rubric for assessing capstone projects. The objective of this paper is to present this rubric, the data collected over two years, and an analysis of the results. A blank rubric, shown in Table 1, is a one-page paper form that is completed by the instructor at the end of the semester. The instructor rates each project in terms of ten attributes on a scale of one to four, providing a direct measure for assessment of program outcomes. The scale of one to four (1-Not Acceptable, 2-Below Expectations, 3-Meets Expectations, 4-Exemplary) was intentionally chosen so that the instructor had to make a clear decision as to whether or not expectations were met. Results and Discussion The instructor completed a rubric for each student at the end of the semester, and results were tabulated and analyzed to identify areas which were satisfactory and areas needing improvement. The rubrics were administered in fall 2007 and spring 2008 to gain experience in analyzing results, and it was repeated in spring 2009. The measure of successful performance in each attribute was the percentage of students achieving a level of three or four. A benchmark of 70% was used to gage the level of success. If 70% of students scored a three or four, then the program was considered successful for that attribute. Table 2 summarizes the results for fall 2007. Six of the attributes were above the benchmark of 70% (depth, timeline, organization, discussion, oral presentation, and written report.) Four of the attributes below the benchmark are shown in boldface (innovation, methodology, references, and teamwork.) These attributes are below expectations and represent areas for improvement. Table 3 summarizes the results for spring 2008. Eight of the attributes were either at or above the benchmark of 70% (depth, innovation, timeline, organization, methodology, discussion, oral presentation and written report.) Two of the attributes below the benchmark are shown in boldface (references and teamwork.) This indicates an improvement, compared to fall 2007, which is probably due to the fact that students were given a copy of the rubric at the beginning of the semester, so they had a better understanding of the expectations and standards. Although the rubric was initially developed to assess outcomes, it was found to increase the quality of students’ work on the projects. P ge 15144.3 Table 1. Rubric for MET Capstone Projects Student Evaluator Semester Attribute 1-Not acceptable 2-Below expectations 3-Meets expectations 4-Exemplary Score Technical depth Little use of collegelevel skills, unclear proposal Lack of technical content, proposed goals were not completed Use of skills from junior and senior courses, proposed goals completed Advanced insight, exceeds goals of the project Innovation Not original, simple, content limited to lower-level courses Limited scope, reproduces existing concepts/analyses Applies original ideas, novel design, insightful Highly innovative, thorough investigation, sophisticated Timeline Lacks self-ambition, delays proposal, weeks with no progress Requires prompting, procrastinates, rush at end of semester Consistent effort throughout semester, documented milestones Superior effort throughout, meets or exceeds goals, expands project scope Organization, neatness Illogical, sloppy, unclear Awkward, difficult to follow Logical, well documented Highly professional, textbook quality Methodology Excludes data, incomprehensible, extremely vague, unclear Presents data without explanation, does not question data, analysis is flawed or inappropriate Clearly evaluates data, thoroughly explains procedure, minor errors or omissions Justifies decisions, documentation is complete, correct, and appropriate Discussion Misrepresents, draws incorrect or no conclusions, lack of understanding limited insight, misses key issues, inconclusive Identifies critical issues, suggests improvements Thorough evaluation, unique insight, examines inconsistencies References, resources Does not collect external information, irrelevant sources, plagiarism, dishonesty Insufficient research, limited use of sources, inadequate background research Presents useful information of sufficient quality and quantity, correctly formatted Collects extensive relevant information from a wide range of sources, validates findings Oral presentation Brief, fails to persuade, lack of adequate illustrations, disjointed Misconstructions, unclear, poor illustrations, focus on others’ work Persuasive, clear communication, effective use of illustrations Succinct, clear, use of multimedia, coherent, focus on new understandings Written report Grammar errors, misspelling, brief, misrepresents information Poor grammar, excess verbiage, insufficient detail Grammatically correct, thorough explanations, straightforward Excellent blend of explanations and illustrations, full detail Teamwork Works alone, argues without resolution, unwilling to cooperate, does not complete tasks Unequal contributions, overreliance on others, needs reminding Contributes equally, cooperates, works toward group goals, self-motivated Values and encourages all members, coordinates efforts, provides appropriate leadership