The Integration of Novel Forms of Feedback into Software Engineering Courses

Software Engineering students exhibit a wide array of learning styles across the perception, input, organization, processing, and understanding dimensions. To improve students’ performance in the classroom, many techniques have been developed to address these variances. Active learning has a long history of usage in the software engineering classroom, and the disciplines strong history of diagramming and visual representations have been very supportive to the large percentage of students who are visual learners. Coaching by faculty in project based courses also is common in the discipline. However, when it comes to providing feedback to students on formally submitted assignments, the main method employed is still written comment, which is not conducive to visual learners. Written comments are embedded in source and marked on design diagrams, using annotations or colors to distinguish them from the original work. This method is most prevalent in the community because overall, it is the simplest form of feedback that a faculty member can provide to students. However, written feedback is often highly ineffective at improving student performance, as many students simply do not read the comments because the students feel they are not relevant to their performance. This survey paper presents an assessment of an alternative methods of providing feedback to students using audio-visual techniques. In lieu of written feedback, students are provided feedback for software engineering exercises through the use of short multimedia files. The article describes the pedagogical foundations for the technique, specifics of the technique used, student perceptions of the technique, and an assessment of the learning gains from using such a method across several software engineering courses. In general, students are shown to prefer the technique versus traditional grading, and an improvement in overall outcomes for the course is shown to exist as well. Introduction The students in software engineering are similar to those in computer science in exhibiting slightly different and stronger learning style preferences than those in other disciplines. Most students in the computing fields can be classified as reflective sensing sequential visual learners. In the classroom, instructors adapt to this. Courses taught in computer science encourage active learning and interactive exercises. In class, concepts are explained visually through diagrams and pictures, even though the ultimate solutions manifest themselves in textual form as source code. However, when it comes to feedback in the computing field, the variety received is not as great. This is unfortunate, for feedback has overwhelmingly been shown to be the single most powerful influence on student success. Hounsell states: “It has long been recognized, by researchers and practitioners alike, that feedback plays a decisive role in learning and development, within and beyond formal educational settings. We learn faster, and much more effectively, when we have a clear sense of how well we are doing and what we might need to do in order to improve.” Appropriate feedback is difficult under the best of circumstances. Feedback must be delivered at the appropriate time and be applicable to the student. If it is delivered too late, it will not be of benefit to the student. Feedback must be engaging to the student as well. If the student does not look at the feedback, learning cannot occur. Last, the feedback itself must be relevant and customized. Generic feedback delivered to multiple students may be ignored by the student that needs that feedback the most. Traditionally, even in the computing fields, the most common mechanism for feedback is the handwritten comment. On preliminary programming assignments, hard copies code will be marked up with comments about the mistakes that were made or electronic copies will be annotated with special grading comments. Design submissions again will be marked up with handwritten comments, scribbles about the design problems or the potential issues in the designed system. Requirements documents are marked up in the same manner. If a program fails during program execution, the student is provided with a cursory note indicating the failure or maybe a static screen shot is returned. These techniques apply regardless of the medium, be it paper, marked up pdf documents, or typed comments in submissions in a LMS system. Overall, the main format of feedback is still in the written format. This format generally provides mechanism provides for an ineffective, static monologue between the instructor and the student. In past studies, it has been shown that for many problems, alternative forms of feedback outside of the written comment can be more effective, the best effectiveness often comes from providing feedback in a variety of modes. Verbal feedback is often employed in a mentorship manner. Most software engineering programs have some form of a capstone project where an advisor or mentor routinely meets with project teams and discusses their project. This mentorship helps students tremendously. Oral presenters are often “coached” by an expert on how to improve their presentation. Medical students are videotaped while performing consultations and then receive comments from their instructors and peers when the videos are played in public. In the software engineering field, we use this approach to provide feedback to students on their oral presentations, videotaping them and critiquing them post presentation. The usage of active learning activities also aids software engineering students. Formal inspections, for example, are traditionally used on software engineering projects to improve quality. However, in the classroom, formal inspections can be used to teach students both how to follow a disciplines software development process and as an active learning exercise to improve projects. However, many tasks given to students in the software engineering field are still traditional assignments, graded using a traditional mechanism, and using only written feedback. How can the effectiveness of these assignments be improved? Audio feedback on assignments has been used traditionally in the performing arts fields, where a musician or artist is critiqued in real time by an adjudicator. This approach has been tried in sciences and the impact has been quite positive. Merry and Orsmond provides details whereby feedback was given to students’ as mp3 files in a biosciences course. Students felt that this was a better mechanism to receive feedback in. The students felt that the feedback had more depth, was easier to understand, and was much more personal. Stroud et al. found that students preferred audio feedback over written comments, especially if the students were either undergraduate. Wallace and Moore indicated that 80% of online students felt that their learning experience was significantly enhanced with audio feedback. While audio feedback certainly represents a different modality than the traditional written comment, in the computing field, it may not be as effective for the visual learner. Depending on the length, students could easily perceive the feedback as monotone in nature, and depending on the deliverable, may not be able to easily translate the verbal feedback into a physical problem or mistake. This leads to the next natural progression beyond audio feedback, namely multimedia feedback. With multimedia feedback, feedback is provided to students through a short video recording the events which transpired during a grading session. In the case of computer programs, this may be program crashes which the students didn’t expect, test cases which resulted in unexpected behavior, or code that may or may not compile as submitted. The video provides irrefutable evidence to the student of these scenarios, which again leads to more effective feedback. Video Feedback Multimedia feedback starts with audio feedback as its basis and adds a visual component to the audio stream. This approach tends to help visual learners, as they tend to obtain a greater understanding through the usage of pictures, charts, diagrams, and other visual representations. Many people have proposed the concept of video feedback outside of engineering, as it has been discussed in numerous blogs. Outside of engineering, there also have been a few conceptual papers published with limited assessment. Cherry et all discussed the application of a tool called video traces to aid dance students in improving their dancing skills. With this tool, a video of students dancing is annotated by the instructor and then reviewed by the students, allowing them to see problems in their dancing. Thompson and Lee reported on a technique they call Veedback, which is used to capture their comments. Overall, they indicate that one of the challenges faced by students is trying to figure out how to apply the new form of feedback that is available to them. Jones et all reported on a similar approach in a business environment. They indicate through a mixed study that there are advantages to the approach over traditional feedback mechanisms, students like the new form of feedback, and that this approach seems to encourage a more constructive dialog between the grader and the students, increasing student engagement. Crook et all report on the development and application of the ASEET System. This system helps faculty and students to increase engagement through the usage of videos for custom feedback. As with the previous studies, this study showed that there was an improvement in student engagement, and overall, both the staff and students felt that the system was worthwhile. The general process for asynchronous multimedia feedback is shown in Figure 1. Students submit their assignments in a traditional manner using a course management system, in this case, Blackboard. The instructor then grades the assignment, capturing in real time