Final Results of Reliability Testing for the Norback-Utschig Presentation Scoring System and Implications for Instruction

In this paper we report the results of our final work completed to improve the reliability and usability of the Norback-Utschig Presentation Scoring System developed at Georgia Tech, and based on executive input. Our approach was the modified Delphi method, a multi-stage feedback process used to generate consensus among diverse stakeholders. The method was used to collect data for seven of the 13 presentation skills not yet having high reliability between raters: “initial and final impressions,” “logical flow,” “key points,” “layout and design,” “graphics,” “vocal quality,” and “personal presence.” Data was collected from a variety of individual stakeholders, and modifications were made to the scoring system. For example, “initial connection” was renamed to “first impression.” The definition of the following skills were clarified to make them easier to understand: “logical flow,” “key points,” “layout and design,” “graphics,” “vocal quality” and “personal presence.” Once the skills were modified, the new scoring system was tested for reliability in three settings—industrial engineering teaching assistants and the two developers of the scoring system, biomedical engineering teaching assistants and one developer, and a class of nuclear engineering seniors. Results indicate that the reliability between raters of all skills tested improved at a significant level. The revised skills now have good to high reliability. Implications for instruction will be discussed.