Vos Data Manager: Providing Immediate Feedback On Teaching Effectiveness

New classroom assessment technologies produce large amounts of data. However, providing useful information to instructors, who may not be familiar with coding or analytic methods, remains challenging. For the past several years, the VaNTH (Vanderbilt Northwestern Texas Harvard-MIT) Engineering Research Center for Bioengineering Educational Technologies has employed the VaNTH Observational System (VOS). VOS allows minute-by-minute capture of both student and instructor activities during a classroom session via handheld personal data assistants (PDAs) and has been previously described. However, generating useful information for instructors based on this data is time-consuming, and instructors often wait days or weeks for the appropriate reports to be generated on a case-by-case basis. Furthermore, multiple files resulting from these methods have been difficult to organize and maintain. The VOS Data Manager has been developed to automate the process of importing, organizing, and analyzing VOS data. The import process checks the underlying data for validity, automatically correcting simple artifacts and identifying more complex ones for manual correction. The entire import process, including artifact correction, takes about five minutes for a one-hour classroom session. Once data has been imported, a variety of reports can be generated within seconds that detail instructional methods, student engagement (or lack thereof), and other classroom activities over a particular session. Additional reports show how one session compares to another, or to an average of other sessions given by the same or different teachers. Instructors can now receive feedback within minutes after a class ends, and researchers can more easily compare data across different sessions, instructors, and/or coders. The VOS Data Manager runs on any computer equipped with Excel (Microsoft Corp., Redmond, WA). Excel was chosen because of its wide install base, familiarity to most instructors and educational researchers, and the fact that it’s built-in macro language, Visual Basic for Applications, provides a high degree of functionality and compatibility across recent versions of the Windows (Microsoft) and MacOS (Apple Computer, Inc., Mountain View, CA) platforms. This technology facilitates rapid development and dissemination of the VOS Data Manager, allowing VOS researchers to provide immediate and useful feedback to instructors following a classroom session. P ge 9.409.1 Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition Copyright © 2004, American Society for Engineering Education Background Prior research suggests that professors will use formative feedback to make positive changes in classroom teaching 1 , and that self-reflection is an important tool for improving teaching 2,3 . Such improvements might include adopting different classroom management techniques 4,5 or other changes in teaching behavior 6 . Since quality of student learning can be evaluated by observing what occurs within classrooms 7 , an observational system that identifies specific elements of the classroom experience as correlated with student learning can be used to evaluate changes in classroom teaching, or to identify areas where change might prove beneficial. The VOS 8 , based in part on the Stallings Observation System 9 , was developed in 2000 to evaluate changes in bioengineering classroom instruction implemented as part of various VaNTH-ERC initiatives. Briefly, the VOS records classroom activity in four parts: (1) repeated 5-item code strings capturing student-teacher interactions, (2) numeric records of students’ levels of academic engagement, (3) free-text narrative notes describing lesson content and process, and (4) Likert-scale global ratings indicative of effective teaching. General aspects of classroom activities are captured (i.e. what the instructor is doing, if students are paying attention, etc.), as well as specific features of instruction related to elements of How People Learn 10 and use of multimedia technology. Observers use PDAs to record data continuously during the classroom session. Following the session data is downloaded from PDAs to a personal computer for analysis. Observations are stored in linked MS Access (Microsoft) tables corresponding to the parts of VOS described above, with each record corresponding to a single observation in a session. Data can be exported from Access to Excel or other programs for generating graphs or additional analysis. While the VOS is an effective data capture tool, with inter-rater reliability above 85% for observations made almost continuously over one-hour (or longer) classroom sessions 11 , data analysis can be cumbersome. Researchers must be familiar with technical tools and methods to create appropriate reports for instructor feedback, in addition to the substantial training and practice needed to code observations 12 . Producing meaningful reports for instructors, who may be unfamiliar with specific coding schemes or general tenants of learning science research, is complex; some coded fields, such as the number of students “on-task”, can be graphed over time directly, while others require grouping, normalization, or computation of derived measures based on combinations of coded data 13 . Finally, since recording data with VOS is a continuous process, if a mistake occurs observers usually simply start a new record rather than taking the time to fix the old one, leaving it incomplete. These occasional incomplete records need to be removed, or “cleaned”, from the raw data prior to analysis. Instructors often wait days or weeks for reports because substantial effort is required to process the coded observational data. As such, by the time an instructor receives a VOS report he or she may not recall details of that classroom session, making it difficult to relate observed effects to specific changes in teaching behavior or to other classroom events. Our primary goal in developing the VOS Data Manager was to automate the process of producing graphical reports so that instructors could receive them almost immediately following a classroom session. Delivering reports promptly to instructors increases the potential for VOS feedback to improve teaching and learning in their classrooms, P ge 9.409.2 Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition Copyright © 2004, American Society for Engineering Education and automated reports allow for wider use of VOS and realization of improved instruction both within and outside our institution. Methods: System Development Development of the VOS Data Manager began with a prioritized list of requirements: 1. Ease of use: Someone having basic familiarity with word processing or spreadsheet applications should be able to use the system without additional instruction. 2. Data repository: All VOS data should be maintained by the system and organized by session and instructor. The system should support shared use of a single data repository by multiple users. 3. Error handling: Any errors in source data (incomplete records, as described above) or program function should be detected and reported to the user. All errors should be automatically corrected if possible, with user confirmation where appropriate. 4. Extensibility: The system should be easily modified to include additional reports or other functionality. Advanced users should be able to define additional custom reports. 5. Windows and MacOS compatibility: The system should be usable on computers running either Windows or MacOS operating systems. 6. Batch data import/export: Data should be able to be imported to, or exported from, the system in a variety of standard formats. 7. Report archive: All reports generated by the system should be archived and accessible for future use. A number of different technologies were considered for development in light of these specifications, including use of “platform neutral” languages such as Java (Sun Microsystems), or a completely web-based application utilizing PHP (open source) or .net (Microsoft) scripts coupled with a database. Given resource constraints for this project, the most efficient solution was to implement the VOS Data Manager in Excel macros utilizing the Visual Basic for Applications (“VBA” Microsoft) language. While this choice did not completely satisfy all requirements (see Discussion below), Excel macros had significant advantages including substantial built-in data manipulation and graphing functions, a simple development platform, and compatibility between recent versions of Windows and MacOS. After selecting the development platform, user workflows were defined. The VOS Data Manager needed to support tasks related to the following: (1) importing raw VOS data, (2) producing reports, and (3) managing stored data. Workflow for importing raw data is straightforward; the user selects a file containing source data, provides ancillary information not captured by VOS such as the coder’s name, and instructs the system to proceed. Producing reports is more complex, requiring identifying the type of report, the session(s) to include, and various options for computing report data. For example, a report across multiple sessions might allow grouping results by instructor or date range, or normalization of results according to the number of sessions. Furthermore, once reports are generated the user may wish to edit, print, or save the report to an archive. The last group of tasks relate to managing stored data, as users need to be able to edit session information such as the instructor or coder, delete sessions, and Page 9.409.3 Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition Copyright © 2004, American Society for Engineering Education merge or split sessions. This workflow involves selecting the session(s) and desired operation, with