The Significance And Limitations Of The Vanth Observation System Within Engineering Classrooms

This paper explores both the significance and the limitations of the VaNTH Observation System (VOS). The VOS was originally developed for use at the VaNTH Engineering Research Center in bioengineering classrooms at Vanderbilt University, Northwestern University, the University of Texas at Austin, and the Harvard/Massachusetts Institute of Technology Division of Health Science and Technology. This instrument is designed to capture (1) the number and dura tion of faculty-student interactions in a classroom, (2) the engagement of students during a lesson, (3) the lesson content, lesson context, and extenuating circumstances in a classroom, and (4) the global aspects of a class lesson. The VOS is a significant assessment tool because it identifies both the time and the duration of student group work that is occurring within a classroom, it parses out faculty and student initiation of higher order learning skills, and it reflects various elements of current “How People Learn” (HPL) learning theory4. In addition, the VOS captures differences among faculty’s teaching styles and identifies the effects of a classroom’s physical layout upon a lecture. Moreover, VOS generates detailed feedback which faculty may use to self-assess their pedagogical skills. However, limitations still exist. Yet despite these limitations, the VOS is a tool that provides qualitative and quantitative classroom observation data and shows how faculty teaching styles might affect student outcomes. Background and Introduction In the fall of 1999, bioengineering and learning science faculties at four universities (Vanderbilt University, Northwestern University, the University of Texas at Austin, and the Harvard/Massachusetts Institute of Technology Division of Health Science and Technology [VaNTH]), received a National Science Foundation grant to support a collaborative effort within an Engineering Research Center (ERC) for bioengineering educational technologies. A main goal of this ERC is the improvement of teaching in bioengineering. Collaborators within the VaNTH assessment and evaluation thrust worked together to develop the VaNTH Observation System (VOS), an assessment tool that captures (1) the number and duration of faculty-student interactions in a classroom, (2) the engagement of students during a lesson, (3) the lesson content, lesson context, and extenuating circumstances in a classroom, and (4) the global aspects of a class lesson. This paper explores both the significance and the limitations of the VaNTH Observation System (VOS) within bioengineering classrooms. Although the VOS is in its implementation phase, the authors hypothesize that the VOS will be significant. It has the capability to measure the integration of “How People Learn” learning theory4 into class lessons in the various domains of bioengineering and because of its ability to provide an empirical measure of postsecondary classroom and teacher activities and create a system that, when used effectively, can provide P ge 8.165.2 Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition Copyright 2003, American Society for Engineering Education detailed feedback about a day’s lesson. Significance of the VOS Experts in postsecondary education agree it has been hard to define effective teaching across domains,3 and there is much debate on how to evaluate a professor’s teaching effectiveness5. The literature repeatedly supports the value of teacher self-reflection in teacher improvement, 1,2 and teacher centers in universities across the nation engage teachers in analyzing videotapes of their own classroom as a major tool in teaching improvement. The VOS provides a way to measure recognized indicators of effective teaching, and data may be used by professors in formative assessment of their teaching. Data from the VOS generates detailed feedback which faculty may use to self-assess their pedagogical skills. An observed faculty member can review a “lesson profile” of a given lesson (or combination of lessons) to determine teaching patterns. This data can serve as a “quantitative mirror” reflecting the teaching within a class and can allow a faculty member to review the “image” of his or her class and see where change might be appropriate. Significance of Classroom Interactions: the CIO The first component of the VOS, the Classroom Interaction Observation (CIO), provides information about person-to-person interactions within a classroom and reflects various elements of current “How People Learn” (HPL) learning theory. Data from the CIO offers useful feedback in areas such as the percent of questions involving higher order thinking skills and the percent of static teaching (lecture) versus interactive teaching. Other areas include types of in-class feedback to student responses and student response or non-response to questions. The significance of the Classroom Interaction Observation is that it provides quantitative measures of what faculty and students are doing in a classroom and identifies both the frequency and the duration of specific classroom activities. Significance of Student Engagement: the SEO The second component of the VOS, the Student Engagement Observation (SEO), provides information about students’ engagement in both sanctioned and unsanctioned classroom activities. Data from the SEO offers useful feedback on student engagement levels throughout a class period and may relate to the various class activities recorded in the CIO. Also, these data can be used to identify student engagement patterns. The significance of the Student Engagement Observation is that it provides measures of students’ lesson engagement, and it allows an instructor to draw inferences about the students’ engagement as it relates to the type of instruction provided. Significance of Narrative Notes: the NN The third component of the VOS, the Narrative Notes (NN), provides a contextual framework of a lesson. Data from the NN can be used to identify other sources of influence that may affect a lesson. The significance of the Narrative Notes is that they provide the contextual reference that can prevent errors in data interpretation and can add a richness that goes beyond the numbers. Significance of Global Ratings: GR The final component of the VOS, the Global Ratings (GR), provides information about P ge 8.165.3 Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition Copyright 2003, American Society for Engineering Education global aspects of a classroom. The GR uses a Likert scale to reflect the presence or absence of specific indicators related to effective teaching and/or to HPLness of instruction and the degree to which these indicators are present in a class lesson. The Global Ratings provide an overview of an entire lesson; these data are significant in their measurement of specific indicators linked to effective teaching and/or HPLness and can be used to track instructor change in these over time. Limitations The VOS offers great potential to contribute to the body of knowledge in classroom assessment at the postsecondary level. However, several things may interfere with accuracy of the data. First, observers require intensive training; they must develop and maintain high interrater reliability if professors and classrooms are to be compared with themselves and with others for assessment purposes. This requires that observers have a thorough understanding of all operational definitions, that they code at the same rate, and that they continually practice using the system if they are to collect data accurately. Second, observers using VOS must have compatible software (such as Microsoft Access and Excel) and hardware (such as Visors or Palm Pilots) if they are to download both the VOS and collected data. Third, observers must be trained in formatting data and in providing faculty with appropriate formative feedback. Fourth, VOS observation requires a commitment to observe the full class period for multiple sessions of a given class; also, when observing more than one class, it is possible that two or more classes may fall at the time period and thus requiring multiple observers. Conclusion Although prior literature indicates it is hard to define effective teaching, the VaNTH Observation System provides a tool that can be used to measure elements of effective teaching in postsecondary classrooms. Currently, this instrument is being used within various domains of bioengineering. However, we believe VOS application could be expanded to assess teaching in other engineering fields and also in a variety of other disciplines as well.