Evaluation Of A Living Learning Community For Engineering Freshmen

The idea of learning communities is not new or novel, however, its role in retaining, engaging, and intellectual development for engineering students has yet to be fully explored. There are numerous learning community studies that quantitatively measure grades and retention, and more recently studies that include engagement as measured through individual and national survey instruments. However, the vast majority of these studies are directed at general freshmen populations and not at engineering students specifically. Additionally, drawing direct causality from the learning community to the outcomes is still problematic. Controlling all the other variables that can affect grades, retention, and engagement from an experimental standpoint in an academic setting is difficult at best; consequently, a more effective methodology for evaluating a learning community program may be examining several pieces of evidence that “point” in a particular direction. This evaluative study considers a body of evidence collectively similar to vector math – by adding the magnitude and direction of each piece of evidence to determine a relative measure of success for the program with respect to the program goals. Background To understand this evaluation process, it is necessary to first understand the type of learning community that was developed and why this method may be particularly useful for engineering students. Second, a review of evaluation methodology for other learning communities sheds light on different ways of conducting a program evaluation. Finally a discussion of the measures and expected outcomes for this evaluation is provided. Type of learning community and justification Learning communities can take many forms. Most concisely Shapiro and Levine 1 identify four major types of learning communities: 1) paired or clustered courses; 2) cohorts in large courses or first-year interest groups; 3) team-taught courses; and 4) residential learning communities. Most learning communities fall within these categories or are combinations of these primary types. The learning community for this evaluation is a combination of three of these general types: clustered courses, first-year interest group, and residential. This learning community model was designed to mitigate high attrition rates and inadequate student preparedness and increase engagement in college activities. With only one half of a percent of the average postsecondary student body enrolling in engineering, 2 and only half of those students remaining in engineering, 3 many professional associations and governmental agencies are concerned about the state of engineering education. Factors causing students to switch from engineering 4, 5 include: institutional factors (i.e., the college “chilly” climate versus a more nurturing high school experience and lack of personal contact with faculty), differing high school and college faculty expectations as well as student P ge 10595.2 Proceedings of the 2005 American Society for Engineering Education Annual Conference & Exposition Copyright © 2005, American Society for Engineering Education expectations, and epistemological assumptions (relating to the belief in the certainty of knowledge). For engineering programs this is particularly disconcerting since many will loose up to half of their students in the first year 5, 6, 4 – including those students who already have the required abilities and high school preparedness. 4 Learning communities hold promise for engineering departments as they have been shown to increase retention, improve student attitudes and engagement and increase academic achievement. 7, 8, 9, 10 Adding an academic component to a residential structure has been suggested by several studies as a way to improve the college experience and increase retention and academic understanding. 11, 12, 7 Blimling & Hample 6 found increases in academic achievement from 0.05 to 0.2 points per quarter when restructuring residential environments around common academic themes. Other research suggests students in a residential program without an academic component are not as likely to show any differences in academic achievement or retention as their non-participating peers. 13, 14 Furthermore, research on learning communities suggests that co-curricular classes can help academic achievement, but do not necessarily show any gains in students’ attitudes and engagement when compared to their peers. 15 Based on research indicating engineering students turn away from engineering because of climate, expectations, and abilities, a learning community for engineering students that results in academic gains, retention, and engagement would be most beneficial when all three parts are combined: the residential component coupled with the common classes and the facilitation of collaborative learning and social support through the small group seminars. The learning community subject to this evaluation has all three of these parts. Entering engineering students self selected into the program and were pre-registered for up to four common classes and one of six non-credit bearing seminar groups. Students were housed on two mixed-gender floors in the same residence hall. The seminars met regularly in assigned classrooms and were facilitated by upperclassmen peers. The peer facilitators were trained prior to the beginning of the semester in mentoring, successful study strategies, and student learning and development theory and application. Evaluation methodology literature review Methods for evaluating learning communities have been proposed by Moore, 16 Tinto, Love, & Russo, 17 Wilkie, 18 and The Living-Learning Program Report. 19 Moore used Perry’s 20 theory of intellectual development as a basis for measuring the effects of learning communities. A survey instrument, the Measure of Intellectual Development (MID) an essay-writing test derived from Perry’s work was used to determine impacts from the learning community. The MID was given to learning community participants and also to peers who were then scored on a 1.0 to 5.0 system relating to where they stand in Perry’s intellectual development scheme. Intellectual development was then compared between the two groups. Results from this study found that learning community participants showed further developmental gains than their non-participating counterparts. Love, Tinto, & Russo 17 approached evaluation by first assuming learning communities were effective ways to respond to the academic and social needs of students. Further, they sought to “casting our nets widely in an effort to be open to unexpected phenomena.” The researchers suggested that by doing this, subjective value judgments were eliminated and instead an P ge 10595.3 Proceedings of the 2005 American Society for Engineering Education Annual Conference & Exposition Copyright © 2005, American Society for Engineering Education understanding developed about how each program met the needs of students at each institution and how it shaped student learning and persistence. Wilke 18 proposed a more “institutional” method by responding to a series of questions divided into three categories: student performance, student retention, and student development. Measures were mixed using both quantitative (grades, retention, course completion, credits completed) and qualitative (students’ responses to learning communities, students perceptions of themselves as learners, and difficulties encountered by students in learning communities) methods. Wilke asserts the inclusion of quantitative data despite arguments against the appropriateness of such measures, because there is value in building a case directed toward administrators for learning communities. The National Living-Learning Communities Report 19 undertook a multi-institutional study to compare types of living-learning communities (the type of learning communities that would fall under the “residential learning communities” based on the Shapiro and Levine categories listed previously) with each other and between institutions. This study is unique as it developed a typology and a standard method of inquiry. Using Astin’s 21 Inputs-Environment-Outcomes (IE-O) theoretical framework, the study provides useful data for benchmarking residential learning communities. The I-E-O theory is one where “outcomes (student characteristics after exposure to college) are thought to be influenced by both inputs (pre-college characteristics) and environments (the various programs, policies, relationships with faculty and peers, and other educational experiences that impact students).” 19 A survey instrument was developed to identify inputs, the environment, and outcomes, and was administered to over 23,000 respondents in 34 colleges and universities. Researchers sought to reduce bias and internal validity threats by identifying and accounting for differences in “inputs.” Doing this, researchers assert this study provides an assessment methodology for multi-institutional and like-program comparison. Measures and outcomes for program evaluation Elements of these evaluation schemas are intertwined in this program evaluation using both quantitative and qualitative data. We would expect students who participated in this learning community to do better in their common classes (as evidenced by higher grades in the common classes) because of additional time-on-task due to the regular seminar meeting, ready-made study group partners, and close proximity to other students taking the same classes in the residence hall. We would also expect increases in learning due to developing and practicing college level study skills as many students are not adequately prepared for the rigor of college study. 22 Another factor that could affect students’ grades are the amount of student preparedness. To reduce this internal threat to validity, we measured AIN and made comparisons between the learning community students and their non-pa