Continuous Quality Improvement In Engineering Education: Fact Or Fiction?

The concept of “continuous quality improvement (CQI)” and its application to educational programs has been the standard in higher education for almost two decades as government and accreditation agencies are requiring program accountability through evidence of student learning and a systematic process that enable faculty to determine the need for specific curricular improvements. This has resulted in considerable activity in educational programs; however, in most cases, the degree of activity has not directly been proportionate to the development of true CQI processes that inform program faculty of strengths and weaknesses in student learning. A program can self-assess their own progress toward developing CQI processes by evaluating several factors: nature of involvement of program constituents, clearly stated outcomes; specific knowledge/skill performance needed for students to demonstrate achievement of the outcomes; understanding of how curricular processes support the learning, practice, development, and demonstration of the desired performance; valid and reliable assessment procedures; processes that evaluate the strength and weaknesses in student learning and propose strategies for improvement; and, implementation of improvement processes. This paper examines how CQI processes should be demonstrated in engineering education and explores some of the common mistakes which can lead to considerable effort on the part of faculty with little evidence that the results are useful in understanding the strengths and weakness of student learning. Without valid and sustainable processes, true CQI can only be an elusive quest. CQI and higher education Application of the principles of CQI lead an organization to develop internal processes which enable them to systematically analyze the variance between its stated goals and the results of reviewing the evidence of achievement of those goals. Based on the evidence, decisions are made to understand the areas of needed improvements. The role of both internal and external constituents is well understood and inputs from both are sought according to the roles they play. Members of the organization are encouraged to take risks in order to better understand more about the organization. 1 When CQI is applied to education, student learning is the focus, and program and classroom effectiveness the concern. Assessment activities are the means by which faculty examine the evidence about what is effective and what needs to be improved. The institution and/or program is committed to quality improvement and actively seeks to improve the quality of the intended learning outcomes. The goal is to enhance program and classroom effectiveness in order to improve student learning 2 The National Institute for Learning Outcomes Assessment recently conducted a national survey of 2800 campuses to better understand what campuses were doing related to outcomes assessment and how the data collected in the process were used. There were over 1500 colleges who P ge 15318.2 responded. The results showed that, although most institutions had defined their learning outcomes, the data collected were not being used to make improvements but primarily to satisfy accreditation requirements. Campus and state officials were encouraged to do more to stress the broader social purpose of assessing learning and to do more to focus attention on how the data should be used to make improvements. The article also stressed the finding of broad-based faculty resistance. 3 Another study conducted by the Educause Center for Applied research examined the integration of the principles of continuous quality improvement processes into higher education. The findings indicated that a growing number of institutions are implementing these principles but primarily as a means of verifying effective administration processes. 4 CQI in engineering education Since the implementation of continuous quality improvement principles in engineering accreditation embodied in the ABET General Criteria for engineering education accreditation 5 fifteen years ago, what is the current state of CQI in engineering education? In order to answer this question, one has to look at the underlying principles of CQI and evaluate the degree of implementation each. This evaluation is based on the subjective experience of the author in interacting with hundreds of faculty members through workshops, symposia, institutes and correspondence as well as interactions with the volunteer leadership of ABET as they review programs seeking accreditation. The CQI principles evaluated will be: constituent input, clearly stated educational objectives and student outcomes, alignment of curricular processes which enable student learning, collection of evidence through valid and reliable assessment methodology, and the use of assessment results to drive decision making for improvement. Constituent input and educational objectives Engineering programs have been very active in engaging constituents for the purpose of determining and validating the program educational objectives they want their graduates to attain within a few years after graduation. There are multiple processes that constituents should be involved in related to the CQI process. They should be involved in the development of the educational objectives, the evaluation of the extent to which the educational objectives have been attained, and the periodic evaluation of the currency of the objectives after they have been set. Some programs still have confusion about which of the many constituents that a program may have should be involved in this process. Should parents and students be involved in the determination of program educational objectives? These groups certainly have a vested interest in how well the program prepares its students for success after graduation, but do they have the requisite knowledge or expertise to know what graduates need to attain a few years after graduation? As a group, probably not. However, overall, most programs appear to be actively engaging appropriate constituents in the determination and evaluation of program educational objectives. Clearly stated program outcomes The development of clearly stated program outcomes is the most critical aspect of the CQI process and the one most often overlooked by programs. It is impossible for programs to have a valid or reliable measurement of the achievement of student outcomes unless faculty have a P ge 15318.3 common understanding of the meaning of the outcomes. This means that each program needs to define what the outcomes mean for their students in terms of measureable performance criteria. A performance criteria specifies the focus of instruction and the level at which students should demonstrate the performance. This enables the program to focus the assessment process and also to have a shared understanding of the outcome. Although the term “performance criteria” does not appear in the General Criteria, if a program does not define the outcome (i.e., Criterion 3. Program Outcomes a...k), it cannot measure them in any meaningful way. Faculty should approach the development of measureable performance criteria in the same way that economists have defined what they call “leading indicators.” That is, they have identified certain characteristics of the economy that are significant indicators of the current economic state and predict future trends. The indicators do not include ALL possible economic indicators, but those that are the most critical in predicting how well the economy is doing. By not defining the program outcomes, faculty fall into the trap of collecting any (and usually all) data that can be included under the umbrella of the outcome. The result is lots of data and no information that is useful in identifying specific areas of student strengths and weaknesses related to the program outcomes. Alignment of curricular processes with the anticipated program outcomes Understanding the relationship between the curriculum and co-curricular activities with the intended learning outcomes as defined by the performance criteria is an important step in an efficient CQI process. This is generally done through the development of a curriculum map that identifies where students get an opportunity to learn, practice, develop and get feedback on their performance related to the performance criteria. This should be done with input from each faculty member as some of the outcomes can be demonstrated through pedagogical techniques such as the integration of teaming into the course, writing assignments, oral presentations, etc. This may not be apparent from looking at a course syllabus and it can also vary when there are multiple sections of the same course. When the map is completed, it is possible to not only understand how well and in what ways the curriculum supports learning related to the outcome, but also enables the program to collect data in an efficient and meaningful way. Assessment methodology Assessment is more than just collecting data. It includes the processes of making decisions about where to collect the data, how many data should be collected, development of the techniques of data collection, analysis of data collected and developing the report of results. If the program outcomes are not defined by a few performance criteria, then there is no common understanding of what the outcomes mean. Data collection becomes an exercise without focus or meaning. The meaning one faculty member ascribes to the program outcome may be different than another faculty member although they are both collecting data and reporting results. Data are also collected in multiple courses on the same outcome and then aggregated across the courses and the results reported. This results in data that has multiple data points on the same student even though their performance may involve different content at different levels of learning. In many cases, it also resul