Sustainable Assessment For Program Improvement And Abet Preparation

One of the fundamental challenges of program assessment is to develop a process that is sustainable and has the rigor to deliver the results required for continuous improvement. At the same time the process should on a steady basis be able to provide the data that is expected to be an integral component in the preparation of the ABET Self-Study when the time comes for requesting accreditation. In this paper we describe such a process. The process consists of three components: 1. A fast feedback procedure to implement continuous improvement at the course level. This procedure includes a course improvement form completed by the course instructor that documents their positive and negative reflections, suggested actions for course improvement, and deviations from the institutional syllabus in their offering of the course. A mechanism for evaluating and implementing suggested improvements is detailed. 2. A program outcome assessment procedure that directly assesses specific program outcomes and implements curricular changes resulting from evaluation of the data. This procedure incorporates an outcome assessment form with assessment tools, rubrics, benchmarks, and suggested curricular actions. The application of the form to the outcome assessment procedure for creating, implementing, and evaluating curricular change is explained. 3. Ongoing documentation of the process is accomplished via an institutionally-mandated annual program audit. The audit is based on essential elements that are consistent with, and mirror the format of the ABET Self-Study. Details and reflections on this audit by two program directors are included. Program outcomes are assessed directly and not inferred from course outcomes. Specific assignments are required in courses designated to provide the evidence in the form of student work that demonstrates the extent to which the program outcomes are met. The result of this process is to consistently gather data that is critical for the ABET Self-Study and provide ongoing curriculum continuous improvement at both the course and program levels. Key observations are that decision-making has become more formalized and subject to more rigorous, documented scrutiny. Faculty buy-in to the process has been very favorable because of the ease and clarity of the tasks to be done. Introduction and Purpose Undergraduate engineering programs are discovering the benefits and costs of continuous improvement. Since the 2006-07 accreditation cycle ABET has required programs to implement a Continuous Improvement Process (CIP). The plan must demonstrate evidence of actions which improve the program based on the assessment and evaluation of a program’s objectives and outcomes. In the 2009-10 accreditation cycle the ABET Criteria for Accrediting P ge 15152.2 Engineering Programs established Criterion 4 Continuous Improvement as a separate criterion. Long before ABET formally required continuous improvement, effective engineering faculty and programs utilized many ad hoc procedures to improve their performance. For example, instructors constantly assess student learning through comparison to standards that represent competency in course material. Self-reflection and peer interaction often occur during and after teaching a course. These ad hoc procedures and interactions create tribal knowledge that form an institutional history. Such a history can enrich and improve the program on a continuous basis. However, in such an environment program administrators are challenged to collate and leverage the collective knowledge of the program’s faculty in their efforts to steer the overall curriculum towards greater effectiveness. Further, Whiteman contends that “If the learning community, to include students, faculty, and constituents, are able to continuously improve the teaching and learning experience for the students, the assessment process should be considered a success.” While ad hoc methods are beneficial, a formal institutional-based continous improvement process is required to create a sustainable process while simultaneously minimizing costs of implementation. Numerous Continuous Improvement Processes (CIP) models exist and are well studied. Such models include plan-do-check-act, Six Sigma, and Total Quality Management (TQM). One key factor that determines the success and sustainability of each of these CIP initiatives is the active participation in the CIP by all participants involved. A second key factor is the communication of the findings and the results to all the participants. A third factor is simplicity since it is unrealistic to expect a high level of training for all participants. The ability to continuously improve is conditioned on the sustainability of the process. The pre-ABET continuous improvement procedures used by faculty, albeit somewhat effective, had been ad hoc, disparate, and largely undocumented. There is an administrative overhead cost that is incurred to standardize and document continuous improvement processes. The benefits to be had include compliance with ABET requirements, systematic change, and documented improvement rising above the course level to the curricular level. The process described in the remainder of this paper has been adopted by multiple engineering and technology programs at our university. It was designed with two overriding specifications: sufficiency and sustainability. The process contains many Kaizen principles such as: • Improvements are made using small changes that do not require large investments • Ideas for improvements come from those closest to the process and most capable for the success • The active parties take ownership of the improvement process In addition, the overall process contains essential elements that are consistent with and mirror the format of the ABET Self-Study.