Tools For Using Course Embedded Assessment To Validate Program Outcomes And Course Objectives

In the last decade, academia has undergone a paradigm change unprecedented since the space race. Education is now directed toward customer-focused programs. No longer can academia tell students, business, and industry what it will teach; academia must now look to its constituencies, meet their needs, and continuously improve to keep meeting those changing needs. One of the key factors in a continuous improvement program is assessment. Quality assessment tools provide the information needed to measure outcomes and objectives so improvement can take place. Purdue University Calumet has been involved in assessment for continuous improvement for a number of years and has developed course-embedded tools to assess program outcomes and course objectives in various disciplines. This paper describes our efforts to develop and use these tools. To that end, this paper defines terminology, reviews different types of assessment, looks at specific assessment measures, investigates issues in assessment such as small class sizes or classes taught online, examines statistical and anecdotal information to support assessment, and then discusses methods of presenting assessment data. Introduction and Definitions With education now directed toward customeror constituent-focused programs the need for continuous improvement grows more important. In order to be responsive to students, business and industry, academia must assess the needs of those groups, meet them, and continuously improve. One of the key factors in a continuous improvement (CI) program is assessment. There are two basic types of assessment, formative and summative. Formative assessments tell “where we are now”; what do the students know and what do they need to learn to accomplish course objectives. Formative assessments attempt to determine whether the students are learning what they should be learning. [1] Summative assessments tell how well the course went; they are conducted at the end of the course to give information on how to improve for the future. Summative evaluation assesses the completed course, and helps instructors know how well they have achieved goals and learning outcomes established before going into the course. [2] P ge 9.312.1 Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition Copyright © 2004, American Society for Engineering Education Depending on the type of assessment, formative or summative, different assessment methods may be used. These will be covered in a later section. Discussion Summative assessments in traditional classes These are typically surveys done at the end of a course with questions such as major, class, whether the class is required, student questions about student behaviors relating to course (attendance, meeting deadlines, etc.), questions related to objectives, instructor effectiveness, course management, course impact questions, and general suggestions for course improvement. These are common types of assessment questions which are used both to improve the course and for promotion and tenure purposes. Courses taught online The remarks here are limited to summative evaluation information, or traditional course-end assessments. At many universities, the growth of online courses has outpaced the mechanisms to assess and evaluate them. A goal of our revised assessment process was to specifically include online classes. When revising summative assessments for traditional courses to gather information for ABET accreditation review, a group of Manufacturing Engineering Technologies and Supervision (METS) Department faculty devised an assessment tool that can be creatively adapted to fit online or traditional courses by incorporating the University-sanctioned evaluation questions to create an assessment process that would obtain data which would be school or university recognized for online courses. The evaluations addressed several components: course objectives and prerequisites, ABET criteria, the course, the instructor, and student behaviors (course attendance, participation, etc). See Table 1 for sample questions. The appropriate Likert scale has been removed to save space. The assessment itself for online courses consists of approximately 30-40 questions (depending on the number of course objectives). This tool is broken down into four parts: Student Selfassessment, General Course Impact (ABET concerns), Course Management, and Course Objectives. This tool has been designed with the first three sections common to all departmental courses and with the course objectives portion easily modified for different courses. The Course Management Section is modified slightly depending on whether the survey is being used for a distance learning class or a live class. (When used for live classes, a computer laboratory is reserved for 30 minutes during class time to allow the students time to complete the survey.) Even with the large number of questions, students complete the online survey very quickly, usually in 15 minutes. From past experience, this is much faster than paper and pencil assessment tools. Although much modified, it is based on the work of Land and Hager [3]. The course assessment tool is part of a larger project to perform integrated, on-line assessment of all courses in the METS Department, and provides a convenient method to gather summative assessment data. P ge 9.312.2 Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition Copyright © 2004, American Society for Engineering Education The assessments are created in Blackboard, and are given in the same way as most traditional course evaluations, as anonymous surveys. Using the assessment tools in Blackboard, the course evaluation survey is posted and all online students are notified of its availability. A student may submit only one evaluation of the course. The Blackboard system tracks who has completed the survey, but no individual responses are available. All responses are collated or pooled, so that only aggregate responses are available. Table 1. Online Student Course Assessment Tool Sample Questions (Likert scale removed) Specific Student Responsibility Questions: 1. I attended scheduled classes and labs. 2. I arrived on time for scheduled classes and labs. 3. I read the course material/text when it was assigned. 4. I was well prepared for class. 5. I participated in classroom discussions and activities. 6. I used the supplemental materials or website (Bb) my instructor provided. 7. My ability to apply knowledge from pre-requisite courses for this course can be rated as, General Course Impact Questions: 8. As a result of this course, my mastery of the knowledge, techniques, skills, and modern tools of the Mechanical Engineering Technology discipline can be described as, 9. As a result of this course, my ability to apply current knowledge and adapt to emerging applications of mathematics, science, engineering, and technology can be rated as, 10. As a result of this course my ability to conduct, analyze, and interpret experiments and apply results to improve processes can be rated as, 11. As a result of this course, my ability to apply creativity in the design of systems, components, or processes appropriate to program objectives can be rated as, 12. As a result of this course, my ability to function effectively on teams can be rated as, 13. As a result of this course, my ability to identify, analyze, and solve technical problems can be rated as, 14. As a result of this course, my ability to communicate effectively can be rated as, 15. As a result of this course, my recognition of the need for, and an ability to engage in lifelong learning can be rated as, 16. As a result of this course, my ability to understand professional, ethical, and social responsibilities can be rated as, 17. As a result of this course, my respect for diversity and knowledge of contemporary professional, societal, and global issues can be rated as, 18. As a result of this course, my commitment to quality, timeliness and continuous improvement can be rated as,