Investigation of Probabilistic Multiple-Choice in a Structural Design Course

Student assessment is a critical part of the learning process. Engineering courses often use objective student assessments to promote impartiality and grading efficiency, but many objective assessment methods do not provide educators sufficient information on their students’ level of knowledge. The probabilistic multiple-choice question is a type of objective assessment that uses a ‘reproducing scoring system’ which encourages students to ‘reproduce’ their true knowledge state. Presently, the authors are unaware of any attempts to use probabilistic multiplechoice assessments in undergraduate civil engineering courses. This paper examines the impact of using probabilistic multiple-choice questions on students in CE404: Design of Steel and Wood Structures. The paper explores whether or not probabilistic multiple-choice assessments positively impact students’ level of critical thought, their understanding of technical language used in the course, their ability to self-assess their level of understanding and confidence, and the propensity for self-learning. Student performance on several assignments is compared to the previous year when probabilistic multiple-choice was not used While there is no clear indication that probabilistic multiple choice had a significant positive impact in most of the areas, students using probabilistic multiple choice did score 4.8% higher overall on major graded events. INTRODUCTION Assessments inform students on their scope and depth of knowledge and serve as an indicator of teaching effectiveness for educators. Assessments can also help students think about course material in a different way . However, assessments require time on the part of students and the educator. This paper examines the use of probabilistic multiple choice assessments in CE404: Design of Steel and Wood Structures, a required course for civil engineering majors at the United States Military Academy (USMA). CE404 builds on students’ understanding of statics, mechanics, and structural analysis to design tension, compression, bending, and beam-column structural members. Students complete seven homework assignments, two mid-term exams, one Engineering Design Problem (EDP), and a final exam during the course. Until 2015, the student’s homework assignments consisted primarily of free response design or analysis problems using a problem solving format. Time is the most demanded resource for cadets at the USMA. A USMA Senior takes an average of 20 academic credit hours per semester not counting an additional military science course, a physical fitness course, mandatory participation in a competitive sport, and leadership responsibilities in the Corps of Cadets. In addition to the mandatory demands on a cadet’s time, the typical cadet is also involved with club events, social media, or other extracurricular activities. The cadets’ time demands create challenges in achieving the typical 1:2 ratio of contact hours to out-of-class hours for an academic course. From 2005 to 2014, out-of-class time survey data indicates that cadets spend on average 60 minutes out of class for every hour in class, half the expected time. The time cadets do spend out of class on homework often involves repeating steps from in-class examples with different values to practice the “problem solving format.” These homework problems are valuable to reinforce equations taught in class and to exercise the cadets’ ability to communicate in writing, but these types of homework problems rarely go beyond the cognitive domain of applying . The nature of structural member analysis and design is often repetitive and students are often able to follow a set of prescribed steps to arrive at the correct answer with minimal understanding of the structural analysis or design process. The concern is that while the student can follow the steps, more like a technician than an engineer, they are not internalizing the depth of knowledge required for understanding and solve problems of greater complexity that they may see in their careers. The regurgitation of in-class problem steps is due in part to the limited time cadets allocate for their out-of-class assignments. Creating homework assignments that challenge students to think critically and do not exceed the expected out-of-class time is difficult. Additionally, free response questions that do challenge the students’ comprehension is burdensome to grade when providing meaningful feedback. USMA does not employ Teaching Assistants, and instructors are required to manage course administration, write the course assessments, teach the material, provide office hours, and grade all course assignments. The time requirement is compounded when providing feedback to students with a poor understanding of the material. ASSESSMENTS Assessments serve a critical function in the education process . A high-quality assessment confirms what the student knows, identifies the areas a student is weak, pulls together what the student has learned in class, exercises higher-order cognitive abilities, and provides a comparative assessment among students. A high-quality assessment also provides the instructor insight on their students’ knowledge states . Science, Technology, Engineering, and Mathematics (STEM) courses gravitate towards objective assessments to test student knowledge. Objective tests fit into two broad categories: selection-type (multiple-choice, matching, etc.) and supply-type (short answer, problem set, etc.) . Selection-type testing, specifically multiple-choice (MC) assessments, offer a variety of different methods for assessing student learning. These methods include, but are not limited to, the conventional MC test, elimination testing, confidence marking, probability testing, and twostem questions. Many of the methods are structured to both discourage student guessing and provide opportunities to capture partial knowledge . MC testing facilitates objective, highspeed grading with the functionality to assess higher level thinking . Research has also found that modifying conventional MC questions can require students to more critically examine the material being tested . The main limitations to MC assessments are the test writer’s creativity, the tools available for grading, and the minimal exercise of student communication skills. PROBABILISTIC MULTIPLE CHOICE Probabilistic MC assessments use a ‘reproducing scoring system’ that provides a valuable service to students and educators. Reproducing scoring systems, also known as a “proper scoring systems,” “admissible scoring systems,” or “scoring systems which encourage honesty,” create a system of rewards that incentivizes students to report their true belief or knowledge state (8) . These systems provide the educator with greater insight into their students’ true knowledge state and encourage students to spend more time assessing how well they understand the material . The conventional single-answer MC question can mask a student’s knowledge state from both themselves and the educator . These assessments provide binary or dichotomous feedback: students get the right answer and full credit or the wrong answer and no credit. Students with low knowledge states who randomly guess on a four-answer-choice question have an expected score equal to 25% of the total points. If the students can eliminate one or two answer choice(s), their expected score moves towards 50% of the total points. The downside for guessing is earning zero point while the upside is getting full credit. The mutually exclusive and collectively exhaustive nature of conventional MC questions provide students little incentive to critically exam all answer choices once their ‘correct’ answer is identified. A modification to this standard MC assessment is removing the mutual exclusivity of the problem by allowing more than one correct answer choice. A four-answer-choice problem with multiple correct answers decreases a student’s chances of randomly selecting the correct answer from a probability of 0.25 to 0.067. The student has a much greater incentive for applying critical thought to each answer choice; however, while the multiple answer modification can capture partial knowledge and increase critical thinking, the assessment technique does not guarantee quality feedback on the student’s knowledge because the educator does not know the student’s degree of uncertainty for their selected answers . Ease of computer access and advancements in computer applications are significantly improving educators’ ability to assess student learning through MC assessments. Computer labs and personal computers provide students a venue for submitting out-of-class assignments, and course websites and assessment software provide instructors an opportunity to expeditiously collect and grade student work. These computer-based assessments can incorporate reproducing scoring rules that award credit for partial knowledge on MC questions (3) . This type of MC assessment requires students to assign a probability to each answer choice being correct. There are various graphical, qualitative, or numeric methods in existence for students to report their probabilities (3) (12) (13) . Reproducing scoring systems can vary in both how they award points and the range of points they award. A spherical scoring rule awards points locally or based only on the probability assigned to the correct answer; quadratic and logarithmic scoring rules can award points globally or based on the probability vector assigned across all answer choices. These different scoring rules also offer different bounds on the number of points a student can lose for assigning a low probability to the correct answer (9) . For example, a spherical scoring rule will not award a student less than 0 points where as a logarithmic scoring rule can award a student up to -∞ points for assigning zero probability to the correct answer. Doctors, meteorologist, intelligence officers, and many