Measuring the success of an educational program through box-and-arrow diagram: A case study of the Quantitative Microbial Risk Assessment Interdisciplinary Instructional Institute

Box and arrow diagrams are simple models that help students synthesize information pertaining to complex environmental systems and allow instructors to identify and address the misconceptions. These diagrams can also be used to evaluate the effectiveness of training, through collection of diagrams before and after instruction. Box-and-arrow diagrams were used to measure the success of the Quantitative Microbial Risk Assessment Interdisciplinary Instructional Institute (QMRA III) at Michigan State University. QMRA III is a one-and-a-halfweek training program designed for advanced graduate students, post-doctoral fellows and early career professionals to assimilate scientific data and implement computer programs towards building a risk assessment for assuring safety and health goals. Each cohort of QMRA III consists of engineering, biological and social scientists with the goal of cross training. As part of the evaluation plan of the program, students were asked to construct a box-and-arrow diagram conveying a risk management plan involving the full range of biologic, economic, social, political, and cultural factors that impact risk during a pathogen exposure. Additionally, experts, professors and career professionals who were also the instructors in QMRA III, were asked to construct similar diagrams. This paper presents the results of the evaluation of the diagrams, and demonstrates how a simple tool, such as box-and-arrow diagram can be used to evaluate an interdisciplinary educational program with emphasis on the expansion of the participant’s knowledge beyond their own discipline. Using this method, effectiveness of the program was assessed by comparing the participants’ understanding before and after the workshop, and also by comparing participants’ diagrams to those conveyed by the experts. Introduction and Background Quantitative microbial risk assessment (QMRA) is the process of applying risk assessment principles to quantify the risk associated with planned or actual exposure to infectious microbes (Haas et al., 1999). QMRA uses four stages to identify the risk: hazard identification, doseresponse, exposure assessment, and risk characterization (Medema and Ashbolt, 2006). Hazard identification describes microorganism and associated diseases, whereas dose-response quantifies the relationship between dose and health outcomes. Exposure assessment looks into the pathway that lead to exposure and identifies the affected population. Finally, risk characterization combines the information from previous three stages to characterize the probability of an adverse health outcome with a description of the variability and uncertainty associated with these estimates. In 2009, the National Research Council published an update to the established paradigm in Science and Decisions. Figure 1 illustrates the risk assessment process and feedback between scoping and management from this document. Figure 1. Risk Assessment Framework (NRC, 2009) Microbial risk assessment is a scientific process that estimates the likelihood of microbial exposure and resulting public and environmental health impact (USDA and USEPA, 2012). Risk assessment framework is inextricably linked with other components of risk analysis which include risk management, risk communication, and other social as well as economic aspects. Figure 2 shows how these components are linked together. Figure 2. Risk assessment framework and its relationship with other components of risk analysis (USDA and USEPA, 2012) A conceptual model describes or visualizes the relationships among various components so that readers get a clearer picture of risk assessment (USDA and USEPA, 2012). Figure 3 shows an example of a box-and-arrow conceptual model used to assess the risk associated with in-plant component of Listeria. The figure demonstrates that development of such a model is an iterative process and requires fitting each component into a large risk assessment framework. Hence, to construct such a conceptual model, one not only must understand the components of the risk assessment framework, but also know how they are linked with each other. QMRA approach is to recognize these interlinkages and follow the iterative process to come up with a management plan that brings the public/environmental health status to the target level. Figure 3. Conceptual model used to assess the risk associated with “in-plant” component of Listeria (USDA, 2003) The Quantitative Microbial Risk Assessment Interdisciplinary Instructional Institute (QMRA III) is a 10-day workshop designed for advanced graduate students, post-doctoral fellows and early career professionals to assimilate scientific data and implement computer programs towards building a risk assessment for assuring safety and health goals. Participants gain hands-on experience with risk assessment software and work on real-world case studies. QMRAIII 2015 was conducted from 1 to 12 August 2015 at Michigan State University. PhD students, Master students, postdocs, early career faculty member and professionals participated in the program. A total of 32 participants, 15 females and 17 males, attended the workshop. Participants came from a diverse background, which included environmental science, environmental engineering, food science, microbial science, biosystems engineering and public health. Students submitted an application including a background survey and statement of their future research interests. A box and arrow diagram is a method used to evaluate the student’s knowledge of the subject matter pertaining to scientific understanding, which has been included in the Cause-Map method developed by Clark et al. (2009). Box-and-arrow diagrams are simple models that help students synthesize information pertaining to complex environmental systems and allow instructors to identify and rectify the misconceptions (Canham et al., 2003; Clark et al., 2009). Also, known as Cyclic Concept Maps, box-and-arrow diagrams are an appropriate tool in representing relationship between concepts (Safayeni et al., 2005). Box-and-arrow diagrams have been used in textbooks to explain the geological systems such as water and rock cycles at least since 1930s (Croneis and Krumbein, 1936). They are conceptual models, where boxes represent stocks and arrow represent the relationship between the stocks including flow and feedback. These diagrams can also be used to qualitatively measure how the understanding of an individual on certain subject matter changes over time. By qualitatively measuring the understanding of the participants on a subject matter before and after some educational program, these diagrams can be used to evaluate the outcome and effectiveness of an educational program. This study seeks to understand the impact of the QMRAIII training model on participant understanding of microbial risk assessment and management using the box-and arrow method. More conventional survey instruments were also used to evaluate the effectiveness of the program as well, but this study focuses exclusively on the box-and-arrow diagram, which we believe provides a more holistic assessment of the participants’ ability to conceptual the application of the complex risk assessment paradigm. In particular, we investigate the changing nature of mental models about risk management plans in the context of biologic, economic, social, political, and cultural factors.