Using Content Analysis to Evaluate Student Inquiry-based Learning: The Case of High School Students Preparing for a Cyber Defense Competition

Inquiry-based learning is a documented, successful method to increase student understanding in science, technology, engineering and mathematics (STEM) areas, as well as a way to facilitate critical thinking. This type of instruction allows students to explore science and resolve problems by reviewing what they know, seeking out additional information about the problem, and coming to a conclusion or resolution based upon evidence they have gathered. This paper evaluates a high school outreach program designed to increase student learning about information technology (IT), in this case specifically information security. The high school students who participated in the outreach program spent the year using the learning materials provided by Iowa State University, asking their own questions about network security and information assurance, exploring additional resources, and determining how to solve the challenges of setting up a secure and viable network. The capstone event for students who participated in the IT club is a two-day cyber defense competition (CDC) on the Iowa State University campus. During the remote setup, the high school students were able to log into a chat room and ask for guidance or clarification from college students supporting the equipment on campus. These chat conversations were logged and this paper utilizes content analysis to quantitatively analyze the chat conversations in terms of the students progressing through Bloom’s taxonomy. The results demonstrated that students were in the Applying, Analyzing and Evaluating stages of learning, showing that the students did perform active and complex thinking in designing, configuring, and securing their cyber defense competition networks. Inquiry-based learning is a documented, successful method to increase student understanding in science, technology, engineering and mathematics (STEM) areas, as well as a way to facilitate critical thinking. This type of instruction allows students to explore science and resolve problems by reviewing what they know, seeking out additional information about the problem, and coming to a conclusion or resolution based upon evidence they have gathered. Inquirybased learning uses an active learning strategy where students move completely through Bloom’s revised taxonomy of cognitive domain. The authors of this paper are involved in a high school outreach program designed to increase student learning about information technology (IT), in this case specifically information security, and wanted to evaluate whether the learning occurring also moved students through Bloom’s revised taxonomy. Since this was an after-school, extracurricular program, traditional measurements used in a classroom such as testing, final reports, or research presentations were not readily available. However, because of the way the program is structured, the authors were able to evaluate the inquiry-based learning taking place using content analysis, a novel, non-intrusive approach. The high school students who participated in the outreach program were part of an information technology (IT) club formed in the fall of the academic year at their high school to study network and information security using an inquiry-based approach. Students spent the year using the learning materials provided by Iowa State University, asking their own questions about network P ge 25423.2 security and information assurance, exploring additional resources, and determining how to solve the challenges of setting up a secure and viable network. The capstone event for students who participated in the IT club is a two-day cyber defense competition (CDC) on the Iowa State University campus. In a CDC the students remotely design and configure a set of servers and a network in a secure manner one month prior to the competition and then come to campus for a two-day event where they protect and maintain the equipment from hackers. During the remote setup, the high school students were able to log into a chat room and ask for guidance or clarification from college students supporting the equipment on campus. These chat conversations were logged and this paper utilizes content analysis to quantitatively analyze the chat conversations. The conversations are analyzed for overarching themes and questions to examine the process the students are going through during the inquiry-based learning of setting up their competition networks. The paper is organized into five sections: Motivation, Project Background, Methodology, Results and Conclusion/Future Directions. Motivation for Using Content Analysis to Understand Inquiry-Based Learning Inquiry-based learning is a multifaceted approach that involves reviewing information about what is known about a problem, gathering additional information, proposing solutions or explanations, and communicating or acting on the results. The focus of all activities is on critical and logical thinking, as well as exploration of alternative solutions. It is an effective way to shape students’ ability to apply their knowledge, search for new information, and critically synthesize material. When using inquiry-based learning in the classroom, an instructor facilitates their learning, but does not formally structure their experiences. The students are given a problem, given background information, and then asked to find their solution which will include finding new information on their own. Inquiry-based learning is the backbone of the educational outreach programs offered by the Information Assurance Center (IAC) at Iowa State University. In the IAC’s implementation of inquiry-based learning in information assurance and computer/network security education, the students are given access to learning materials, as well as a set of questions to explore. They are then asked to provide solutions to challenges presented to them based upon their own exploration, additional resources they find, and experimentation. The materials provided focus on security-based concepts for designing, implementing, configuring, securing, and protecting a network, its servers, and end users from attacks from the outside world. Although the IAC conducts inquiry-based information assurance and computer/network security programs targeted at college-age students from Iowa State University (see paper by the authors at this conference), community colleges across the state and four year institutions nationwide, this paper focuses on an inquiry-based program developed for and delivered to high school students within the state. The use of content analysis to understand inquiry-based learning is a novel approach to evaluation that the IAC is undertaking as part of its outreach program. Content analysis is the scientific, systematic, quantitative analysis of messages. It is widely used in analyzing messages in journalism, mass communication, sociology, psychology, and business. Historically, in the mass communication discipline, it has been used to examine word usage, as well as the context of conversations and human interactions in mediated messages transmitted by newspapers, television, and radio. With the advent of electronic mass media, content analysis P ge 25423.3 has expanded to include the evaluation of web sites and blogs. Additionally, work in content analysis has evolved to include non-mediated messages, such as conversations between individuals. It is these non-mediated messages found in the chat conversations between high school students and the college students supporting their remote setup for a CDC that this paper evaluates. The authors use content analysis to find themes in the students’ inquiry-based learning as they design and configure their competition network. Using these coded themes, the authors examine how the student conversations and questions change over time from the beginning of setup to the end of the configuration period. These coded themes are them examined in the context of Bloom’s taxonomy to see if the students are moving through the cognitive learning process with this program. Bloom proposed a taxonomy of educational learning objectives which was based on a set of conferences in the early 1950’s. The taxonomy is composed of a classification of various objectives which can be utilized by educators when developing content for their students. The taxonomy is a tiered model with each level representing a particular degree of cognitive complexity. The levels of the taxonomy build on each other such that functioning by an individual at a particular level within the hierarchy is dependent on mastery of the material at the next lower level. Anderson and Krathwohl modified Bloom’s taxonomy based on input from three stakeholder groups: cognitive psychologists, curriculum theorists, and assessment specialists. This revised model utilizes verbs as opposed to nouns to emphasize the engagement of the individual in the learning process. Furthermore, several name changes were used with regard to categories. Figure 1 shows the revised taxonomy with both the title words as well as supporting words detailed within each level. Also, notice the increasing mastery of the individual with the educational material as you move from bottom to top in the hierarchy.