PLAY Minecraft! Assessing Secondary Engineering Education using Game Challenges within a Participatory Learning Environment

This paper describes an initial step towards understanding how computational tools such as natural language processing and machine learning might be used to assess K-12 student learning in engineering education. The study used an online participatory learning environment, PLAY! (Participatory Learning and YOU!), as a platform for student work. Minecraft, an online construction game popular with young teens, was chosen as the learning topic to be assessed. Within PLAY, students created and shared Minecraft ‘challenges’ during a focus session consisting of five boys, ages 9 to 16. Machine learning techniques were used to create a classification scheme for engineering standards based on the Science and Engineering Practices in the Next Generation Science Standards. Natural language processing and data mining techniques were applied to student challenges to assess and report on students’ engineering domain and topic learning. Results show that student application of engineering standards and student discussion of domain topics varied consistently by age. Responses to a corresponding questionnaire showed that the session was a highly positive experience for the children. The potential for use in engineering education is discussed. Participatory Learning Participatory cultures are defined as cultures with relatively low barriers to artistic expression and civic engagement, strong support for creating and sharing one’s creations, and some type of informal mentorship. Participatory cultures foster new media literacies that build on traditional literacy skills taught in the classroom but focus on social skills developed through collaboration and networking, such as play, performance, simulation, appropriation, distributed cognition, and judgment. As schools shift from traditional systems of teaching, student engagement, and professional development, the norms for education are changing. How do individuals best learn and communicate through visual media? How is teaching quality improved by pulling in diverse resources and perspectives that are typically overlooked? How do you build enthusiasm and alignment behind these new models of teaching? The PLAY! (Participatory Learning and You!) platform is a social environment helping students and teachers tap into broad interest-based peer communities – encouraging the four C's of participation in the learning process, creativity, connection, collaboration and circulation. It is a visual platform that enables people to collaborate and problem-solve through creating and sharing media-rich content. The basic unit of collaboration on PLAY is called a canvas; however, the term challenge was used before adopting the more neutral term canvas and is the term we used with participating users in the study (hence the title of the paper). This multimedia canvas encourages users to engage, discuss and share their questions, ideas and knowledge with others. To build a canvas, users simply drag and drop media elements onto the canvas. They can include a wide range of media, from text to photos to videos to RSS streams and more. Canvases can be published publically, or shared privately with a selected group of people. Viewers can take action on a canvas through the Your Turn feature, which enables them to respond to the canvas via a photo or a video. Users can also drive discussions with specific questions that they pose in What Do You Think, the targeted commenting feature. And if a viewer feels inspired, they can remix the learning canvas and build upon the idea to further expand on the creative exchange. PLAY is based on five core principles for participatory learning that have been identified over two years of working with elementary and secondary teachers from the Los Angeles Unified School District who were seeking to develop a more participatory environment in their classroom. • Participants have many chances to exercise creativity through diverse media, tools, and practices; • Participants adopt an ethos of co-learning, respecting each person’s skills and knowledge; • Participants experience heightened motivation and engagement through meaningful play; • Activities feel relevant to the learners’ identities and interests; • An integrated learning system – or learning ecosystem – honors rich connections between home, school, community and world. The affordances of Web 2.0, associated with social software such as blogs, folksonomies, and peer-to-peer media sharing, and the ubiquity of networked computers in K-12 schools have made possible new educational practices like PLAY, that have the potential to produce “radical and transformational shifts” in learning. However, for new educational practices to become accepted they must fit into the curriculum, be aligned to state standards, and have appropriate assessments. This research described in this paper addresses the latter two challenges, NGSS standards alignment and learning assessment. Assessment Pipeline for STEM Learning The goal of the project was to develop a standards-based assessment pipeline for PLAY and to test the feasibility of using it as a STEM learning environment. The work was an interdisciplinary collaboration between researchers at the University of Southern California’s Annenberg School of Communication and Journalism and USC’s Information Sciences Institute. Engineering was chosen as the STEM domain, and the game of Minecraft (2013) was chosen as the learning topic. Work included development of • A data pipeline for analyzing the text and context of a PLAY canvas; • A machine learning classification system for identifying the application of engineering standards and domain topics; • Visual analytics for instructional assessment. The work was conducted over ten weeks during the summer of 2013, and was designed as a collaborative research experience for two undergraduate engineering students, one entering his sophomore year, and one entering his junior year, both of whom are co-authors of this paper. Breadth of research was emphasized over depth of research, as was appropriate given the students’ level of expertise, and which allowed for a wider range of research experiences, including using different computational and analytic tools, and conducting a small focus session to collect data that could be analyzed. Next Generation Science Standards The 2013 Next Generation Science Standards (NGSS) are arranged by disciplinary core ideas (e.g. heredity and energy) for each subject (e.g., life science and physical sciences), for each school level (e.g., middle and high school). NGSS sets performance expectations for each disciplinary core idea based on age level appropriate Science and Engineering Practices, Disciplinary Core Ideas (previous building blocks), and Crosscutting Concepts (Figure 1, left). Figure 1. On left, NGSS Science and Engineering Practices associated with a core disciplinary idea. On right, the top-level page of the Minecraft Wiki, showing its main topics.