A data driven approach to assess team performance through team communication.

For teams working in complex task environments, instilling effective communication between team members is a primary goal during task training. Presently, responsibility for evaluating team communication abilities resides with instructors and outside observers who make qualitative assessments that are shared with the team following a training exercise. Constructing technologies to automate these assessments has historically been prohibitive for two reasons. First, the financial cost of instrumenting the environment to collect team communication data at the necessary fidelity has been too high for an operational setting. Second, past research on using team communication as a proxy for team performance assessment has relied on defining communication through traditional algorithmic design, an approach which does not properly capture the varied nature of communication strategies amongst different teams. Recent scientific research in team dynamics provides a theoretical framework leading to a data-driven solution for analyzing the effectiveness of team communication. By framing team communication as an emergent data stream from a complex system, one may employ machine learning or other statisticalanalysis tools to highlight communication patterns and variance, both shown as effective means for assessing team adaptability to novel scenarios. Furthermore, low-cost wearable computers have opened new possibilities for observing people’s interactions in natural settings to better analyze and improve team performance. We summarize research conducted in developing a data-driven approach to analyzing team communications within the context of Surfaced Piloting and Navigation (SPAN) training for submariners. Using Dynamic Bayesian Networks (DBNs), this approach created predictive models of communication patterns that emerge from the team in different contexts. Based upon data collection conducted in the lab and within live submarine crew training, our results demonstrate the robust nature of DBNs by still identifying key communication events even when teams altered their speaking patterns during these events to accommodate for novel changes in the scenario. Introduction Complex tasks that demand a coordinated effort benefit from the capacity of a team to pool resources via an exchange of information and coordinated action, though the effectiveness of a team may be contingent on a variety of factors [1]. Team effectiveness has particular impact within a military setting, as within combat situations the performance of a group has a direct bearing on the survival of the group and those dependent on them [2], situation that holds true when considering the success of naval operations [3]. In an attempt to determine the critical elements that make up an effective team in a military setting, variables related to team effectiveness have been examined from a variety of perspectives, including team cohesiveness (i.e., shared interpersonal closeness and group goal-orientation) [4], [5] collective orientation [1], shared mental models (i.e., synthesis of input from individual team members) [6], [7], [8], team selection and composition (e.g., the skills possessed by the individual team members, how long the members have been working together) [5], [6], [9], quality of decisions made by commanders [10], [11], cognitive readiness and adaptive decision making at the group level [12], training adequacy [5], the workload involved [13], and even neurophysiologic synchrony between team members, as assessed via electroencephalogram [14]. In the context of naval operations, assessment of the quality of teamwork has proven difficult, with such assessments relying on the observations of subject matter experts, skilled instructors, or a self-evaluation within teams during live or simulated exercises [3]. These judgments are subjective by their very nature, leading to a potential lack of consistency with regard to the quality of assessment. This issue has been recognized, and there have been attempts to resolve it, such as through outcome-based Copyright © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. assessments that use goal-attainment as an objective measure of team effectiveness, with goal-attainment defined using Hierarchical Task Analysis for teams [3]. Historically, methods such as this that attempt to create a more quantifiable way of assessing team effectiveness have proven prohibitive such that while they achieve some success in ameliorating the issue of subjectivity they are time consuming and costly enough to make wide implementation infeasible. Teamwork has been defined as the interdependent components of performance required to effectively coordinate the performance of multiple individuals [15], with the authors noting the critical role of communication in team performance. It is precisely this aspect of team effectiveness—communication—that serves as the focus of our work. Previous research in this domain has shown that the ability of a team to adapt to situational demands is reflected by the variance in their communication patterns [13],[16], a finding that our work exploits in an attempt to yield an automated, quantitative measure of team communication, which would allow instructors and teams to assess changes in communication patterns in correlation with scenario events. Sandia National Laboratories has made several investments within the Automated Expert Modeling for Automated Student Evaluation (AEMASE) technology [17] which implements statistical machine-learning techniques for identifying behaviors of interest within spatio-temporal data streams of individuals/teams operating within a live or simulated environment. Instructors using debrief tools with AEMASE integration may generate behavior models through a programming-byexample approach [18] by flagging positive and negative examples of desired behavior. These models can observe the behaviors of other individuals/teams and provide a measure of similarity that serves as an assessment metric. This research conducted a study on utilizing AEMASE for the US Navy Submarine fleet within their Surfaced Piloting and Navigation (SPAN) trainers. Through the use of Dynamic Bayesian Networks as the underlying machine-learning approach, this technology shows promise for identifying vocal communication patterns and providing valuable feedback for instructors and teams. Modeling Team Communication In devising machine-learning algorithms for representing models of effective team communication, one must consider the multiple modalities of data available for analysis. Potential kinds of data that might be used by such a system include: trainee verbal communication, physical actions of the trainees (e.g. movement or control actuation), static factors such as team history or features of the specific training scenario being conducted, along with data available from the training scenario and actions taken in response by the team. As there were time-consuming engineering or social hurdles associated with this research, we ultimately settled on recognizing patterns in trainee verbal communications. To further streamline our approach, we chose not to rely on automated speech recognition technology. Thus, the data stream we chose to analyze indicated who was speaking at any given time during a group training exercise, and the challenge was to use this data to recognize domain-relevant activity patterns. Dynamic Bayesian Networks Bayesian Networks [19] are graphical models that represent conditional dependencies between random variables. For example, the simple Bayesian network in the example below indicates that the availability of downtown parking is conditionally dependent upon both (a) whether or not the time is prior to 8AM and (b) whether or not the day is a weekend. The same network further implies that whether or not it is before 8 AM is independent of whether or not it is a weekend.