Beyond Likert Scales: Exploring Designers' Perceptions through Visual Reflection Activities

When it comes to the assessment of design behaviors and outcomes, direct observations by external viewers and subjective reflections by the participants themselves can all yield important information. External viewers, for example, may code video evidence or apply design metrics to a designer’s solutions, both of which can lead to interesting statistical analyses and detailed insights. In collecting and analyzing designers’ personal reflections and perceptions, researchers often utilize Likert-type scales, multiple choice questions, or short open-ended prompts. While these modes of data collection are useful and valid, they also constrain participants’ responses to fixed options in the case of Likert-type scales and multiple-choice questions, and to verbal expressions in the case of open-ended prompts. Few examples of other types of reflection activities (e.g., graphing, sketching) have been presented or studied in the engineering education literature. In this project, which is part of a larger investigation into high performance design teams, we explored the use of graphing and other visual techniques for recording designers’ perceptions of their design processes and products. Our primary aim was to introduce greater richness into the evaluation of designers’ behaviors and outcomes as we posed research questions about their relationship to cognitive variables. In this paper, we will discuss two of these reflection activities— an emotional state plot and a graphical assessment of a design solution’s feasibility, usefulness, and novelty—in the context of a team design challenge. While the designers under study worked as teams, each individual designer provided her/his perceptions of the team process and design solutions using these reflection activities. Our analysis of individual perceptions from seven design teams of 4-5 designers each will be reported here. This investigation also includes the cognitive style of each participant (as measured by Kirton’s Adaption-Innovation inventory—KAI®) as a potential mediating variable on the individual perceptions of emotions and outcomes. This paper will provide details of the two reflection activities, our associated methods of analysis, key findings related to the designers’ perceptions and cognitive styles, benefits and limitations of these two reflection activities, and implications for design educators. 1.0 Introduction and Project Context The work presented here is part of an NSF-funded effort in which we are mapping the individual characteristics of design team members and their interactions to their performance in terms of creative design to identify the behavioral building blocks of design teams that produce high performance outcomes (i.e., High Performance Design Teams). We anticipate that the identification of such behavioral building blocks will lead to scientific cognitive-behavioral models of design teams that will be applicable in academic and industry environments, as well as new tools for improving the effectiveness of those teams. In that original context, our aim is to identify and map the behavioral building blocks of High Performance Design Teams (HPDTs) through two functional objectives: 1. Identify the behavioral interaction sequences and individual characteristics that characterize high performance design teams (i.e., the HPDT “genome”); and 2. Map these sequences and characteristics to creative design outcomes. The project utilizes a unique team interaction measurement system called the Interaction Dynamics Notation (IDN)1 to characterize interaction behaviors between individuals on a team, as well as the Kirton Adaption-Innovation inventory (KAI)2,3 to measure individual cognitive styles. These data are then analyzed in conjunction with design outcome measures. Team outcomes are measured in two ways: (1) the concept prototypes delivered by each team are analyzed in terms of standard metrics such as novelty, feasibility, paradigm relatedness, and elaboration; and (2) the reflections of team members on their design performance are recorded via a debrief survey. So far, data have been collected from a total of 31 design teams across academia and industry, and preliminary results from the analyses of team interaction behavior and individual characteristics from a sample of the data have been published4,5. In this paper, we focus on two reflection questions included in the debrief survey—i.e., an emotional state plot (behavior) and a graphical assessment of each design solution (outcome)— that prompt the team members to record their reflections visually. The emotional state plot asks each participant to graph her/his emotional “journey” across the course of a team design challenge, while the design solution graphical assessment asks them to plot the relative feasibility, usefulness, and novelty of each complete design solution on a triangular map. Within this context, our research questions are directed toward the patterns in and the relationships between the emotional state graphs and design solution maps, as well as the influence of cognitive style in each case.