Premium
Assessing Conceptual Knowledge Using Three Concept Map Scoring Methods
Author(s) -
Watson Mary Katherine,
Pelkey Joshua,
Noyes Caroline R.,
Rodgers Michael O.
Publication year - 2016
Publication title -
journal of engineering education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.896
H-Index - 108
eISSN - 2168-9830
pISSN - 1069-4730
DOI - 10.1002/jee.20111
Subject(s) - categorical variable , concept map , rubric , comparability , categorization , inter rater reliability , competence (human resources) , computer science , ranking (information retrieval) , reliability (semiconductor) , artificial intelligence , machine learning , psychology , mathematics , mathematics education , statistics , social psychology , power (physics) , rating scale , physics , combinatorics , quantum mechanics
Background Conceptual understanding is a prerequisite for engineering competence. Concept maps may be effective tools for assessing conceptual knowledge, yet further work is needed to examine scoring methods. Purpose Our purpose was to evaluate the efficacy of three concept map scoring methods. Traditional scoring requires judges to count concept map components. Holistic scoring requires judges to analytically evaluate concept maps using a rubric. Categorical scoring requires judges to categorize concepts according to relevant categories before quantification of overall complexity. Design/Method In this study, concept maps were collected from 72 undergraduates before and after a capstone design course. Judges analyzed the concept maps using three methods. Using statistical analyses, we examined the comparability, interrater reliability, and convergent/divergent validity of scoring methods. Results While traditional scoring allowed judges to score concept maps relatively quickly, holistic scoring allowed judges to better capture changes in knowledge structure, because ranking of data was required without assuming hierarchical concept arrangement. Only categorical scoring provided insight into content and structure of student knowledge. For all methods, interrater reliability was acceptable, and convergent/divergent validity was established. Conclusions Due to its applicability to concept maps of varying structures, holistic scoring is best if more than one judge is available; traditional scoring is appropriate if there are time constraints or if multiple methods are used. Categorical scoring has potential as a supplemental or stand‐alone method, depending on the needs of the instructor or researcher.