Premium
Expanded understanding of student conceptions of engineers: Validation of the modified draw‐an‐engineer test (mDAET) scoring rubric
Author(s) -
Thomas Julie,
Hawley Leslie R.,
DeVoreWedding Beverly
Publication year - 2020
Publication title -
school science and mathematics
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.135
H-Index - 2
eISSN - 1949-8594
pISSN - 0036-6803
DOI - 10.1111/ssm.12434
Subject(s) - rubric , inter rater reliability , reliability (semiconductor) , mathematics education , test (biology) , psychology , kappa , computer science , mathematics , statistics , rating scale , geometry , paleontology , power (physics) , physics , quantum mechanics , biology
With this research, we sought to expand the pool of engineering education assessment instruments. We drew upon earlier efforts to define an expanded protocol for prompting children's drawings of engineers; and evaluate the reliability of a new scoring rubric for the modified Draw‐an‐Engineer Test (mDAET). An earlier paper introduced our ground theory approach in gathering theoretical propositions derived from children's drawings, explanations of engineers ( n = 940) to develop the mDAET Scoring Rubric. Here we evaluate the reliability of the mDAET, a component of the larger mDAET rubric validation process. In the current study, eight participants scored twelve sets (including anchor sets) of drawings ( n = 36) and, scores across drawings were evaluated by percentage match with the known‐score and interrater reliability. Kappa values for exact matches across eight raters met the threshold for “substantial” agreement and matches that were exact or adjacent indicated “almost perfect” agreement. Notably, 92% of the possible matches among common anchor items were either exact or adjacent matches. These results indicate adequate inter‐rater reliability evidence of the mDAET scoring rubric and demonstrate the feasibility of largescale administration that may no longer require individual child interviews.