Premium
A Cost–Benefit Analysis of Automatic Item Generation
Author(s) -
Kosh Audra E.,
Simpson Mary Ann,
Bickel Lisa,
Kellogg Mark,
SanfordMoore Ellie
Publication year - 2018
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12237
Subject(s) - software deployment , offset (computer science) , computer science , context (archaeology) , cost–benefit analysis , investment (military) , order (exchange) , operations research , operations management , economics , engineering , software engineering , finance , paleontology , ecology , politics , political science , law , biology , programming language
Automatic item generation (AIG)—a means of leveraging technology to create large quantities of items—requires a minimum number of items to offset the sizable upfront investment (i.e., model development and technology deployment) in order to achieve cost savings. In this cost–benefit analysis, we estimated the cost of each step of AIG and manual item writing and applied cost—benefit formulas to calculate the number of items that would have to be produced before the upfront costs of AIG outweigh manual item writing costs in the context of K‐12 mathematics items. Results indicated that AIG is more cost‐effective than manual item writing when developing, at a minimum, 173 to 247 items within one fine‐grained content area (e.g., fourth‐ through seventh‐grade area of figures). The article concludes with a discussion of implications for test developers and the nonmonetary tradeoffs involved in AIG.