English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Machine Learning models have many potentially beneficial applications ineducation settings, but a key barrier to their development is securing enoughdata to train these models. Labelling educational data has traditionally reliedon highly skilled raters using complex, multi-class rubrics, making the processexpensive and difficult to scale. An alternative, more scalable approach couldbe to use non-expert crowdworkers to evaluate student work, however,maintaining sufficiently high levels of accuracy and inter-rater reliabilitywhen using non-expert workers is challenging. This paper reports on twoexperiments investigating using non-expert crowdworkers and comparativejudgement to evaluate complex student data. Crowdworkers were hired to evaluatestudent responses to open-ended reading comprehension questions. Crowdworkerswere randomly assigned to one of two conditions: the control, where they wereasked to decide whether answers were correct or incorrect (i.e., a categoricaljudgement), or the treatment, where they were shown the same question andanswers, but were instead asked to decide which of two candidate answers wasmore correct (i.e., a comparative/preference-based judgement). We found thatusing comparative judgement substantially improved inter-rater reliability onboth tasks. These results are in-line with well-established literature on thebenefits of comparative judgement in the field of educational assessment, aswell as with recent trends in artificial intelligence research, wherecomparative judgement is becoming the preferred method for providing humanfeedback on model outputs when working with non-expert crowdworkers. However,to our knowledge, these results are novel and important in demonstrating thebeneficial effects of using the combination of comparative judgement andcrowdworkers to evaluate educational data.

Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement