English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The use of Automated Machine Learning (AutoML) systems are highly open-endedand exploratory. While rigorously evaluating how end-users interact with AutoMLis crucial, establishing a robust evaluation methodology for such exploratorysystems is challenging. First, AutoML is complex, including multiplesub-components that support a variety of sub-tasks for synthesizing MLpipelines, such as data preparation, problem specification, and modelgeneration, making it difficult to yield insights that tell us which componentswere successful or not. Second, because the usage pattern of AutoML is highlyexploratory, it is not possible to rely solely on widely used task efficiencyand effectiveness metrics as success metrics. To tackle the challenges inevaluation, we propose an evaluation methodology that (1) guides AutoMLbuilders to divide their AutoML system into multiple sub-system components, and(2) helps them reason about each component through visualization of end-users'behavioral patterns and attitudinal data. We conducted a study to understandwhen, how, why, and applying our methodology can help builders to betterunderstand their systems and end-users. We recruited 3 teams of professionalAutoML builders. The teams prepared their own systems and let 41 end-users usethe systems. Using our methodology, we visualized end-users' behavioral andattitudinal data and distributed the results to the teams. We analyzed theresults in two directions: what types of novel insights the AutoML builderslearned from end-users, and (2) how the evaluation methodology helped thebuilders to understand workflows and the effectiveness of their systems. Ourfindings suggest new insights explaining future design opportunities in theAutoML domain as well as how using our methodology helped the builders todetermine insights and let them draw concrete directions for improving theirsystems.

Towards Evaluating Exploratory Model Building Process with AutoML Systems