Open AccessAutomated semantic relevance as an indicator of cognitive decline: Out‐of‐sample validation on a large‐scale longitudinal dataset
Author(s) -
Stegmann Gabriela,
Hahn Shira,
Bhandari Samarth,
Kawabata Kan,
Shefner Jeremy,
Duncan Cayla Jessica,
Liss Julie,
Berisha Visar,
Mueller Kimberly
Publication year - 2022
Publication title -
alzheimer's and dementia: diagnosis, assessment and disease monitoring
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 2.497
H-Index - 37
ISSN - 2352-8729
DOI - 10.1002/dad2.12294
Subject(s) - metric (unit) , relevance (law) , reliability (semiconductor) , computer science , context (archaeology) , cognition , scale (ratio) , measure (data warehouse) , natural language processing , sample (material) , artificial intelligence , correlation , data mining , psychology , mathematics , paleontology , power (physics) , operations management , physics , chemistry , geometry , chromatography , quantum mechanics , neuroscience , political science , law , economics , biology
We developed and evaluated an automatically extracted measure of cognition (semantic relevance) using automated and manual transcripts of audio recordings from healthy and cognitively impaired participants describing the Cookie Theft picture from the Boston Diagnostic Aphasia Examination. We describe the rationale and metric validation. We developed the measure on one dataset and evaluated it on a large database (>2000 samples) by comparing accuracy against a manually calculated metric and evaluating its clinical relevance. The fully automated measure was accurate (r = .84), had moderate to good reliability (intra‐class correlation = .73), correlated with Mini‐Mental State Examination and improved the fit in the context of other automatic language features (r = .65), and longitudinally declined with age and level of cognitive impairment. This study demonstrates the use of a rigorous analytical and clinical framework for validating automatic measures of speech, and applied it to a measure that is accurate and clinically relevant.