HarP: The EADC‐ADNI Harmonized Protocol for manual hippocampal segmentation. A standard of reference from a global working group

This special issue of Alzheimer’s & Dementia marks the completion of an initiative kick-started as long as 6 years ago. At the time, hippocampal atrophy was regarded by International Working Group criteria as one of the biomarkers for the early diagnosis of Alzheimer’s disease (AD) [1], academic memory clinics were pioneering its use in real life diagnostic studies [2–4], and clinical trials of disease modifiers were starting to use hippocampal atrophy rates as a secondary outcome measure [5]. Widely different measurement protocols prevented the comparison of diagnostic accuracy and biologic drug efficacy. Standard operating procedures were clearly needed—an effort greatly facilitated by the availability of ADNI (Alzheimer’s Disease Neuroimaging Initiative) harmonized image acquisition parameters and procedures [6]. The initiative was kicked off at a feasibility workshop organized by the Alzheimer’s Association in Chicago in 2008 where ADNI and European Alzheimer’s Disease Consortium key members took part, and experts in imaging biomarkers. A survey was set out to identify the 12 most frequently used protocols for hippocampal segmentation in the Alzheimer’s literature and differences in image treatment procedures and anatomical landmarks (Fig. 1), a mandatory step to develop a harmonized protocol [7]. In an exercise reminiscent of a LEGO block game, the preliminary phase did a virtual break down of the hippocampi resulting from the aforementioned protocols in a finite number of three-dimensional (3D)-units summarizing their entire anatomical variability. Biometric features of the 3D-units were then empirically quantitated (e.g. measurement stability, contribution to Alzheimer’s-associated atrophy) and fed to a panel of world experts, including the developers of the 12 originally selected protocols, whowere charged of coming out with a unique and harmonized segmentation protocol. Thanks to a Delphi procedure adapted to accommodate quantitative information, it took experts five rounds to converge onto a definitive version (the harmonized protocol—HarP). Five expert researchers on hippocampal segmentation (“master tracers”) were then asked to segment 40 representative hippocampi taken from the ADNI database following the HarP, thatwould beused as the standard of truth of any ensuing procedure (so-called “benchmark labels”). Fourteen tracers coming from 12 imaging laboratories from eight countries in three continents which had not been exposed to the development of the HarP were asked to segment another set of 40 hippocampi representative of the ADNI dataset, trained and qualified to segment following the HarP on an ad hoc webbased environment, and asked to re-trace the same ADNI hippocampi following the HarP. An appropriately balanced design allowed to test the concurrent validity of the HarP versus local protocols, and compare the error variance of hippocampal volume estimates due to theHarPwith other sources of error. Finally, HarP hippocampal volumes were validated versus pathological findings in a sample of brains where both high-resolution structural magnetic resonance imaging and a post-mortem examination were available, and publicly available HarP labels were expanded to a large set of 270 ADNI hippocampi to allow the training of automated segmentation algorithms based on machine learning technology. Clearly, participation to the project was effortful and time consuming at all levels. Protocol developers were engaged in multiple hour-long teleconferences to tease out the finest details of their protocols, master tracers were pressed to carry out the assigned segmentations in due time, Delphi panelists were forced to go through similar sets of questions and issues over and over again until the convergence of most panelists, and naive tracers had to segment dozens (some of them hundreds) of hippocampi in a restricted time frame. Everyone’s engagement was astoundingly intense. Importantly, the HarP is the result of the concerted effort of many minds (Fig. 2). The working group met twice a year for the past 5 years and at each meeting the project was finetuned and its experimental design continuously improved thanks to input from participants. An off-workplan expansion of the number of segmented hippocampi was required by automated segmentation algorithm developers, and duly carried out. A large and representative set of certified hippocampal labels obtained with the HarP is now publicly available in the web that can be used to train and qualify human tracers and automated algorithms. Thanks to the HarP, it is now possible to directly compare the segmentation accuracy of the many automated