Optical Biopsy of Bladder Cancer Using Crowd-Sourced Assessment

Crowdsourcing and optical biopsy are emerging technologies with broad applications in clinical medicine and research. Crowdsourcing, an interactive digital platform that uses multiple individual contributions to efficiently perform a complex task, has been successfully used in diverse disciplines ranging from performance assessment in surgery to optimization of tertiary protein conformations. 1,2 Optical biopsy technologies provide real-time tissue imaging with histology-like resolution and the potential to guide intraoperative decision making. 3-5 An example is confocal laser endomicroscopy (CLE), which can be used for the diagnosis and grading of bladder cancer. 6 To further assess the adoptability of optical biopsy as a diagnostic tool, we applied crowdsourcing to determine the barriers to learning how to diagnose cancer using CLE. We hypothesized that a nonmedically trained crowd could learn to rapidly and accurately distinguish between cancer and benign tissue. Methods | Amazon Mechanical Turk (Amazon.com) users were recruited as the crowd using a software platform developed by C-SATS. Each crowd worker first completed a validated training module 6 and answered a standard screening question, and then assessed a CLE video sequence randomly selected from a set of 12 sequences derived from a benign (n = 3) or cancer-ous (n = 9) urothelium (Figure 1). Videos were previously annotated by an expert user (J.C.L.), and diagnoses were confirmed by pathology under a Stanford University institutional review board–approved protocol. For a video to be categorized as showing a cancerous urothelium, correct classification by at least 70% of the crowd, which is the lowest statistical threshold for differentiation from random guessing, was required. Agreement with the expert user by at least 70% of crowd workers was also used to classify microscopic features with 2 categories (papillary structure, organization, morphology , cellular cohesiveness, and cellular borders). Microscopic vascular features with 3 categories were categorized based on a lower threshold of 35% agreement. Crowd workers were compensated 50¢ for each video assessed and blinded to patient history and diagnosis. Results | A total of 1283 ratings from 602 crowd workers were received in 9 hours, 27 minutes. A total of 1173 ratings were eligible for analysis based on correct screening response. The crowd accurately distinguished a cancerous urothelium from a benign urothelium in 11 of 12 video sequences (92%) (Figure 2). The single erroneous classification was of low-grade bladder cancer. In the assessment of microscopic characteristics , the crowds achieved the highest accuracy for cellular borders (10 of …