AI News, A artificial intelligence
Use of Crowd Innovation to Develop an Artificial Intelligence–Based Solution for Radiation Therapy Targeting
A 3-phase, prize-based crowd innovation challenge over 10 weeks, including 34 contestants who submitted 45 algorithms, identified multiple AI solutions that replicated the accuracy of an expert radiation oncologist in targeting lung tumors and performed the task more rapidly.
Radiation therapy (RT) plays a critical role in the treatment of this disease, and 20% of early stage (I-II) and 50% of advanced stage (III-IV) lung cancer patients receive RT2 with projections of approximately 96 000 patients requiring this treatment modality in 2020.2,3 The precise and accurate volumetric segmentation of tumors, which determines where the radiation dose is delivered into the patient, is a critical part of RT targeting and planning, and has a direct impact on tumor control and radiation-induced toxic effects.
However, there is significant interobserver variation even among experts (eg, 7-fold variation among 5 experts in 1 study),4,5 and the quality of segmentation may directly impact clinical outcomes.6-8 Even in prospective clinical trials with prespecified RT parameters, major RT planning deviations occur in 8% to 71% of patients and are associated with increased mortality and treatment failure.9 Unlike cancer image analysis for diagnostic purposes, which produces a single binary answer (yes or no) to a single question (“Is a mass present?”), therapeutic tumor segmentation involves interpretation of medical imaging on a voxel-by-voxel basis to classify cancer vs normal organ and incorporates an intrinsic risk-benefit assessment of where the radiation dose is to be delivered.
For an expert radiation oncologist, this requires both training and intuition, and experience may directly impact lung cancer outcomes.10 However, this critical human resource is not accessible to many underserved patients in both the United States and globally.11,12 Although approximately 58% of lung cancer cases occur in less developed countries,13 these countries have a staggering shortage of radiation oncologists, with an estimated 23 952 radiation oncologists required in 84 low- and middle-income countries by 2020 yet only 11 803 were available in 2012.14 We used a novel combined approach of crowd innovation and artificial intelligence (AI) to address this unmet need in global cancer care.
Crowd innovation has been successfully applied to a variety of genomic and computational biology problems by using prize-based competitions to identify extreme value solutions that outperform those developed by conventional academic approaches.15-18 Online contests expand the pool of potential problem solvers substantially beyond traditional academic expert circles to include individuals with a more diverse set of skills, experience, and perspectives.
Examples include diagnosis of skin cancers from photographs,19 lung cancer on screening CT images,20,21 retinal diseases using optical coherence tomography,22 and breast cancer using mammograms,23 or pathology specimens.24 However, applying AI techniques to therapeutic processes in medicine has not been equally well explored because of a lack of large data sets that are well curated by medical experts, and the need for AI techniques capable of adjusting to alterations in practice patterns or risk tolerance and style of individual treating physicians for specific diseases.
To address the global shortage of expert radiation oncologists, we designed a crowd innovation contest to challenge an international community of programmers to rapidly produce automated AI algorithms that could replicate the manual lung tumor segmentations of an expert radiation oncologist.
well-curated lung tumor data set segmented by an expert clinician for contestants to train and test their algorithms.An objective scoring system for automatic evaluation of submitted algorithms to provide contestant feedback and final rankings.Motivating and guiding contestants to produce clinically relevant solutions with a cost-effective prize pool, information sharing, and access to feedback from the expert clinician.
After review of phase 1 performance identified deficiencies in tumor localization as the major limiting factor, the contest was redesigned in phase 2 to allow contestants to focus their efforts on the therapeutic task of lung tumor targeting, as opposed to the distinct task of tumor diagnosis.
total of 564 contestants from 62 countries registered for this challenge, and 34 (6%) submitted algorithms, including 244 unique submissions in phase 1 (mean, 8.4/participant), 164 in phase 2 (mean, 14.9/participant), and 180 in phase 3 (mean, 36/participant).
Although the ensemble solution did not exceed the performance of the intraobserver benchmark (Figure 2 and Figure 3), approximately 75% of ensemble segmentations exceeded an S score of 0.60 (the lower threshold of intraobserver performance) which suggests that the contest produced algorithms capable of matching expert performance.
The results of this study show that a combined approach that leverages crowd innovation to access computational expertise to develop AI algorithms coupled with human medical expert feedback can enable rapid development of multiple solutions for a complex medical task with performance comparable to human experts.
Developing AI solutions for time-intensive tasks such as tumor segmentation can increase productivity and time with patients for busy clinicians by reducing computer-based work, and solve the known oncology workforce crisis (eg, number of trained radiation oncologists) in under-resourced health care systems worldwide.12 Artificially intelligent algorithms can also replicate and transfer expert-level knowledge for education, training, and/or quality assurance to raise the quality of global RT care.
Providing quality assurance for RT trials is particularly important because variation in radiation planning even in highly structured protocols may be substantial enough to negatively impact outcomes and drive the results of trials toward the null.9 In addition, the ability to generate automatic tumor segmentations rapidly and accurately could revolutionize therapeutic response assessment in oncology in general, by allowing quantitative assessments of tumor imaging features during and after treatment, which may provide better predictive capability than traditional, manual, linear measurements (eg, RECIST).33,34 Deep learning methods like CNNs have been increasingly used for visual pattern recognition to automate important diagnostic tasks in medicine.19,21,24,35,36 For the therapeutic task of targeting lung tumors, the top algorithms produced by this challenge (Dice = 0.79 compared with human expert) performed comparably to algorithms used to detect/segment pathologic entities in prior studies, including invasive breast cancer (Dice = 0.76),24 and brain white matter hyperintensities (Dice = 0.79).35 Previous work to apply deep learning to RT by academia and private industry include automated AI segmentation for both tumor targets37,38 and normal organs.39-41 Our top algorithms compared favorably against these limited studies as well (eg, Dice = 0.81 for nasopharyngeal tumors).37 Furthermore, this study demonstrated that crowdsourced AI algorithms significantly outperform existing commercially-available, semi-automated segmentation tools, which have historically focused on atlas-based,42 PET-based,43 and single click region grow auto-segmentation44 approaches.
Although past contests have leveraged crowds to produce AI solutions to problems in diagnostic oncology including in the 2017 Kaggle Bowl (early lung cancer detection in low-dose CT screening scans)45 and the 2016 DREAM Challenge (identifying breast cancer on digital mammograms)46,47 which also used a phased approach, the study reported here applied a multiphase crowd innovation approach with collaborative components to address a therapeutic problem.
Although on-demand crowd workers are now widely used in a range of tasks (eg, ride sharing, labor market platforms like UpWork/Freelancer), our research demonstrates that crowds can also augment and complement traditional academic research with considerable cost, time, and performance benefits.
As demand for AI talent increases across the economy, and new AI methods rapidly evolve across a range of academic and industrial settings, this study demonstrates that a relatively low-cost, crowd innovation approach can be used to democratize the development of AI solutions in oncology beyond traditional academic and industrial circles, by enabling individual oncologists to access AI expertise, on demand to improve their own clinical and research practices.
Furthermore, the production of the lung tumor segmentations relied on a single human expert, and the contest’s AI algorithms may have acquired the natural biases of that expert and generated segmentations representative of the training, experience, and judgment of 1 individual rather than objective ground truth (eFigure 6 in the Supplement).
Artificial intelligence is on the brink of a 'diversity disaster'
The consequences of this issue are well documented, from hate speech-spewing chatbots to racial bias in facial recognition.
Report author Kate Crawford said that the industry needs to acknowledge the gravity of the situation, and that the use of AI systems for classification, detection and predication of race and gender 'is in urgent need of re-evaluation.'
Speaking to The Guardian, Tess Posner, CEO of AI4ALL, which seeks to increase diversity within AI, says the sector has reached a 'tipping point,' and added that every day that goes by it gets more difficult to solve the problem.
The artificial intelligence field is too white and too male, researchers say
The artificial intelligence industry is facing a “diversity crisis,” researchers from the AI Now Institute said in a report released today, raising key questions about the direction of the field.
Women and people of color are deeply underrepresented, the report found, noting studies finding that about 80 percent of AI professors are men, while just 15 percent of AI research staff at Facebook and 10 percent at Google are women.
Diversity, while a hurdle across the tech industry, presents specific dangers in AI, where potentially biased technology, like facial recognition, can disproportionately affect historically marginalized groups.
“It can be seen in unequal workplaces throughout industry and in academia, in the disparities in hiring and promotion, in the AI technologies that reflect and amplify biased stereotypes, and in the resurfacing of biological determinism in automated systems.”
Artificial intelligence performs as well as experienced radiologists in detecting prostate cancer
UCLA researchers have developed a new artificial intelligence system to help radiologists improve their ability to diagnose prostate cancer.
The system, called FocalNet, helps identify and predict the aggressiveness of the disease evaluating magnetic resonance imaging, or MRI, scans, and it does so with nearly the same level of accuracy as experienced radiologists.
However, it typically takes practicing on thousands of scans to learn how to accurately determine whether a tumor is cancerous or benign and to accurately estimate the grade of the cancer.
scans were fed into the system so that it could learn to assess and classify tumors in a consistent way and have it compare the results to the actual pathology specimen.