The Lack of Rigor in AI Research, Beware of the Hype

This blog aims to build on Nada’s ophthalmology blog, raising awareness of the hype surrounding AI, and talks about the issue of health data poverty and why we need to address this.

The lack of rigour in AI research, beware of the hype

The literature within the AI healthcare field has grown considerably in the last three years. Several reviews have discussed current and future use-cases across healthcare, and NHSX have summarised these applications (see Figure 1) within their report: ‘Artificial Intelligence: How to get it right’. Topol outlined how the benefits would be impactful at three levels:

·       Clinicians- primarily assisted by rapid diagnostic aid in patient management (particularly image interpretation e.g radiology, pathology, ophthalmology [as Nada discussed]), and automation of repetitive administrative tasks to free up time.

·       Health systems- operational applications in non-patient facing, back-end systems that improve workflow, and a reduction in variability and in medical errors.

·       Patients- process patients’ health data to empower them in their own health promotion.

Figure 1: Summary of AI use-cases across different domains in medicine in the NHSX report: ‘Artificial Intelligence: How to get it right’.

However, it seems AI is still in its infancy. Despite the excitement surrounding AI, there is surprisingly limited employment of AI systems in clinical practice. In fact, within the recent NHSX report, a national survey and horizon scan of evidence have found that AI is still in the early stages of the Gartner Hype Cycle (figure 2).

Figure 2: The Gartner Hype Cycle

Liu et al conducted the first ever systematic review of AI research, comparing the performance of deep learning algorithms with healthcare professionals in detecting disease from medical imaging. They analysed 20,500 studies and found the majority was retrospective, lacking in transparency and at a high risk of bias. It should be noted that several of the high-quality studies did report that AI could detect disease as accurately as health professionals, although they had not substantially outperformed in accuracy. A more recent systematic review by Nagendran et al supported these findings and stated that evidence standards must improve to enhance real world clinical relevance.

This paper by Kelly et al, a team from Google’s deep mind outlines key challenges to translating AI to clinical practice. It is also promising that recent papers by Faes et al and Vollmer et al provide clearer standards for AI research, and hopefully the development of the new CONSORT-AI and SPIRIT-AI guidelines will address the rigour of research being produced.

Health data poverty and the risk of AI exacerbating inequalities

A viewpoint has warned of the risk of reinforcing healthcare inequalities if new AI health technologies are based on unrepresentative datasets. This was following a global review of publicly available datasets for ophthalmological imaging. They identified 94 open access datasets containing 507,724 images and 125 videos from 122,364 patients. They detailed the source of each dataset, its accessibility, and the populations, diseases and types of images represented within it. A key finding was that most datasets originated from Asia, North America, and Europe. In addition, disease populations were unevenly represented, with glaucoma, diabetic retinopathy, and age-related macular degeneration disproportionately overrepresented in comparison with other eye diseases; most likely due to these images being collected routinely in screening and healthcare services in modern healthcare systems. Finally, the reporting of basic demographic characteristics such as age, sex, and ethnicity was poor, including at the aggregate level.

So why is the lack of geographical diversity and demographic data in these datasets concerning? As Nada touched on, machine learning (ML) models are only as good as the datasets used to train, test and validate them. If these datasets are not representative of the target population the AI models will be implemented in, they may not work effectively and could pose a risk to patient safety and exacerbate health disparities. If basic demographic metadata is lacking, how can we even discern whether the datasets used to train models are representative of different subgroups.

A blog by Luke Oakden-Rayner, an expert in medical AI, discussed how AI has the ability to trivially recognise the self-reported race of patients. AI has also been able to identify the gender of a patient from photographs of a fundus, something which no clinician is able to do. This therefore provides direct mechanisms for models to incorporate the biases in medical practice into their decisions. We cannot let this happen and must take steps to mitigate this.

Clearly this is an important issue and the researchers have stated that health systems and researchers must place greater focus to invest in the development of high-quality datasets and address demographic gaps. There must also be clearer regulatory standards for AI models to report the demographic data their dataset was trained on, and detail how their model performs on different population subgroups.

This is an evolving issue and one which experts are still trying to figure out how to address. But hopefully, this blog has brought your attention to this issue and highlights how there are still many challenges to clinically translate AI, before it can assist clinicians in improving the delivery of care and patient outcomes.

Here are some additional resources to get you started if you would like to learn more about this area:

·       If you would like to learn more about medical AI, consider watching some of Dr Chris Lovejoy’s medical AI youtube series.

·       There are also some podcast episodes by Mustafa Sultan to help you get more insight into the key issues: Bias in medical AI- Prof Obermeyer + SPIRIT-AI and CONSORT-AI- Prof Deniston & Dr Xiao Liu

·       Here are 4 papers regarding Bias in medical AI on a new initiative called Explainthispaper, where researchers break down the paper to make it easier to understand.

Written by Shaswath Ganapathi

References

1.     Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, et al. A guide to deep learning in healthcare. Nat Med. 2019;25(1):24–9.

2.     Becker A. Artificial intelligence in medicine: What is it doing for us today? Health Policy Technol. 2019 Jun 1;8(2):198–205.

3.     Loh E. Medicine and the rise of the robots: a qualitative review of recent advances of artificial intelligence in health. BMJ Lead. 2018 Jun 1;2(2):59–63.

4.     Amisha, Malik P, Pathania M, Rathaur VK. Overview of artificial intelligence in medicine. J Fam Med Prim Care. 2019 Jul;8(7):2328–31.

5.     Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017 Dec 1;2(4):230–43.

6.     Hashimoto DA, Rosman G, Rus D, Meireles OR. Artificial Intelligence in Surgery: Promises and Perils. Ann Surg. 2018 Jul;268(1):70–6.

7.     Ahuja AS. The impact of artificial intelligence in medicine on the future role of the physician. PeerJ [Internet]. 2019 Oct 4 [cited 2019 Dec 4];7. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6779111/

8.     Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019 Jun;6(2):94–8.

9.     NHSX_AI_report.pdf [Internet]. [cited 2019 Nov 4]. Available from: https://www.nhsx.nhs.uk/assets/NHSX_AI_report.pdf

10.   Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019 Jan;25(1):44–56.

11.   Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019 Oct 29;17(1):195.

12.   Hype Cycle Research Methodology [Internet]. Gartner. [cited 2020 Apr 4]. Available from: https://www.gartner.com/en/research/methodologies/gartner-hype-cycle

13.   Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. 2019 Oct 1;1(6):e271–97.

14.   Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ [Internet]. 2020 Mar 25 [cited 2020 Mar 29];368. Available from: https://www.bmj.com/content/368/bmj.m689

15.   Faes L, Liu X, Wagner SK, Fu DJ, Balaskas K, Sim DA, et al. A Clinician’s Guide to Artificial Intelligence: How to Critically Appraise Machine Learning Studies. Transl Vis Sci Technol. 2020 Jan 28;9(2):7–7.

16.   Vollmer S, Mateen BA, Bohner G, Király FJ, Ghani R, Jonsson P, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ [Internet]. 2020 Mar 20 [cited 2020 Mar 29];368. Available from: https://www.bmj.com/content/368/bmj.l6927

17.   Best interests decision making for adults who lack capacity toolkit [Internet]. The British Medical Association is the trade union and professional body for doctors in the UK. [cited 2021 Oct 28]. Available from: https://www.bma.org.uk/advice-and-support/ethics/adults-who-lack-capacity/best-interests-decision-making-for-adults-who-lack-capacity-toolkit