Artificial intelligence (AI) has the potential to transform healthcare as we know it. There have been dramatic announcements of AI solutions for almost every medical need including screening, triage, diagnosis, and treatment recommendation. The public interest, financial opportunities and enthusiastic coverage has meant that there are unusually strong drivers to accelerate the introduction and implementation of innovative AI health interventions. But how can we be sure that they will deliver on their promise to bring about benefits to patients and the public? We need high quality evidence before we can be truly confident that AI health interventions are ready for the real world.
Understanding the evidence gap
For any new treatment, test or other health intervention, we want to know that it is safe, clinically effective, and cost effective. This should be true for all interventions including those involving AI systems. The reality, however, is that, in most cases, the evidence for AI health interventions falls a long way short of this. There is a need to move from the initial ‘proof-of-concept’ studies which show the potential of the technology, to clinical trials that evaluate these AI systems within their intended clinical pathways and which measure the real benefits (or harms) to patients. This evaluation in the real world is critical if we want to rigorously and robustly evaluate the safety, clinical effectiveness, and cost effectiveness of AI health interventions.
Bridging the evidence gap
We are beginning to see the welcome emergence of prospective clinical trials evaluating AI health interventions in real-world settings. The evidence from these studies will support regulators, funders, and policy makers when making decisions about whether or not an AI health intervention should be commissioned. It is imperative, therefore, that these studies are conducted and reported to the highest standards. Minimum reporting guidelines for clinical trial protocols and reports have been instrumental in improving the quality of clinical trials and promoting completeness and transparency for the evaluation of new health interventions. The current guidelines – SPIRIT and CONSORT – are suited to traditional health interventions, such as drugs, but do not adequately address-AI specific issues. Furthermore our previous work highlighted the extent to which current studies fail to even reach these traditional standards: indeed fewer than one per cent of studies were of sufficient quality of design or reporting to enable confidence in their results.
SPIRIT-AI and CONSORT-AI extensions
The team at the University of Birmingham and University Hospitals Birmingham has worked with leading institutions from around the world to develop AI-specific extensions – SPIRIT-AI and CONSORT-AI – to the current guidelines. These new guidelines which have just been released were developed by a multi-disciplinary group of international experts using a consensus-building methodological process. The extensions include a number of essential new items that are considered sufficiently important to be reported in addition to the core items. Each item, where possible, was informed by challenges identified in existing studies of AI systems in health settings, meaning that these guidelines were developed using lessons learned in real life.
Confidence that AI is ready for the real world
The SPIRIT-AI and CONSORT-AI extensions will improve the quality of clinical trials evaluating AI systems through improvements in their design and delivery as well as the completeness and transparency of their reporting. Future clinical trials evaluating AI health interventions will be expected – and even required – to use them. These new international guidelines – led from the Birmingham Health Partners Centre for Regulatory Science and Innovation – will enable us to efficiently identify the safest and most effective AI health interventions and commission them with confidence for the benefit of patients and the public.
Authored by Professor Alastair Denniston, Lead for AI at Birmingham Health Partners Centre for Regulatory Science and Innovation, and Consultant Ophthalmologist at University Hospitals Birmingham.