Eliminating the AI ‘Black Box’ to Make Clinical Data Usable

As Gen AI continues to mature, there will be ways to employ these technologies thoughtfully and safely in healthcare today. The key is to continue to embrace new breakthroughs – with strong guardrails for safety, privacy and transparency.

As recently as 2022, large language models (LLMs) were virtually unknown to the general public. Now consumers and entire industries around the world are experimenting with and deploying LLM-based software in what is now broadly termed ‘Generative AI’ to answer questions, solve problems, and create opportunities. 

But when it comes to using Gen AI in healthcare, clinicians and policymakers face the additional challenge of ensuring this technology is implemented safely to protect patients and securely to safeguard patient information. 

Clinicians understandably are wary about the quality of information they would receive from Gen AI platforms, because these programs tend to invent facts or “hallucinate” in ways that are difficult to prevent and predict. LLMs in many ways are considered “black boxes,” meaning how they work is not easily understandable, leading to a lack of accountability and trust. So while AI may provide clinical recommendations, it often can’t include links to data sources or the reasoning behind those recommendations. This makes it difficult for clinicians to exercise their own professional oversight without having to wade through vast amounts of data to “fact-check” AI.

AI also can be susceptible to intentional and unintentional bias based on how it’s trained or implemented. Further, bad actors who understand human nature may attempt to stray beyond the bounds of ethics to gain technical or economic advantage through AI. For these reasons, some form of government oversight is a welcomed step. The White House responded to these concerns last October by issuing an executive ordercalling for the safe and ethical deployment of this evolving technology. 

Mainstream foundational Gen AI models are not fit-for-purpose for many medical applications. But as Gen AI continues to mature, there will be ways to employ these technologies thoughtfully and safely in healthcare today. The key is to continue to embrace new breakthroughs – with strong guardrails for safety, privacy and transparency. 

Breakthroughs in medical-grade AI are advancing its safe use

Gen AI software performs analyses or creates output via the ability of LLMs to understand and generate human language. Thus, the quality of the outputs is impacted by the quality of source material used to build the LLMs. Many Gen AI models are built on publicly available information, such as Wikipedia pages or Reddit posts, that aren’t always accurate, so it’s no surprise that they may provide inaccurate outputs. That, however, simply isn’t tolerable in a clinical setting.

Fortunately, advances in medical AI now make it possible to leverage deep-learning models at scale for use in healthcare. Built by medical experts who understand the clinical relationships, terminologies, acronyms, and shorthand that are indecipherable or inaccessible to Gen AI software and traditional NLP, these experts are spawning the development of medical-grade AI for healthcare applications. 

LLMs today are being trained on massive sets of annotated medical data to operate accurately and safely within the healthcare industry. Essential to realizing this goal is the ability of well-trained LLMs and medical AI to access free-form clinical notes and reports and other unstructured text, which comprises about 80% of all medical data, based on industry estimates. 

Medical-grade AI developed in recent years can extract, normalize, and contextualize unstructured medical text at scale. Clinicians need AI systems that can ingest and understand a patient’s entire chart and data scientists and researchers need systems that can do the same for a health system’s entire EHR system. Medical-grade AI has been designed for enterprises to rapidly process and understand millions of documents in near-real-time, most of which are in unstructured form. This hasn’t been possible until now.

Reducing clinician burnout

Another area of concern is that if deployed inappropriately, Gen AI has the potential to drown its users in a firehose of unhelpful information. LLMs can also suffer from what is called the “lost in the middle” problem, where they fail to utilize information from the middle portions of long documents effectively.  For clinicians at the point of care, this results in frustration and wasted time searching through voluminous outputs for relevant patient data. As the amount of medical information available continues to grow, this promises to make it even harder to find and process the data clinicians need. Rather than making the jobs of clinical workers more manageable, Gen AI can exacerbate clinician burnout.

In contrast, medical-grade AI strikes a balance between recall and precision, giving clinicians just the right amount of accurate and relevant data for making sound evidence-based decisions at the point of care and linking information back to the original data in the patient’s chart. This provides transparency, allowing clinicians to check their sources of information for veracity and accuracy without a time-consuming search. By enabling clinicians to do their jobs more effectively and efficiently and spend more time focusing on patients, medical-grade AI can boost job satisfaction and performance while reducing time spent after-hours catching up on clerical work. 

Beyond the black box

The current opaqueness of Gen AI algorithms makes it premature to use them except in limited ways in healthcare and medical research. What clinicians want and need is information at the point of care that is accurate, concise, and verifiable. Medical AI has the ability now to meet these requirements while safeguarding patient data, helping to improve outcomes, and reducing clinician burnout. As all AI technologies continue to evolve, transparency – not black boxes – is critical to employing these technologies in the most efficacious and ethical ways to advance quality healthcare.