August 6, 2026
Can LLMs Reliably Extract and Code Clinical Data? What a 272-Note Benchmark Found
Executive Summary
- emtelligent evaluated purpose-built clinical AI, general-purpose LLMs, a task-specific medical coding model, and a RAG-assisted configuration on clinical concept extraction and SNOMED CT entity linking.
- The evaluation used 272 hospital discharge summaries and an expanded dataset containing nearly 79,000 labeled medical concepts.
- emtelligent’s purpose-built model achieved an 89.85% F1 score with a 0% hallucination rate in the evaluated extraction and coding task.
- The research comparison table reports 61.21% F1 for Claude Opus 4.6 and 55.44% F1 for GPT-5.2, with hallucination rates of 24.94% and 28.31%, respectively.
- The best RAG-assisted configuration reduced code hallucination to approximately zero while recording a 22.64% F1 score.
What did the healthcare LLM benchmark test?
Healthcare organizations are rapidly exploring large language models for summarization, clinical documentation, administrative workflows, research, and other applications.
Clinical data extraction introduces a specific technical requirement. A system has to identify clinically meaningful information in narrative text and connect those concepts to valid medical terminology.
emtelligent’s research examined how several AI approaches performed on this task.
The evaluation tested models on their ability to identify clinical concepts in hospital discharge summaries and map those concepts to SNOMED CT, a standardized clinical terminology used to represent medical concepts consistently across healthcare systems.
The comparison included:
- emtelligent’s purpose-built clinical AI
- KIRI, a third-party task-specific medical coding model
- Claude Opus 4.6
- GPT-5.2
- a RAG-assisted LLM configuration
The research specifically evaluates clinical concept extraction, entity linking, and coding. Results from this benchmark apply to those tasks and the configurations tested.
That scope matters when healthcare leaders evaluate AI claims. Summarization, clinical reasoning, coding, document classification, and entity linking are different technical problems, and performance in one area does not automatically predict performance in another.
What is clinical entity linking?
Clinical entity linking connects language written in a medical record to a standardized medical concept.
A physician may describe a diagnosis, medication, symptom, procedure, measurement, or finding using natural clinical language. Entity-linking technology identifies that information and connects it to the corresponding concept in a clinical terminology such as SNOMED CT.
The peer-reviewed results of the SNOMED CT Entity Linking Challenge describe the task as identifying clinical concepts in free text and linking them to concepts in a medical terminology so the information can be analyzed consistently by computers. The challenge dataset contained 74,808 annotations across the same 272 discharge notes used as the basis for emtelligent’s expanded evaluation.
This process can make information from clinical narratives more useful for search, analytics, research, coding, and downstream AI systems.
Where did the clinical data come from?
The evaluation was built around 272 real hospital discharge summaries drawn from MIMIC-IV.
The underlying SNOMED CT Entity Linking Challenge uses clinical notes from MIMIC-IV-Note. That dataset is hosted on PhysioNet, a global biomedical research platform managed by the MIT Laboratory for Computational Physiology that publishes curated, de-identified clinical data, software, models, and research resources.
emtelligent’s research team updated and expanded the dataset’s medical labels using a current version of SNOMED CT. According to the whitepaper, the resulting dataset contained nearly 79,000 labeled medical concepts, with approximately:
- 55,000 concepts used for training
- 24,000 concepts reserved for testing
The research also describes using an AI language model as an independent reviewer to flag labels that were incorrect, contextually mismatched, or imprecise.
Explore the SNOMED CT Entity Linking Challenge on PhysioNet
How was performance measured?
The benchmark reports two important measures: F1 score and hallucination rate.
F1 score
F1 combines precision and recall into a single measure.
For clinical entity linking, that means the score reflects both how accurately the system identifies clinical concepts and how completely it finds the concepts that should have been identified.
A strong F1 score therefore requires a model to balance correctness with coverage.
Hallucination rate
The term hallucination has a specific definition in this research.
Here, hallucination measures fabricated or invalid medical codes generated during the evaluated coding task.
This definition is narrower than the broad use of “AI hallucination” to describe any unsupported information generated by a language model. Keeping that distinction clear helps healthcare organizations understand exactly what the benchmark measured.
What did the benchmark find?
The model-specific comparison table in emtelligent’s research reports the following results:
| Model | F1 Score | Hallucination Rate |
|---|---|---|
| emtelligent purpose-built clinical AI | 89.85% | 0.00% |
| KIRI, task-specific benchmark | 77.50% | 0.00% |
| Claude Opus 4.6 | 61.21% | 24.94% |
| GPT-5.2 | 55.44% | 28.31% |
| Best RAG-assisted configuration | 22.64% | ~0.00% |
These numbers describe performance on the clinical entity-linking and coding evaluation conducted for this research.
emtelligent purpose-built clinical AI
emtelligent’s purpose-built model achieved an 89.85% F1 score with a reported 0% hallucination rate at the evaluated extraction and coding layer.
The research attributes code validity to an architecture that validates outputs against a closed clinical ontology, constraining the coding layer to valid concepts.
KIRI
KIRI, an existing third-party model developed for medical coding and included as an independent task-specific benchmark, achieved a 77.50% F1 score with a reported 0% hallucination rate.
KIRI was also the winning model in the original SNOMED CT Entity Linking Challenge. The peer-reviewed challenge paper describes KIRI as a dictionary-based approach that used external resources to expand its training dictionary and link spans of clinical text to SNOMED CT concepts.
Claude Opus 4.6
The Claude Opus 4.6 configuration reported in emtelligent’s comparison table achieved a 61.21% F1 score with a 24.94% hallucination rate.
GPT-5.2
The GPT-5.2 configuration reported in the table achieved a 55.44% F1 score with a 28.31% hallucination rate.
The general-purpose models were able to identify and link clinical concepts, although their reported F1 scores were lower and their invalid-code generation rates were higher than the purpose-built approaches included in this evaluation.
What happened when RAG was added?
Retrieval-Augmented Generation, commonly called RAG, gives a language model retrieved information or candidate answers that can help ground its response.
Healthcare organizations are exploring RAG for many purposes, including search, question answering, summarization, and clinical information retrieval.
emtelligent’s benchmark tested a RAG-assisted approach specifically for the SNOMED CT entity-linking task.
The best RAG-assisted configuration reduced the reported hallucination rate to approximately 0%. Its F1 score was 22.64%.
The research reports that recall fell substantially, meaning the configuration failed to identify many of the relevant clinical concepts in the test data.
The finding is specific to the task and RAG configuration evaluated. Different retrieval systems, models, datasets, and applications can produce different results.
For teams designing clinical-data pipelines, the result highlights an important consideration: code validity and overall extraction performance need to be measured separately.
A configuration can substantially reduce fabricated codes while still missing clinically relevant concepts.
Where do general-purpose LLMs fit in healthcare AI?
Large language models provide capabilities that can be extremely useful in healthcare.
They can support:
- summarization
- conversational interfaces
- synthesis across information
- natural-language interaction
- reasoning over supplied information
- generative workflows
Clinical extraction and terminology mapping bring additional requirements. Medical concepts need to be identified accurately, connected to accepted terminology, and represented with enough context for downstream systems to use them.
This is where architecture becomes important.
emtelligent’s Medical Language Engine currently combines clinical data extraction, medical document AI, and AI-assisted chart review. Its clinical NLP capabilities include entity linking, polarity and uncertainty detection, experiencer detection, measurement and temporality relationships, and medication identification.
The research supports using purpose-built clinical technology to create structured, validated inputs that LLMs can then reason over or summarize.
That approach also reflects what healthcare leaders are discussing in the market. Following HIMSS26, Kim Perry told Healthcare IT News that organizations were increasingly asking how LLMs could be combined with purpose-built technology and data pipelines to produce more accurate, efficient, and sustainable solutions.
Read Kim Perry’s HIMSS26 perspective in Healthcare IT News
What makes a purpose-built clinical data layer different?
Clinical data contains more than medical terms.
Context can change the meaning of a concept entirely.
Consider the difference among:
- a patient has a condition
- a patient does not have a condition
- a clinician suspects a condition
- the patient previously had a condition
- a condition belongs to the patient’s family history
Systems working with clinical language need ways to represent those distinctions.
emtelligent’s current Medical Language Engine supports entity linking along with clinical assertions and relationships. Its website lists capabilities including polarity and uncertainty detection, experiencer detection, measurement and temporality relations, and medication identification.
The platform also supports document processing and traceability, maintaining connections between extracted information and source documents for workflows where users need to verify results.
Explore emtelligent’s clinical AI capabilities
Comparison: general-purpose LLMs and a purpose-built clinical data layer
| Capability | General-Purpose LLM | Purpose-Built Clinical Data Layer |
|---|---|---|
| Language generation | Core strength | Can supply structured inputs for summarization |
| Summarization | Core strength | Can supply structured inputs for summarization |
| Clinical concept extraction | Possible, performance varies | Designed for clinical extraction |
| Medical ontology validation | Depends on implementation | Can be built directly into the coding workflow |
| Invalid-code prevention | Requires controls around generation | Closed-ontology validation can constrain outputs |
| Structured clinical output | Requires configuration | Core capability |
| Assertions and clinical relationships | Model and architecture dependent | Explicitly modelled in emtelligent’s clinical NLP |
| Source traceability | Requires surrounding architecture | Supported in emtelligent’s document and review workflows |
Capabilities vary among models and implementations. This comparison reflects the architectural distinction examined by emtelligent’s research and current clinical AI platform.
Why does clinical data validation matter?
A clinical-data error can move downstream into other systems and workflows that depend on that information.
emtelligent’s whitepaper highlights several examples.
Risk adjustment and compliance
Risk adjustment workflows depend on accurate clinical information and coding. Incorrect or fabricated codes can introduce risk when information is used in payer programs or CMS submissions.
Explore emtelligent solutions for payers
Clinical quality and revenue integrity
Clinical documentation and coding feed quality and financial workflows. Errors introduced during extraction can carry into processes that depend on that data.
Research and evidence generation
Real-world evidence and clinical research depend on reliable datasets. Incorrect clinical concepts can affect the quality of information available for trial design, post-market research, and other analyses.
The impact also changes with scale. Even a modest error rate can produce a large volume of problematic outputs when applied across millions of documents.
Data quality therefore deserves its own evaluation when healthcare organizations assess an AI system.
Kim Perry makes a similar point in her 2026 Healthcare Business Today article, where she argues that AI initiatives depend heavily on the quality and context of the underlying clinical data.
Read Clean Before You Compute in Healthcare Business Today
What are some best practices for evaluating LLMs for clinical data extraction?
Healthcare organizations can ask a few basic questions before comparing AI performance claims.
Ask what task was actually benchmarked
Clinical AI evaluations can measure very different capabilities:
- document classification
- concept extraction
- entity linking
- coding
- summarization
- question answering
- clinical reasoning
A benchmark result becomes much more useful when the task is clearly defined.
Ask which dataset was used
The dataset determines the type of information a system encountered during testing.
The emtelligent study was built around 272 hospital discharge summaries associated with the SNOMED CT Entity Linking Challenge and MIMIC-IV clinical data.
The original challenge and dataset are publicly documented, making it possible to understand the clinical notes and annotation task behind the research.
Ask which metric is being reported
Accuracy, precision, recall, F1 score, and hallucination rate measure different aspects of performance.
A strong evaluation should identify the metric alongside the task and dataset.
For example, 89.85% F1 should be described as an F1 score. Calling it “89.85% accuracy” would imply a different metric.
Ask how medical codes are validated
Organizations using AI for standardized terminology should understand how the system generates and validates clinical codes.
Questions include:
- Can the model freely generate a code?
- Are outputs checked against an accepted ontology?
- What happens when a concept cannot be confidently mapped?
- How is code fabrication measured?
Ask how much relevant information the system misses
A low hallucination rate alone does not capture extraction quality.
The RAG portion of this benchmark illustrates why. The tested configuration substantially reduced fabricated codes while its F1 score fell to 22.64%.
Precision, recall, F1, and code validity together provide a more complete picture.
Ask whether results can be traced to the source
Reviewers may need to verify where a diagnosis, measurement, medication, or other finding came from.
emtelligent’s current product capabilities include document traceability designed to connect information back to the underlying source.
OCR vs. Clinical AI: A Practical Comparison
| Capability | Traditional OCR | Purpose-Built Clinical AI |
|---|---|---|
| Converts scanned documents into searchable text | ✓ | ✓ |
| Understands medical terminology | — | ✓ |
| Identifies diagnoses, symptoms, and medications | — | ✓ |
| Recognizes negated findings | — | ✓ |
| Maps concepts to SNOMED CT, ICD-10, LOINC, and RxNorm | — | ✓ |
| Preserves clinical context and temporality | — | ✓ |
| Produces traceable clinical evidence | — | ✓ |
| Supports downstream healthcare workflows | Limited | ✓ |
| Creates AI-ready structured clinical data | — | ✓ |
Is Your Organization Ready for Clinical AI?
Healthcare leaders evaluating AI initiatives should consider a few foundational questions before investing in new technologies.
Clinical Data
- Can we ingest both scanned and digital clinical documents?
- Can we identify clinically meaningful concepts rather than simple keywords?
- Can every extracted concept be traced back to its original documentation?
- Are concepts normalized using standardized clinical vocabularies?
Workflow Integration
- Does structured data integrate with our existing systems?
- Can clinicians validate AI-generated outputs?
- Does the technology reduce manual work rather than create new processes?
- Can extracted information be reused across multiple departments?
Governance
- Are outputs transparent and auditable?
- Can we measure operational impact?
- Does the solution support enterprise-scale deployment?
- Can we establish clear ownership and governance?
Organizations that can confidently answer “yes” to these questions are generally better positioned to scale AI responsibly.
What does this research show, and where are its limits?
Clear scope makes benchmark results more useful.
The emtelligent research evaluates:
- clinical concept identification
- clinical entity linking
- SNOMED CT coding
- fabrication of invalid medical codes
- purpose-built clinical AI
- a task-specific medical coding model
- general-purpose LLM configurations
- a RAG-assisted configuration
The research does not directly compare:
- general clinical reasoning
- patient-chart summarization
- patient-facing AI
- every RAG architecture
- every healthcare LLM implementation
- every clinical terminology
- every healthcare workflow
Those applications require evaluations designed around their own tasks and outcomes.
The strongest conclusion supported by this research is specific: emtelligent’s purpose-built clinical approach achieved higher F1 performance and a lower reported code-hallucination rate than the general-purpose LLM configurations included in this SNOMED CT entity-linking benchmark.
Why this matters now
Healthcare organizations are moving further into production AI.
At HIMSS26, industry leaders described a growing focus on practical implementation, governance, workflow integration, ROI, and the role of LLMs within larger healthcare technology architectures. Kim Perry specifically highlighted the question of how organizations can combine LLMs with purpose-built technology and data pipelines to improve accuracy, efficiency, and sustainability.
As AI systems move closer to clinical and operational workflows, healthcare teams need to look beneath the interface and ask:
- What was the model tested on?
- What task was measured?
- Which performance metric was used?
- How much clinically relevant information was missed?
- Can the system generate invalid medical codes?
- How are outputs validated?
- Can a reviewer trace a finding back to its source?
Those questions help organizations understand where each AI technology fits and how much confidence to place in its output.
The findings from emtelligent’s research suggest that the clinical data layer deserves as much attention as the LLM operating on top of it.
Ready to explore a purpose-built clinical data foundation for your AI strategy?
Visit emtelligent.com to learn more about the Medical Language Engine and its clinical data extraction, medical document AI, and AI-assisted chart review capabilities.
For the complete methodology and benchmark findings, download Your Healthcare AI Strategy Has a Gap from emtelligent’s research library. View the full research whitepaper
Key Takeaways
- Purpose-built AI outperforms general-purpose LLMs in clinical coding.
- General-purpose LLMs frequently produce invalid medical codes.
- Adding RAG eliminates hallucinations but drastically degrades recall.
- Rigorous benchmarking requires multi-metric evaluation.
- Contextual accuracy depends on structured assertions.
About the Author
Kim Perry is Chief Growth Officer at emtelligent, where she leads growth strategy and works with health systems, payers, life sciences organizations, and healthcare technology companies to turn unstructured clinical data into actionable intelligence. She brings more than two decades of healthcare industry experience across Fortune 500 companies, high-growth technology organizations, and private equity-backed businesses.
About emtelligent
emtelligent helps healthcare organizations unlock the value of unstructured clinical data through purpose-built clinical AI.
Its Medical Language Engine transforms physician notes, pathology reports, discharge summaries, radiology reports, referrals, and other clinical documentation into structured, standardized, and traceable clinical intelligence that supports providers, payers, life sciences organizations, healthcare technology companies, and research institutions.
Related Resources
Internal Resources
Your Healthcare AI Strategy Has a Gap
The complete research whitepaper behind this article.
View the research
Medical Language Engine
Learn how emtelligent structures unstructured clinical information for downstream healthcare workflows.
Explore the Medical Language Engine
Clinical AI Features
Explore clinical data extraction, document processing, traceability, chart review, and related capabilities.
Explore emtelligent features
Solutions for Payers
Explore applications across risk adjustment, quality, prior authorization, payment integrity, care management, and related payer workflows.
Explore payer solutions
Company and Leadership
Learn more about Kim Perry and the emtelligent team.
About emtelligent
External References
Healthcare IT News, April 2, 2026
Read the article
Clean Before You Compute: The Data Foundation Payers Can’t Afford to Ignore
Kim Perry, Healthcare Business Today, January 7, 2026
Read the article
SNOMED CT Entity Linking Challenge
Journal of the American Medical Informatics Association, 2025
Read the peer-reviewed study
SNOMED CT Entity Linking Challenge
SNOMED International
Explore the challenge
PhysioNet
MIT Laboratory for Computational Physiology
Learn about PhysioNet
MIMIC-IV-Note
PhysioNet
Explore MIMIC-IV-Note