Impact and Challenges of Large Language Models in Healthcare

Large language models (LLMs) continue to revolutionize healthcare delivery and patient engagement. However, as these models have matured, one challenge has emerged as the defining concern: context management. When trained on robust health datasets and provided with the right contextual information at the right time, LLMs can play a crucial role in enhancing patient care through data-driven insights.

This guide explains the current state and challenges of large language models in healthcare, and how organizations can implement them responsibly with proper context management strategies. Here is what we’ll cover:

• FAQs about LLMs in healthcare
• The challenges of LLMs in healthcare
• 4 practical steps for implementing healthcare LLMs
• The key to success with healthcare LLMs: Context-aware AI architecture

FAQs about LLMs in healthcare

What is an LLM?

LLMs are deep learning models trained on massive neural networks. In essence, LLMs can process vast sequences of text and extract meaning — fast. Since mid-2024, these models have become dramatically more capable, with context windows expanding to 200K+ tokens and costs dropping by 80-90%.

When applied to healthcare, large language models offer proven solutions for improving patient care and streamlining processes. 

What is the role of LLMs in medicine?

LLMs can answer questions, summarize text, paraphrase complicated jargon, and translate words into different languages. They can now also use tools, call external systems, and orchestrate complex multi-step workflows. From quickly scanning a long patient file to helping patients understand their diagnosis to autonomously scheduling follow-up care, these models are transforming healthcare delivery.

What are the applications of medical LLMs?

Medical providers now rely on large language models in healthcare to:

• Streamline administrative tasks: Clinicians spend roughly 33% of their workday on activities outside of patient care. LLMs now automate scheduling, prior authorization requests, referral management, and follow-up care instructions with minimal human oversight.
• Manage clinical documentation: LLMs summarize patient notes and medical histories, extract structured data from unstructured text, and generate draft care plans. Modern implementations use retrieval-augmented generation (RAG) to pull relevant context from electronic health records (EHRs) in real-time.
• Proactively detect adverse events: Using data from EHRs, LLMs can automatically detect patterns indicating potential adverse health events, drug interactions, or care gaps that require intervention.
• Orchestrate care workflows: Agentic LLM systems can manage complex, multi-step care coordination tasks — from identifying high-risk patients to coordinating outreach to tracking engagement to escalating to clinicians when needed.

Despite the extensive benefits of LLMs, implementing them effectively in healthcare environments presents unique challenges.

The challenges of LLMs in healthcare

Context management

The healthcare industry has learned a crucial lesson: the quality of LLM outputs is almost entirely determined by the context provided. While early concerns focused on model accuracy and hallucinations, the real challenge is ensuring models have access to the right information at the right time.

Brendan Smith-Elion, VP of Product Management at Arcadia, describes this as “the context problem.”

“These models are incredibly capable, but they’re only as good as the information you feed them. The hard part isn’t the AI — it’s architecting systems that can dynamically assemble the relevant patient data, clinical guidelines, organizational policies, and real-time information the model needs to make good decisions.”

Let’s review several factors that have made context management the primary concern.

The rise of the Model Context Protocol (MCP)

Anthropic’s Model Context Protocol, released in 2024, standardized how LLMs connect to data sources. This has accelerated adoption but also revealed the complexity of healthcare data integration. Organizations now need strategies for:

• Connecting LLMs to EHRs, claims systems, HIEs, and external data sources
• Managing permissions and data access patterns across systems
• Ensuring context freshness and accuracy
• Orchestrating multiple context sources for complex queries

Agentic architectures demand better context

Modern healthcare LLM applications aren’t single-shot queries — they’re autonomous agents that make decisions over minutes or hours. These agents need to maintain context across multiple steps, tools, and data sources. Poor context management leads to agents that lose track of patient state, repeat actions, or make decisions on stale data.

Regulatory scrutiny on data provenance

The U.S. Food and Drug Administration (FDA) and Office of the National Coordinator for Health Information Technology (ONC) now require clear documentation of the data used to inform AI decisions. This means tracking not just model outputs, but the complete context provided to models — a significant architectural challenge.

Implementation challenges

Healthcare organizations also encounter challenges in implementing these new solutions, including:

Context assembly and orchestration

The biggest challenge isn’t model capability — it’s assembling the right context from disparate healthcare systems. Healthcare organizations must:

• Build context pipelines: Develop systems that can query EHRs, claims databases, lab systems, ADT feeds, and external sources in real-time
• Implement smart retrieval: Use vector databases and semantic search to find relevant patient history, similar cases, and applicable guidelines
• Manage context windows strategically: Even with 200K token windows, you must prioritize what information matters most for each use case
• Handle context freshness: Ensure lab results, medication changes, and care plan updates are reflected immediately

Model drift and knowledge boundaries

LLMs constantly receive updates, but healthcare organizations need stability. Modern approaches include:

• Versioning strategies: Lock models to specific versions for regulated use cases
• Hybrid architectures: Use general-purpose LLMs for reasoning but fine-tuned smaller models for domain-specific medical knowledge
• RAG as a hedge: Retrieval-augmented generation (RAG) reduces reliance on model training data by grounding responses in your organization’s current documentation

Trust, transparency, and auditability

Healthcare users demand to know why an AI made a recommendation. This requires:

• Explainable context chains: Track what data the model accessed and how it influenced the output
• Human-in-the-loop by design: Structure workflows so clinicians verify AI decisions at critical points
• Audit trails for compliance: Log all context provided, model responses, and user actions

Infrastructure cost and complexity

While per-token costs have dropped dramatically, the infrastructure around context management is expensive:

• Vector databases for semantic search: Pinecone, Weaviate, or AWS OpenSearch
• Real-time data pipelines: Streaming EHR events, claims updates, and external data
• Security and PHI protection: Encrypted data stores, secure enclaves for model inference, BAA-compliant infrastructure
• Integration layers: MCP servers, FHIR APIs, custom connectors to legacy systems

4 practical steps for implementing healthcare LLMs

Healthcare organizations need a structured approach that prioritizes context management. The “Plan, Do, Study, Act” system provides this framework. Watch the following video for an in-depth explainer of this process from Brendan Smith-Elion, or continue reading for a quick overview:

Plan: Design your context architecture

Before selecting models or use cases, design how you’ll manage context. Start with an inventory of your data sources:

• What systems contain patient data? (EHR, claims, labs, pharmacy, ADT, registries, etc.)
• Which sources have APIs? Which ones require custom integration?
• What’s the latency for each source? (real-time vs. batch)
• Where are the gaps in your data?  

Then, map use cases to context requirements:

• Build a grid ranking opportunities by: data availability, integration complexity, latency requirements, and clinical value
• For each opportunity, document: What context is required? Where does it live? How fresh must it be? Who has access rights?

Next, choose your architecture pattern:

• RAG for document-heavy workflows: Prior authorizations, clinical guideline adherence, care plan generation
• Agentic for multi-step processes: Care coordination, patient outreach campaigns, referral management
• Hybrid for complex use cases: Combine RAG for knowledge retrieval with agents for workflow orchestration

Finally, plan for MCP integration:

• Identify which systems will expose MCP servers
• Design your context server architecture (centralized vs. distributed)
• Plan for authentication, rate limiting, and access control

Do: Implement context first

With architecture designed, build context management infrastructure first and models second. Start with context pipelines:

• Implement real-time data streaming from critical systems (ADT, labs, medications)
• Build semantic search over clinical notes and patient history
• Create context assembly logic that gathers relevant information based on use case

Then, experiment with model selection:

• Test multiple models (Claude, ChatGPT, Gemini) with the same context to compare outputs
• Consider fine-tuned medical models (Med-PaLM, BioGPT derivatives) for specific domains
• Use smaller, faster models for high-volume tasks; reserve large models for complex reasoning

Next, implement reinforcement learning from human feedback with context awareness:

• Collect feedback not just on outputs, but on whether the model had sufficient context
• Train models to request additional context when uncertain
• Build feedback loops that improve context retrieval, not just model responses

Finally, use MCP to standardize integrations:

• Implement MCP servers for your key systems
• Leverage existing MCP connectors for common platforms (Epic, Cerner, Athena)
• Build custom MCP servers for legacy systems
     

Study: Evaluate context completeness

Assessment now focuses on context quality as much as output quality. Begin by conducting an expert review with context auditing:

• Show clinical experts both the model output AND the context provided
• Ask: “Did the model have enough information to make this recommendation?”
• Identify systematic gaps in context assembly

Then, measure context metrics:

• Context retrieval latency (are we fast enough for real-time use cases?)
• Context relevance (does semantic search return the right documents?)
• Context completeness (are we missing critical data elements?)
• Context freshness (how old is the data we’re providing?)

Next, test edge cases:

• Complex patients with extensive histories
• Rare conditions where guidelines may be ambiguous
• High-risk scenarios where missing context could harm patients

Finally, monitor for context drift:

• Track whether your data sources maintain consistent structure
• Alert on schema changes in upstream systems
• Audit for data quality issues that degrade context

Act: Operationalize with context governance

Moving to production requires ongoing context management. Start by implementing context monitoring:

• Dashboard showing context assembly success rates by source system
• Alerts for stale data or retrieval failures
• Audit logs showing what context informed each AI decision

Then, build feedback loops:

• Capture when users override AI recommendations (suggests insufficient context)
• Track “dead ends” where workflows don’t complete (context gaps?)
• A/B test different context assembly strategies

Next, establish governance:

• Clinical advisory board reviews model performance monthly
• Data stewards monitor context quality and completeness
• Regular audits of what data is accessed and how it’s used

Finally, plan for regulatory requirements:

• Document your context sources and assembly logic
• Maintain versioned datasets showing what data was available when
• Prepare audit trails for FDA/ONC review

The key to success with healthcare LLMs: Context-aware AI architecture

Healthcare organizations succeeding with LLMs share a common pattern: they’ve invested heavily in context management infrastructure before worrying about model selection or fine-tuning.

Key architectural elements include:

• Unified health data platform: Centralized repository with real-time streaming from source systems
• Vector database for semantic retrieval: Fast, relevant context retrieval from unstructured data
• MCP server layer: Standardized connections between LLMs and healthcare systems
• Orchestration layer: Manages multi-step agent workflows and context assembly
• Observability and governance: Tracks what context is used and how models respond
     

This infrastructure is expensive and complex to build, but it’s table stakes for production LLM applications in healthcare. Organizations trying to skip ahead to model fine-tuning without solving context management consistently struggle with accuracy, auditability, and user trust.

Modern context management in action

With proper context architecture, several use cases have matured significantly:

• Prior authorization automation: RAG systems pull relevant clinical notes, guidelines, and plan criteria; agents orchestrate form completion and submission; success rates now exceed 85% for routine cases.
• Care gap closure: Agents identify gaps, assemble patient context, generate personalized outreach, track responses, and escalate to care managers; operating at scale across entire populations.
• Clinical decision support at the point of care: Real-time context assembly from EHR provides relevant guidelines, drug interactions, and similar patient outcomes delivered at the point of care in seconds.
• Patient engagement: Conversational agents with full patient context handle appointment scheduling, medication questions, and care plan education, maintaining context across multiple interactions over weeks.

The future of LLMs in healthcare

As LLMs continue to evolve, context management will remain the differentiator between experimental pilots and production healthcare applications. The Model Context Protocol represents a critical standardization effort, but healthcare organizations must still solve the hard problems of data integration, real-time assembly, and governance.

The opportunity is massive: LLMs that truly understand patient context can transform care delivery. But the path requires disciplined investment in data infrastructure, not just AI models.

Get in on the ground floor of context-aware healthcare AI. Learn how Arcadia’s data platform provides the unified health data foundation needed for LLM applications.