Who Is Liable When Clinical AI Gets It Wrong? – 2026 Legal Guide

The essentials in 30 seconds

Introduction

Clinical AI promises faster diagnoses, personalised treatment plans, and lower costs. Yet every year, misclassifications or algorithmic drift surface, raising the inevitable question: who is liable when clinical AI gets it wrong? The answer is never simple, because responsibility weaves through manufacturers, hospitals, and the clinicians who ultimately press “accept”.

Since 2016, Galeon has been co‑creating an intelligent EHR with front‑line caregivers. Deployed in 19 hospitals—including two university medical centres—Galeon now supports more than 3 million patient records and 10 000+ health professionals. The platform’s Swarm Learning® engine lets hospitals collaboratively improve AI models while keeping data on‑premise, a model that aligns with emerging European data‑sovereignty expectations.

With the EU AI Act entering full application in 2024 and the new MDR (Medical Device Regulation) amendments in 2026, regulators are demanding concrete human oversight and rigorous post‑market surveillance. This article untangles the legal framework, highlights the gray zones, and shows how hospitals can structure their processes to minimise exposure.

“Liability is not transferred by technology – it is reshaped by the contracts, controls and documentation you put in place.” – Legal counsel, European HealthTech Association, 2025.

Who are the parties involved in a clinical AI decision?

Both the AI supplier and the health‑care institution can be held accountable, depending on contractual terms, the level of integration, and the presence of human oversight.

The typical chain includes:

• AI manufacturer / software vendor – designs the algorithm, provides the intended‑use dossier, and runs pre‑market conformity assessments.
• Hospital / health‑care provider – integrates the tool into clinical workflows, configures thresholds, and ensures staff are trained.
• Clinician (physician, radiologist, etc.) – makes the final diagnostic or therapeutic decision based on AI output.

When an error occurs, courts look at who had “effective control” over the decision. If the AI was a “black‑box” with no explainability and the clinician relied blindly, liability may tilt toward the manufacturer for inadequate safeguards. Conversely, if the hospital failed to implement mandatory oversight, it can be deemed negligent.

What does the EU AI Act require for human oversight of medical AI?

The AI Act classifies most AI used for diagnosis, treatment planning, or triage as “high‑risk” and mandates real‑time human supervision.

Key obligations include:

• Transparency: the system must inform the user that the output is AI‑generated and provide understandable explanations.
• Robust risk management: a documented risk‑assessment file that is updated whenever the model is retrained.
• Human‑in‑the‑loop: a qualified professional must verify AI suggestions before patient care actions are taken.
• Post‑market monitoring: continuous collection of performance data, incident reporting, and periodic audits.

The Act does **not** absolve hospitals of responsibility; instead, it formalises the supervisory role that clinicians already perform.

Reference: European Commission, AI Act (Regulation (EU) 2021/0106), EUR‑Lex.

How does the classification of AI as a medical device affect liability?

When an AI tool is classified as a medical device under the EU MDR, the manufacturer must obtain a CE mark and comply with the device’s conformity‑assessment procedures.

Consequences for liability:

• Manufacturer liability: If the device is found non‑conforming, the maker can be subject to product‑liability claims under Directive 85/374/EEC.
• Hospital liability: Even CE‑marked devices require the hospital to verify that the device is used according to its intended purpose and that staff are trained.
• Shared responsibility: Joint‑and‑several liability may arise when the hospital modifies the AI (e.g., custom thresholds) without manufacturer approval.

In practice, the safest route is to keep the AI unchanged from the version that received the CE mark and to document any local customisation.

Source: European Medicines Agency – “Guidance on the clinical evaluation of medical devices”, EMA Guideline.

What documentation must hospitals keep to demonstrate compliance?

Hospitals must maintain a comprehensive “AI‑use dossier” that proves adherence to both the AI Act and MDR requirements.

The dossier typically contains:

• Risk‑assessment report (initial and updated after each model retraining).
• Validation and verification results, including sensitivity, specificity, and calibration metrics.
• Human‑oversight SOPs – step‑by‑step procedures showing when and how clinicians intervene.
• Training records for all users who interact with the AI system.
• Incident‑report logs and corrective‑action plans for any adverse events.
• Data‑governance records, especially when using Swarm Learning® or other federated approaches.

All records must be audit‑ready and stored on an HDS‑certified (Health Data Hosting) platform that aligns with ISO 27001:2022. Failure to produce this documentation on request can result in fines up to €30 million or 6 % of annual turnover under the AI Act.

Reference: ANSSI – “HDS certification in 2026: what hospitals must check before signing”, HDS 2026 guide.

How do validation and monitoring processes change with the AI Act?

Under the AI Act, validation is a continuous activity, not a one‑off checkpoint.

Key shifts include:

• Pre‑deployment validation: Must be performed on a representative local dataset; external validation alone is insufficient.
• Real‑time performance monitoring: Hospitals need dashboards that flag drift, bias, or unexpected error rates.
• Periodic re‑assessment: Every 12 months, or after a major software update, a re‑evaluation must be documented.
• Stakeholder feedback loops: Clinicians should be able to report false positives/negatives directly into the system for model refinement.

Galeon’s smart EHR integrates automated drift detection and audit trails, making it easier for hospitals to stay compliant while preserving data sovereignty.

Read more about the differences between a smart EHR and a traditional one: Smart EHR vs Traditional EHR.

What should CIOs and hospital CEOs focus on when implementing clinical AI?

For CIOs (IT leadership)

Prioritise secure, HDS‑certified hosting, robust API governance, and seamless integration with existing EHR workflows. Ensure that all data used in Swarm Learning® stays on‑premise and that audit logs are immutable.

For CEOs / Medical Directors

Establish clear accountability matrices, allocate budget for ongoing validation, and embed AI‑ethics committees that review risk assessments quarterly.

Limits and challenges to be aware of

• Regulatory interpretation still evolves: National courts may apply the AI Act differently, creating jurisdictional uncertainty.
• Explainability vs performance trade‑off: Highly accurate deep‑learning models often lack transparent reasoning, complicating mandatory human oversight.
• Data quality remains the bottleneck: Even the best Swarm Learning model cannot overcome biased or incomplete local datasets.
• Resource burden on hospitals: Building and maintaining the AI‑use dossier demands dedicated compliance staff and IT tooling.
• Insurance gaps: Not all medical‑malpractice policies cover AI‑related claims; bespoke coverage may be required.

FAQ

Can a hospital delegate all AI decisions to the vendor?
No. The AI Act requires a qualified clinician to verify AI outputs before any clinical action.

Is every radiology AI tool automatically high‑risk?
Only those that influence diagnosis or treatment planning are high‑risk; tools that merely organise images are low‑risk.

What happens if an AI model drifts after deployment?
Hospitals must trigger a post‑market review, suspend the model if safety thresholds are breached, and retrain using validated data.

Do Swarm Learning models need separate CE‑marking?
Each participating hospital must ensure that the federated model complies with the original CE‑marked version; updates are considered a change in the technical file.

How long must incident logs be retained?
At least 10 years, in line with the EU Medical Device Regulation and national health‑data retention rules.

Is there a “no‑fault” shield for AI manufacturers?
No. While product‑liability law provides some defence if the device met all standards, negligence in risk management can still lead to liability.

Can clinicians be personally sued for relying on AI?
Only if they ignored required oversight procedures or acted contrary to documented SOPs.

In summary

Liability when clinical AI gets it wrong is a shared, multi‑layered construct that involves AI manufacturers, hospitals, and the clinicians who ultimately approve a recommendation. The EU AI Act crystallises the need for real‑time human supervision, continuous risk management, and a robust AI‑use dossier—elements that hospitals must embed into their governance framework. Galeon’s intelligent EHR, powered by Swarm Learning®, offers a practical pathway: on‑premise data residency, automated compliance artifacts, and built‑in human‑in‑the‑loop checkpoints that align with both the AI Act and MDR. By coupling rigorous documentation with transparent, federated model improvement, hospitals can mitigate legal exposure while still benefitting from cutting‑edge AI assistance.

Discover how data quality fuels predictive medicine – read our deep‑dive.

Sources

• European Commission, Artificial Intelligence Act (Regulation (EU) 2021/0106) – EUR‑Lex
• European Medicines Agency, Guidance on Clinical Evaluation of Medical Devices – EMA Guideline
• ANSSI, HDS certification in 2026 – HDS 2026 guide
• CNIL, “Artificial Intelligence and health data: regulatory considerations” – CNIL article
• Galeon internal report, “Smart EHR vs Traditional EHR: Why Hospitals Are Switching to AI in 2026” – Smart EHR vs Traditional EHR
• Galeon blog, “Predictive medicine in hospitals – why AI depends first on data quality” – Predictive Medicine & Data Quality

‍