AI Surgery Incidents: Preventing Algorithm-Driven Operating Room Errors

As hospitals embed AI into pre-op planning, intra-op navigation, and post-op documentation, the incident surface expands far beyond model accuracy. Enterprises already show the pattern: 87% use AI in core operations, yet errors and rework still cost over $67 billion annually. [1] In surgery, similar failures mean preventable harm, not just lost margin.

---

1. Map the New Incident Surface for AI-Assisted Surgery

Surgical AI is a mesh of systems touching:

• Imaging and 3D reconstruction
• EHR data and perioperative checklists
• Robotic consoles and navigation systems
• Operative notes and coding workflows

Incidents often emerge from interactions between these parts, not a single prediction.

⚠️ Risk expansion

LLM-based attacks—data poisoning, adversarial prompts, model inversion—can manipulate or extract sensitive data from assistants that draft notes, summarize histories, or suggest plans. [2] A poisoned pre-op summarizer that downplays anticoagulation history could bias many surgeons toward unsafe choices.

MLOps research shows a single misconfiguration can leak credentials, poison training data, or silently alter deployments. [10] When pre-op risk models, intra-op guidance, and post-op analytics share infrastructure, one flaw can propagate corrupted scores or contours across the perioperative pathway.

📊 Documentation as an incident vector

Clinical evaluation of LLMs for medical summarisation finds hallucinations and unsafe summaries common enough to require safety frameworks and expert review. [11] In surgery, this can mean:

• Mis-summarised contraindications and wrong device selection
• Hallucinated steps in operative notes, distorting medico-legal records
• Omitted complications, undermining quality metrics and audits

“Quiet” failures are equally dangerous. In other industries, LLM agents omit critical details, contradict policies, or answer outside scope without alerts. [12] In surgery, an AI that generates perioperative checklists but sometimes drops antibiotic timing or misstates consent language can break protocol without any security signal.

💡 Key takeaway: AI incidents in surgery are system-level failures across data, pipelines, and documents that invisibly reshape human decisions.

---

2. Architect AI Surgery Systems for Security, Not Just Accuracy

Because incidents arise from the full system, curated accuracy benchmarks are necessary but insufficient. AI security guidance stresses the model is not the security boundary; the entire system—data flows, tools, and integrations—is the attack surface. [5] In the OR, this includes:

• EHR connectors for medications and allergies
• Imaging repositories feeding planning tools
• Robotic and navigation interfaces translating plans into motion
• OR device APIs reporting vitals and device states

Any channel can become a control path for adversaries or accidental overreach.

📊 Agentic AI as a new insider

Studies on agentic AI show over 40% of projects risk cancellation due to unclear value, messy data, and over-privileged access. [3] In hospitals, over-privilege is a safety issue: a scheduling agent that can reorder cases, modify fasting instructions, or place lab orders directly affects patients.

Security research on non-human identities warns machine identities will outnumber humans 80:1, and autonomous agents form a new insider class. [6] Each planning agent, navigation bot, or OR assistant should be treated as a privileged non-human identity, with:

• Strong, individual credentials
• Least-privilege access to data and tools
• Comprehensive audit trails for every decision and action

⚠️ Supply chain and framework risk

Vulnerabilities in open-source AI tools—remote code execution, prompt tampering, access-control flaws—show that “peripheral” monitoring or annotation components can be weaponized. [7] In surgical pipelines, a compromised labeling or prompt-management tool could:

• Corrupt segmentation labels for tumor margins
• Alter intra-op guidance prompts in real time
• Exfiltrate OR video feeds

Framework-level issues such as ChainLeak, enabling cloud key exfiltration and SSRF against AI hosts, show a conversational assistant can become a pivot for cloud takeover if its framework is not patched and isolated. [8]

💡 Key takeaway: Architect surgical AI as a Zero Trust system: treat every agent, connector, and framework as a potential insider, enforcing strict isolation and least privilege from day one.

---

3. Build a Surgical AI Safety Program: Monitoring, Red Teaming, Governance

A secure architecture only works if operated safely. Surgical AI must be run like critical infrastructure, not experimental software.

📊 Adversarial testing tuned to surgical harm

Model safety red teaming shows jailbreak success rates of 80–100% for leading models, and regulators expect documented adversarial testing for high-risk systems. [4] For surgical AI, red teaming should probe:

• Misrouting or mislabeling instruments in robotic workflows
• Incorrect dosage or infusion-rate suggestions during anesthesia
• Misleading consent or discharge instructions for patients

LLM security work shows naive agents can leak data across sessions and be steered into unauthorized tool use via prompt injection. [9] In the OR, that requires:

• Strict session isolation between patients and cases
• Hardened tool whitelists with explicit approval for new integrations
• Routine probe-based tests of assistants before each production release [9]

⚠️ End-to-end monitoring and human control

Secure MLOps research using MITRE ATLAS shows adversaries can target every phase, from data collection to deployment. [10] Surgical incident response playbooks must cover:

• Compromised pre-op datasets (for example, manipulated imaging archives)
• Tampered model artifacts or configurations
• Real-time anomalies in intra-op recommendations

Clinical LLM safety frameworks recommend explicit scoring of hallucination and safety error rates with expert review. [11] In surgery, this means continuous sampling of AI-generated summaries, checklists, and recommendations, with surgeons labeling incidents and driving rapid updates.

Enterprise experience shows AI errors flourish when outputs are trusted without review. [1] Surgical governance should:

• Mandate human verification for all high-stakes outputs
• Restrict full automation until safety KPIs are consistently met

💡 Key takeaway: Treat AI surgery incidents as preventable through continuous red teaming, monitoring, and enforced human oversight.

---

AI will reshape surgery, but the same forces driving AI incidents in enterprise, MLOps, and security research now operate inside the OR, where failures are measured in lives, not dollars. By treating surgical AI as a system, hardening architectures around non-human identities and supply-chain risk, and institutionalizing red teaming and clinical safety evaluation, hospitals can capture algorithmic benefits while keeping surgeons in control.

Hospitals planning or running AI-assisted surgery should establish an AI safety council (surgeons, anesthesiologists, IT security, MLOps), mandate adversarial and hallucination audits before major releases, and require that no AI output can alter a patient’s course of care without explicit, documented human sign-off.