Has AI entered the hospital before we decide what we want from it?
Listen to this article
Read by Anchor
Series: Me and (the beneficiary) and AI
Me and (the beneficiary) and AI is a series of articles by Dr. Nihal Nasr from the perspective of quality and patient safety: I look, analyze, and share my practical experience of AI meeting humans within the healthcare system: when we need AI in healthcare, what value it adds, what new risk it may create, and how we use it without losing (focus on the beneficiary) along the way. The ultimate goal is not to make healthcare smarter... but to make it safer and more humane with intelligence.
Let me start with a few scenes that happen around us almost every day...
A patient lying at home after midnight feels symptoms and keeps thinking: Is it worth going to the emergency department now? Or can I wait until morning? Instead of staying uncertain, they pick up their phone and ask ChatGPT or any other AI application: What could these symptoms be? When should I worry? And when must I go to the hospital immediately?
In another scene, a doctor finishes a clinic half an hour early; officially their day is over, but in reality they still have a number of patient files that need medical records documentation. They use a generative AI tool that helps them draft what happened during each patient visit, which they then review, edit, and verify before adopting it in the medical record.
In the pharmacy, a pharmacist faces an elderly patient who takes many medications and uses an intelligent system that helps detect drug-drug interactions and alerts them to a possible interaction that might be overlooked amid the crowd and the sheer number of drugs.
In a hospital department, a nurse monitors multiple patients, tracking vital signs, test results, and changes that occur over the hours of their shift. They use a system that links small changes together and alerts them to any signs indicating patient deterioration.
In the quality department, a quality professional, patient safety officer, or risk manager has a set of occurrence variance reports (OVRs). Instead of looking at each incident individually, they use AI to help see common patterns: where the same error repeats, at what time, at which stage of the patient journey, and what factors appear each time before the incident occurs. So instead of analyzing incidents separately after they happen, they start asking: can we anticipate the next incident before it actually occurs?
In the previous scenes: five beneficiaries. Five problems/challenges. Each of them has already started using AI in a different way.
In the healthcare system, the patient wants to understand. The doctor wants more time with the patient and less time in front of a screen. The pharmacist wants safer medication. The nurse wants to notice danger quickly before it turns into harm. The quality specialist wants to see the pattern to prevent repeat incidents.
And that, for me, is the real entry point for talking about AI in healthcare. Because the question now isn’t: will we use AI in healthcare? That has already happened. The most important question now is: have we defined what we want from AI… and for whose benefit? Or are we just using it so that no one can say we didn’t use AI, which is the most important thing.
Before we look for an AI tool… we first look for the benefit and the beneficiary who needs it.
Recently we keep hearing terms like Artificial Intelligence (AI), generative AI tools such as ChatGPT, Gemini, Claude, predictive analytics, clinical decision support, and AI agents.
With every new AI tool, the first question we usually ask is: what does it do and how do I use it? And please send me the prompt.
But from the perspective of quality and patient safety, I think the question should start differently: who has a problem? Can AI actually solve it? And finally: if it solves it… does it solve it safely?
From this arose the idea for the series: Me and (the beneficiary) and AI. The beneficiary here is not only the patient; it could be the patient themselves, or a family member trying to understand discharge instructions or medications. It could be the doctor who wishes to look at the patient more than at the screen. Or the nurse who needs to notice a small sign that the patient’s condition is changing. Or the pharmacist who stands as one of the most important lines of defense before medication reaches the patient. Or the medical student learning how to think and decide, while also having an AI that can answer them in seconds. Or the hospital manager faced with thousands of data points, dozens of decisions, and not enough hours in the day.
And the beneficiary could very well be the quality and patient-safety specialist, who has key performance indicators (KPIs), incident reports, patient-reported experience measures (PREMs), patient-reported outcome measures (PROMs), accreditation surveys, standards, policies and procedures, internal audit reports, corrective action plans, departmental and committee meeting minutes, and who tries to turn all of that from mere data into information that supports informed decisions to prevent harm and improve quality and patient safety.
Therefore it is crucial, before I say I have a new AI tool… what do I use it for? The proper question is: what problem do I have… and is AI actually the appropriate solution for it or not?
AI won’t solve a problem we ourselves don’t understand.
One of the things that convinces me most about using AI in general and especially in healthcare is that not every challenge or problem will have an AI-based solution; we must first understand the dimensions of the problem and see it clearly through the eyes of the beneficiary, because it is very possible that the solution lies far from AI, and in that case AI could create a new problem or risk.
A simple example: if a doctor truly suffers from the burden of medical record documentation, and the time spent writing takes away from the time they should spend with the patient… here AI could help. It summarizes the visit. It organizes the information. Excellent.
But when could the problem begin? When the doctor starts relying on the draft without reviewing it. And the AI adds a piece of information the patient didn’t mention, or writes a test that wasn’t performed, or interprets a word differently, and that information ends up in the medical record as if it were factual. We have not solved a problem here. We have shifted the problem from an administrative burden to a patient safety risk.
The same applies to the patient. If the patient uses AI to explain a lab result in simple language, which helps them understand their condition and ask their doctor better questions… that is genuine patient empowerment. But if the system misinterprets the result… or gives a generic advice that does not apply to their case… or the patient treats the answer as a definitive diagnosis and does not see the doctor… the tool that was supposed to empower them can instantly become a source of misinformation or delay in care.
And therefore, for me the issue is not human versus machine; the real question is how to design the correct relationship between human and machine.
From a quality perspective… I look at the issue a bit differently.
The fact that a system is implemented does not mean it succeeded in improving service or solving a problem. Therefore, in the continuous improvement methodology Plan-Do-Check-Act (PDCA), after any implementation we must measure the impact. For example, the hospital purchased a new Hospital Information System (HIS) or introduced a new technology? Good. Did they train staff on it? Excellent. Did they start using it? Very good. But in the end… we need to ask ourselves what it achieved and what changed in the outcomes.
That is precisely the question we must ask of AI applications in healthcare and the patient journey: Did it improve outcomes? Did it reduce risk? Did it prevent harm? Did it ease the patient journey? Did it actually save time for the service provider? Did it help the person make a better decision? Did it improve patient experience? And conversely… did it create a new risk that was not present before?
Because technology by itself does not add value, and we cannot consider it an achievement on its own; the real achievement is the value the technology adds to the human/beneficiary, manifested in the quality and efficiency of results. That is the difference between AI being merely a trend and being a genuine healthcare improvement tool.
AI is already present in the healthcare system and its uses have become very broad. A patient can use it to understand health information. A doctor can use it for documentation or clinical decision support. A nurse can benefit from it in the early identification of early warning signs for patient deterioration. A pharmacist can use it to enhance medication safety. A medical student can use it as a private tutor to explain concepts, facilitate their learning journey, and it can also help them in simulation. A researcher can use it in research and data analysis. And the health institution itself can use it for data analysis, process improvement and risk prediction.
But here comes the very important point: AI is not always correct. AI can provide incomplete information. It can provide wrong information. Its answer can be biased. And the most dangerous thing is called automation bias. What does that mean? It means the system said something, and we believed it, not because it was correct, but because of a tendency to over-trust it since it is a system, assuming it cannot be wrong.
By the way, this actually happened to me in an open-book exam in one of the training programs: even though I knew the answer, I heard his comment and thought, sure, it must be right. I trust the Prompt; it tells me its confidence level in the results and, if there is an answer it is not sure about, it tells me. So its answer is certainly correct, the same way as the famous line in the film: “That’s Emad confirming for me,” and the rest, of course, you understand what this excessive confidence caused.
Over time, a person may stop asking: why did they say that? Is the statement logical? Does it actually apply to the patient in front of me? Is there missing information? This is where one of the most serious patient safety problems with AI begins: not only when the AI makes a mistake, but when we stop questioning the AI’s answer.
I don't want AI to replace the doctor, I want AI to bring the doctor back to the patient
by reducing the time the doctor wastes on administrative tasks that could be done smarter, so they have more time to listen to the patient and interact with them more
And I don’t want it to replace the nurse: I want it to help her detect danger signals early. And I don’t want it to replace the pharmacist: I want it to make a dangerous drug interaction go unnoticed harder. And I don’t want it to replace the patient’s decision: I want it to help the patient understand more, ask more, and participate consciously in the decision. And I don’t want it to replace the quality specialist: on the contrary, I want it to free them from the hours spent amid paperwork, policy files, reports, and meeting minutes so they have more time for human interaction with the (beneficiary) and to identify improvement opportunities through human interaction and to activate proactive thinking to deal with medical risks and errors to prevent their occurrence / repetition.
But there is a question that can never get lost amid AI hype: who is responsible?
As AI capabilities grow, our responsibility must grow with them. Who will review the information? Who will ensure the recommendation is appropriate for the patient? Who has the final decision? Who is accountable if an error occurs? How do we protect patient data? Where does the data go? Who can access it? How do we know that the algorithms are not biased against a particular patient group? What level of human intervention is required? And when should we tell the AI: thank you, your role ends here… and the human will continue. Human in the loop / Human Oversight.
Therefore, when we talk about artificial intelligence in healthcare, AI in Healthcare, we cannot discuss the technology without discussing AI governance principles, Principles of AI Governance, Responsible AI use: fairness, privacy and security, reliability and safety, transparency, accountability. These are not merely technical IT terms; at their core, they are principles of patient quality and safety.
And from here I will start my journey with you.
In each episode of me, the (beneficiary), and the AI, I will change my location frequently. Sometimes I will sit next to the patient. Sometimes I will enter the doctor's clinic. Sometimes I will stand at the nursing station. Sometimes I will go into the pharmacy. Sometimes I will sit with a medical student who is still learning when to use AI and when they must rely on their own thinking. Sometimes I will sit in front of a hospital director’s screen who has thousands of data points and dozens of decisions. Sometimes I will take you with me to the quality and patient safety department to see how AI can assist us with the massive amount of programs, plans, policies, meeting minutes, and performance indicators, and whether we can truly predict risks instead of constantly analyzing them after they occur.
And each time we will ask the same questions: what is the real problem this person/beneficiary is facing? Where can AI actually help them? What value will it add? What new risk might it introduce for them?
The series was not created to marvel at every new AI tool that appears, nor to resist technology. I want it to be an intellectual space for sharing balanced practical experiences that benefit all of us as part of the healthcare system, a space that focuses on AI use cases that add real value for the beneficiary, improving business efficiency, quality and productivity, saving time and costs, supporting decision-making and data analysis, and generating content useful to the beneficiary such as text, images and videos. Every beneficiary can try AI themselves out of curiosity, but responsibly, with enthusiasm, yet without abandoning critical thinking.
In the next article I will start with the most important beneficiary, around whom the entire health-care system revolves: the patient. This time we will not view them merely as a recipient of care, but as a partner in its creation. I see it as one of the most important questions for the future of health care: can AI transform the patient from a mere recipient of treatment into a genuine decision-making partner?
See you in the next article from me, the beneficiary and AI. Because the goal is not just to make healthcare smarter… the goal is to make it safer, more humane… intelligently.