The Red Line

What AI Must Never Do in Medicine

What happens when a system meant to help with a medical decision quietly starts to shape the decision itself, and it becomes harder to know who is responsible?

“Dr. Khalpey writes with the precision of a physicist and the conscience of a clinician. She names the hard truths that many in our field are still trying to avoid: that systems entering our hospitals must be made legible, auditable, and answerable before they ever touch a patient.”

Professor Andrew A. Gumbs MD, MSc, MBA, PhD(c), FACS
Editor-in-Chief, Artificial Intelligence Surgery
President, Artificial Intelligence Organization for the Next Generation of Surgery
Director of Artificial Intelligence Surgery
Hôpital Antoine Béclère, Paris

PaperbackKindle

Coming soon

What if AI is better?

Maybe it will be. On some medical tasks, AI has already outperformed doctors in studies, and we should take that progress seriously.

We are constantly being shown scores that tell us an AI system performed better than doctors on a test. Those results matter. They tell us something important about what the technology can do. They do not tell us everything.

A benchmark is simply a test used to compare performance. It measures medicine as it is graded, not medicine as it is lived.

A system may identify a disease more accurately than a doctor, or predict the risks of a treatment more precisely. That still does not tell us how much authority it should have over a person’s care, or who answers when things go wrong. And knowing the risks more precisely does not decide which risks a patient is willing to live with.

That is a different kind of judgment.

Accuracy should make us pay attention. It should not, by itself, give a system the authority to decide what happens to a patient.

What does “better” actually mean? →

What do we mean when we say AI?

AI has become an umbrella term. We use the same two letters for systems with very different abilities, levels of autonomy and access to the world around them.

That matters because the questions we should ask change as the system changes.

AI as a tool

Gives you something to examine.

An AI system can mark a region on a brain scan for a radiologist to review. It does not make the decision. It shows where to look. The judgment is still the radiologist’s.

Illustrative axial brain MRI shown twice. Both images contain the same faint shadow in the white matter beside the left lateral ventricle. In the second image, it is outlined in red for review. — Figure 1. Illustrative axial T1-weighted brain MRI. A faint shadow in the white matter beside the left lateral ventricle appears in both images and is outlined in red in the second. The patient’s left is shown on the right side of the image.

AI as an agent

Can take actions across many steps.

A hospital agent might move through several systems to complete a task.

Reads records Fills in paperwork Books appointments Sends messages

AI helping build more capable AI

Can contribute to experiments and model development.

It can help shape the systems that come after it.

Designs experiments Tests code Shapes the next system

They may all be called AI. They do not have the same abilities, create the same risks or require the same controls.

These are not three official categories of artificial intelligence. They are a useful way to think about responsibility.

The more a system can do on its own, and the more it can reach, the stronger the controls around it should be.

Why these distinctions matter

The language around AI is changing quickly. On September 29, 2026, the White House directed the executive branch to use “Super Intelligence” and “SI” in place of “Artificial Intelligence” and “AI” in official communications.

But a medical-imaging classifier, an autonomous agent and a system helping researchers build the next generation of AI are not the same thing.

If we use one name for systems with very different capabilities, autonomy and access, we risk losing the distinctions that should determine how they are governed.

Read why definitions matter →

YOUR CHILD DID NOT QUALIFY.

That is the sentence Maya sees on her phone while her eight-year-old daughter sits beside her.

An AI system has reviewed the case. It has looked not only at the child, but at Maya too: years of unexplained symptoms, repeated normal tests, a diagnosis that came late, her treatment and its cost. Not as evidence of what medicine missed. As evidence that the case is complicated and may cost too much.

The therapy that might prevent the same cardiac condition from taking hold in her daughter will not be covered.

Maya’s worst fear was never that her daughter would inherit her genes. It was that she would inherit her data.

That is where The Red Line begins.

Through composite clinical stories and documented research, the book follows what happens when systems built to assist begin to shape decisions about real people: an AI-written note changes the meaning of a patient’s chest pain without inventing a single fact, and an algorithm sees a woman’s price instead of her illness.

The clinical scenes on this site and in the book are composite or illustrative narratives. Empirical claims and quantitative findings are drawn from documented research.

Why this site exists

The Red Line took four years to write. Two completed manuscripts came before it, and I chose not to publish either of them.

I began by asking what AI could do for medicine and how it might be used safely. Eventually, I understood that those questions were not enough.

I had to follow the technology beyond the bedside: to the data, the human labor, the jobs it may change or displace, the electricity and water that sustain it, the communities that carry those costs, and ultimately to the question of who is responsible for what these systems do.

I came to understand that something can be technically accurate and still be ethically incomplete.

The book is finished, but those questions are still open.

This site is where I want to keep following them. I want to look closely at the AI systems already being used in healthcare, what they actually mean at the bedside today, how they are changing the work of clinicians and the experience of patients, and where the choices we make now may lead medicine five or ten years from now.

And I want to make what I learn understandable to anyone who wants to know what is changing, what it means, and what questions we should be asking.

Where do you want to go next?

UNDERSTAND

What is an AI agent? What is a benchmark? What do AGI and superintelligence actually mean?

Plain-English explanations of the ideas and technologies changing medicine.

Learn →

APPLY

What should hospitals ask before deploying consequential AI? What can patients and families ask when AI shapes care, coverage or access?

Beyond the book →

FOLLOW

What is AI already doing in healthcare, and what do new developments mean at the bedside? The latest note, Safety before the race, looks at the Hugging Face incident and asks what happens when an AI agent reaches beyond the access it was given, coordinates with other agents, or changes the information a human decision depends on.

Read the latest note →

“Amina Khalpey brings both the perspective of someone who has worked at the edge of what is technically possible and the human dimension that data alone will never capture. She has written something rare: a book about artificial intelligence that never loses sight of the patient.”

Michael A. Caligiuri, MD
Former President and Physician-in-Chief
City of Hope National Medical Center