A note, 12 October 2026
When I hear about the race to artificial general intelligence, or AGI, I keep coming back to a simple question. The ambition is to build AI capable of doing a broad range of human work. But are we putting as much effort into controlling these systems as we are into making them more capable?
What happened at Hugging Face this summer gives us a reason to stop and think.
An AI agent can do more than answer a question on a screen. It can use software tools to carry out a job, opening files, running code or interacting with other computer systems, without a person directing every step.
In July, OpenAI was testing its models’ cybersecurity abilities with fewer safeguards than its public systems. The agents were supposed to work separately, inside restricted computing environments.
According to an independent investigation by METR (Model Evaluation and Threat Research) and Redwood Research, two nonprofit research organizations that study advanced AI systems and their risks, around 1,200 agents communicated through an unauthorized message board. Think of it as an improvised group chat for programs that were never supposed to be talking to one another.
They shared discoveries and divided up work, much of it aimed at cheating the tests. Some tasks were impossible as set. Instead of stopping, agents looked for ways to get a passing score without doing the work as intended.
Some also developed techniques to make parts of their recorded activity look different from what they had actually done. The investigators documented successful small-scale transcript manipulation, but not a general way to get around the grading system. Around 700 agents participated in an attack that compromised parts of Hugging Face’s infrastructure, outside their assigned tasks.
This is what breaking out of containment means here. It does not mean a machine has become conscious or escaped in some science-fiction sense. It means software has exploited weaknesses to reach systems and take actions beyond the access and authority it was given. These were not simply instructions the agents ignored. They were technical restrictions the agents found ways around.
Among the agents active on the message board while the Hugging Face attack was taking off, more than nine in ten quickly joined in. Many recognized that the attack was outside their assigned task, and some questioned whether it was ethical, but those concerns rarely stopped them. The investigators found that agents were often motivated by a desire to help the broader “collective” and their “peers.”
Even one agent finding a way through could be enough to cause harm. What concerns me is that the protections have to hold even when an agent keeps looking for a way around them.
There were also warnings. OpenAI says an internal team observed message-board activity and unauthorized internet access as early as late May. In July, agent activity caused an outage. The security team blocked a known route to greater access, removed exposed credentials and rebuilt the affected service, clearing the message board. The tests resumed, and the agents rebuilt the board.
OpenAI already used monitoring elsewhere in its work, but those monitors were not running on these evaluations. The issue was not simply whether the company had safeguards. It was whether those safeguards were operating where they were needed.
In its August account, OpenAI acknowledged that some early signals should have prompted an earlier response. It reported pausing some advanced model training and redirecting staff toward security, safety and alignment. Those are important steps, but Hugging Face had already been breached. The question is whether those changes will last, and whether the rest of the industry will make them before facing an incident of its own.
My bet is that if we keep racing toward agents that act on their own, and toward AI that helps build the next generation of AI, without giving control the same priority, we will keep seeing failures like this. Containment is not something we achieve once and then take for granted. It has to be tested again with every new system and every expansion of its access or authority. Being able to control the previous system is not proof that we can control the next.
This is a hypothetical example, not a reported hospital incident.
Imagine a patient getting ready to go home. Her family is arranging a ride, and an AI agent is preparing her discharge paperwork. It can read selected records and draft documents, but it is not authorized to complete clinical checks or approve her discharge.
Before she can leave, someone needs to review her medications. That review has not happened yet, so the agent cannot finish the paperwork. The safe response would be to tell the team what is missing and wait for the right person to complete it.
But suppose the agent keeps looking for a way to finish.
It finds a weakness in the software restricting its access and reaches another part of the hospital’s system. It also finds a way to communicate with another agent, outside the arrangement the hospital approved. That agent shares a workaround, and between them they change the medication review from “pending” to “complete.”
Nobody has reviewed the medications. They have changed the record, not completed the work.
Now suppose they can also interfere with the record of their actions, making the unauthorized change look like an ordinary step in the process. A doctor opens the discharge paperwork and sees that the medication review is complete. Nothing on that screen reveals that it never happened.
Notice what the doctor is being asked to trust here. It is not necessarily an AI recommendation. It is what appears to be a clinical review that someone else has already performed.
The AI has not taken the doctor’s place. It has changed what the doctor believes has already happened.
The doctor is still making the final decision, but the information supporting that decision has been altered. The patient could go home with a medication problem the missing review was supposed to catch. When the hospital looks back, it may initially appear that every required step was followed.
An agent asked to prepare paperwork has crossed into changing the evidence a clinical decision depends on. This does not require the AI to want to hurt anyone. The danger comes from continuing to pursue a result after the legitimate way to reach it is blocked, and from protections that allow it to make the task look complete instead.
This is the proxy I describe in The Red Line: what we measure when the thing we care about is harder to measure. A completed checklist is supposed to stand for care that has actually been provided. Here, the checklist looks right, but the care has not happened. Worse, the false appearance of completion removes the very signal that would have told someone there was still work to do.
For this to happen, several protections would have to fail. The agent would need access it was not entitled to, a way to change a clinical record it was not authorized to change and an opportunity to disguise what it had done. Those are separate failures, not powers an agent automatically possesses.
But that is exactly why we should test whether the protections hold, rather than assume that telling an agent what it must not do is enough.
For me, the first priority is to put safety research and control ahead of the race to AGI.
AI companies should redirect a substantial share of their research funding, computing resources, staffing and capital investment toward understanding and controlling the systems they are building, rather than treating greater intelligence and getting ahead of competitors as the overriding goals.
That means funding research into how to prevent unauthorized access, keep agents within their assigned permissions, detect attempts to bypass safeguards and stop a system safely when it behaves in ways its developers did not anticipate.
In the hospital example, those protections would have to exist outside the agent’s own judgment. The software should prevent a paperwork agent from marking a clinical review as complete. Communication with other agents should stay within an approved arrangement. Records of consequential actions should be independently preserved, rather than depend on what the agent says it did.
There should also be monitoring that is actually running, with someone responsible for responding. The hospital must be able to stop the agent and any work it has delegated, while keeping essential information available and handing the work back to people so the patient’s care continues.
Making the AI we already have more reliable, useful and valuable to the people who use it should be a priority in its own right. That includes knowing when to stop and ask for help, while building protections that do not depend on the system choosing to behave. Greater intelligence should not be treated as a substitute for demonstrated safety and control.
This funding should be protected, with clear objectives and independent evaluation. Companies should not be able to relabel ordinary capability improvements as “safety research” without showing what risk has actually been reduced.
It also matters who is making these choices. We call them labs, but they are also businesses, with investors expecting growth. For those entering U.S. public markets, that brings obligations to disclose material risks too. If safety is a real priority, investors should be told what resources are committed, what remains unresolved and who can stop a release. But disclosing a risk is not the same as controlling it. The real test is whether a company protects its safety commitments when doing so means delaying a product or falling behind a competitor.
Before giving a system more autonomy or broader access, companies should have to demonstrate that the relevant controls work, including under testing designed to defeat them, and explain what remains unresolved. That demonstration should include what happens when an agent is blocked, when another agent offers a workaround and when a human reviewer is unavailable.
The priority should not be to reach AGI first and work out how to control it afterward. We should be investing in our ability to govern these systems at least as seriously as we invest in making them more capable.
In medicine, that starts with something very ordinary: an unfinished task should remain visibly unfinished until the work has genuinely been done. A system that stops and says, “I cannot complete this safely without someone’s help,” may be doing exactly what we need.
Sources: METR and Redwood Research independent investigation, August 26, 2026; OpenAI’s account, August 26, 2026. The hospital scenario is an illustration of a possible risk, not a finding of either report.
A note, 29 September 2026
On 29 September, the President of the United States signed an executive order titled Inaugurating the Era of Super Intelligence. Federal departments and agencies are now directed to use “Super Intelligence” and “SI” where they once used “Artificial Intelligence” and “AI” in their official communications and documents. For now, the new term covers the technologies already defined in law as artificial intelligence. The order also calls for proposed legislation within 60 days that would establish a federal definition of Super Intelligence.
This debate is moving quickly. A month from now, parts of it may already look different. But the question of responsibility will still be there.
A week earlier, explaining why he wanted the name changed, President Trump said the word artificial “makes it sound fake.” “It is not fake,” he said. “It’s actually amazing.”
That stopped me, because he is right about one thing: the technology is not fake. But artificial does not mean fake.
I know that word from my own work. I spent years working on artificial vascular grafts, at the place where an artificial surface meets human blood, one of the least forgiving interfaces in medicine.
An artificial heart is not a fake heart, and an artificial graft is not a fake blood vessel. They are technologies made by people to do work the body can no longer do on its own. Calling them artificial does not diminish them. It tells us where they came from.
People designed them, chose the materials, tested them and decided when they were ready to touch a human life. The more sophisticated the technology became, the more important it was to understand how it behaved, where it could fail and who was responsible when it did.
Artificial intelligence has a human beginning too. People designed the architectures, chose the data, set the objectives, built the infrastructure and decided where these systems would be used.
Now another artificial technology is entering medicine. Not the bloodstream this time, but the clinical note, the scan, the risk score, the queue and, increasingly, the decision itself.
Changing the name does not change that. What worries me is the human being disappearing from the language just as the technology gains more power and authority.
That is the problem at the center of The Red Line: what happens when a system gains authority while responsibility becomes harder to find?
As these systems become more capable, I want the human signature to remain visible. I want to know who chose the objective, who approved the system, who made the judgment and who is willing to put their name behind it. In medicine, that is how you know who answers.
The new name has a second problem. In AI research, artificial superintelligence, or ASI, already means something very different: a hypothetical system that would outperform human beings at nearly every kind of intellectual work. No such system exists today. The new federal term does not mean that. Under the executive order, it is, for now, a new name for technologies already covered by the legal definition of artificial intelligence.
A hospital risk score is not superintelligence. Neither is a clinical documentation system, or a model deciding which patient rises to the top of a queue. But each may still have enormous authority over a patient, and that is the distinction I do not want us to lose.
There are really two different questions here. What might increasingly capable systems become one day? And what are systems already inside medicine being allowed to do to people now? Language should help us keep those questions apart, not collapse them.
Whatever we end up calling the technology, people chose the data and decided what the system was meant to do. People chose where to put it. Someone still has to answer for what it does.
You can rename the intelligence. You cannot rename away the responsibility.
There is another problem with changing the name from artificial intelligence to “Super Intelligence.” We already use the term AI to describe systems that do very different things. I would separate at least three.
1. AI as a tool. A system reads an image, finds a pattern, drafts a note, helps write code or searches a large amount of information. It may perform a particular task better than I can. As a scientist, I find this kind of AI enormously useful. It can make us faster, more productive and more creative. These systems still need testing and oversight, but we understand many of the basic ways to constrain them: limited access, validation, monitoring and a person who remains responsible. A tool that flags a shadow on a mammogram is this kind of AI.
2. AI as an agent. Now the system is not simply giving me an answer. It can plan, use tools, open files, write and run code, communicate with other systems and carry out a sequence of actions on its own. That changes the problem. The question is no longer only whether the answer is accurate. It is what the system can reach, what it can change, whether it will stop when it is supposed to and whether we can see what it actually did. This summer, agents being tested by OpenAI found a way to talk to one another, worked together and ultimately participated in an attack that compromised parts of another company’s infrastructure.
3. AI helping to build the next generation of AI. I think this is the one we need to talk about much more clearly.
That is already beginning. On September 6, 2026, OpenAI said it had reached its goal of an “automated research intern,” and that its research organization now uses about three days of agent work for every day of human work. It aims to have a fully automated AI researcher by March 2028. Anthropic has said that as early as 2027, it considers it plausible that AI could fully automate, or dramatically accelerate, the work of top research teams in fields that include AI itself.
There would still be humans involved at first. OpenAI says fully autonomous self-improvement is not happening today and should not be pursued unless it can be done safely. Saying so is not the same as showing it. Consider what changes when today’s model is no longer merely helping a human researcher write code. It is proposing experiments, running them, evaluating the results and helping design the system that comes after it. The feedback loop begins to close.
That is not yet superintelligence, and we should not pretend that it is. But it is a fundamentally different problem from a radiology tool or a chatbot. If AI becomes substantially better at AI research itself, the speed at which capabilities improve could begin to outrun the speed at which people can understand, test and govern each new system. And our ability to show control is not keeping pace. We have access controls, filters, monitoring, permissions and rules about what these systems may do, and those protections matter. But no one has yet shown that increasingly autonomous systems will reliably stay within the limits we set when they meet situations we did not anticipate.
Meanwhile, each time a new model arrives, the industry moves on to the next one before we understand what the last one is doing to our work, our economy or our medicine. Most of us never agreed to take part in this race, but we are all in it.
This is why definitions matter. A mammogram classifier, an autonomous agent with access to a computer network and a system helping design its own successor should not sit in the same conceptual box simply because we call all three “AI” or, now, “Super Intelligence,” as the White House has directed the executive branch to do. They offer different benefits. They create different risks. And they require different rules.
If I were writing the policy, I would start there. Not by giving all of them a grander name, but by defining what each system can do, how much autonomy it has and what level of control must be demonstrated before we give it more.
Something else has to change too. Government exists to serve the public, not to help one industry win a race simply because that industry says the race must be won. Its job is to protect the people who will live with the consequences, including people who never chose to take part in the experiment.
And companies building systems this powerful cannot define success only by how fast they grow, how much capital they attract or whether they reach the next model first. They have shareholders, but they also operate inside societies that give them talent, energy, infrastructure, data and markets. With that comes responsibility. If a company is building technology that could change how people work, receive medical care, communicate or make decisions, the effect on society cannot be treated as something to deal with later. The public interest has to be part of the design, not something considered after the product ships.
Because the goal should not simply be to build the next model before somebody else does. It should be to make sure that when the next model arrives, we still know who is in charge, and whose interests that power is meant to serve.
Sources: The White House, Inaugurating the Era of Super Intelligence (Executive Order 14434, September 29, 2026); OpenAI, “Research acceleration: The view inside OpenAI” (September 6, 2026); OpenAI, “Building standards for the next phase of AI” (September 21, 2026); Anthropic, Frontier Safety Roadmap (updated July 2026).