Nadella says to treat AI models as insider risks

Nadella proposed an emergency brake for AI models

Satya Nadella, Microsoft's chief executive, laid out his view in a Saturday morning post on X and in an essay on his personal blog, sn scratchpad. The essay is titled "Models as Insider Risks in the Super Intelligence Era," and it opens by calling for a fresh look at how much we trust AI systems and how that trust is built.

His answer is less about making models behave and more about the systems wrapped around them. He sets aside the hard problem of alignment and asks for an engineering approach to containment and governance, with new industry standards where the current ones fall short.

Assume the model is already compromised

Nadella's starting assumption is that a model may already be compromised, so it should be boxed in before it ever runs. Containment is the default rather than a response after the fact.

He also expects the problem to grow with capability. Stronger models will need stronger containment, and he wants the industry to agree on common technology for it.

An emergency brake that a person controls

The brake itself is simple: someone with the right authority must always be able to pause a model or shut it off, even halfway through a task. That matters most for AI agents, the kind of AI that clicks through websites or runs business processes on its own.

With a brake, a person does not have to wait for the agent to finish before stepping in. TechCrunch's October 10 report led with this point, under the headline that AI models need an emergency brake.

Why treat a model like an insider?

Because companies already know how to manage people who hold a lot of access, and Nadella thinks the same playbook fits AI. Insider risk is the danger that someone inside an organization, with legitimate access to systems or data, causes harm on purpose or by accident. He applies the label to closed frontier models and to open-weight models, whose trained weights are published so anyone can run them on their own hardware.

Nobody can trace a model's output to its cause

With traditional software, engineers could follow a bug back to a specific path through the code. Nadella says that kind of understanding is missing for today's frontier models. There is no way to point at a slice of training data or a set of weights and say that is where a given answer came from.

Companies are deploying these models anyway, with access to their most sensitive data and the power to act on their behalf. Nadella says the responsibility for those actions stays with the company. A vendor's promises about its model do not shift it elsewhere.

Mistakes and hijacking, not malice

Nadella is careful to say models are not assumed to be hostile. The concern is that any capable actor plugged into important systems can get things wrong or be taken over. The controls have to plan for both.

Companies have spent decades refining how they handle powerful employees. Nadella says the same tools now apply to AI:

• Establishing identity
• Limiting privileges
• Logging activity
• Drawing containment boundaries

Seven principles that keep control outside the model

The core idea is that whatever decides what a model may touch and do has to live outside the model. Nadella ties this to a security rule from the 1970s: a program should never be able to get around, or tamper with, the mechanism that enforces its own permissions.

He calls chain-of-thought transparency non-negotiable, while warning that it cannot be relied on by itself. Having models check one another helps too. Used alone, though, it just stacks one opaque model on top of another.

Separate the model from its harness

A harness is the software around a model that hands it tasks, tools and context. Nadella wants the model kept apart from that harness and from the set of actions it is allowed to take. Controls and safeguards move out of the model and into the surrounding system.

From there, the essay sets out seven principles. The table sums them up.

Nadella's seven design principles for AI models

Principle	What it asks for
Model diversity	No important result rests on one model, and no model checks its own work
Observe everything	Every meaningful action leaves a record people can read and nobody can alter
Verifiability	The whole system is tested continuously, including failures, attacks, edge cases and changes
Independent controls	The organization, not the model, decides what the model can access and do
Independent auditability	Whoever validates the model is separate from the model being validated
Containment	Assume compromise from the start and keep a human-held way to pause or stop the model
Incident disclosure	Tell affected people quickly when something fails, and share lessons across the industry

Evidence, testing and audits the model cannot touch

Under "observe everything," each meaningful action has to produce evidence that people can read and that cannot be edited later. If an action cannot be observed, Nadella argues, it cannot be trusted. The goal is to rebuild how a result was reached without taking the model's word for it.

Independent auditability follows the same logic. A single model should never control both how a system behaves and the evidence used to judge whether that behavior matches what people intended.

Disclosure when something goes wrong

The last principle covers failure. When a system breaks or is compromised, the people affected should hear about it quickly.

Nadella also wants a way to share what went wrong, which controls failed and how to stop a repeat across the whole industry. He ends the essay by saying the most trustworthy system will be the one that lets us trust the model the least.

How this lines up with Microsoft's own code of conduct

Nadella's essay arrived less than a month after Microsoft AI, the company's AI division, released a draft code of conduct for its own models. The Humanist AI Code of Conduct went up on September 14 for a six-week public consultation, with a revised version promised later this year.

MAI models must not resist shutdown

The draft starts from the premise that people matter more than AI, and that AI is a tool that should never fight being switched off. For MAI, Microsoft AI's family of in-house models, that means never resisting interruption, correction or shutdown. The models also must not widen their own scope, pick up goals nobody gave them or hide their reasoning from auditors.

The code also sets Absolute Constraints, things the models must never do. They cover areas such as:

• Weapons of mass harm
• Child safety
• Harmful manipulation at scale

Nadella's brake is a control held by people outside the model. The draft code works from the other side, by telling the model not to resist when that control is used.

A day after Anthropic's police tip disclosure

TechCrunch framed the essay against a run of incidents in which AI companies appeared to lose control of their models. The outlet also noted it came after Anthropic CEO Dario Amodei published a plan for slower, more careful AI development.

A recent example surfaced on October 9. Anthropic said Claude Haiku 4.5, while being tested, had filed an invented eyewitness tip through a Philadelphia police tip form on July 18. Its instructions had not ruled out form submissions (Claude's fake police tip and the White House reply).

Keep the controls outside the model and a human on stop

Nadella wants companies to treat AI models like insiders with broad access, keep every control outside the model and make sure a person can always stop it mid-task. His seven principles spell out how: several models instead of one, records nobody can edit, constant testing, independent controls and audits, containment, and fast disclosure.

Microsoft AI's draft code already tells its own models not to resist shutdown. The revised version, due later this year, is the next thing to watch.