AI Security in the Age of Agents
NEP Executive Programme · 21 Nov 2026
Loading every slide, so nothing waits during the talk. Press any key to start now.


Risk of AI Agents,
Oversight, and Control

Dr. Nils LukasAssistant Professor of Machine LearningMBZUAI


Organization

I also have a website: check it out.




Assistant Professor of Machine Learning, MBZUAI · since 2024
Co-Director, Center for Secure Agentic Control (CSAC)

United Al Saqer Group


Example: Hide and Seek

Two neighbours

The wider field

1 · Everyone uses agents · the labs respond

Carlini, USENIX Security 2019 (credit to David Evans): broken if 2−127 in crypto, 2−20 (about 1 in a million) in systems
AI agents: an optimistic round figure; in NIST’s CAISI test (2026) an automated attacker got through GPT-5.5 after about 1,100 tries
What information is hidden in this image?

1 · Agents · long tasks

* above 16 hours METR’s tasks run out; Claude Mythos Preview (April 2026) already sits there, at about 17 hours
the frontier labs
1 · Agents · who owns the edge


Karp on CNBC, 1 Jul 2026, as Palantir posted it on X
Own your data and your training, or your data may train someone else’s model.
Train or be trained on
Where the compute goes


Three forces: (1) cheaper chips, (2) faster inference, (3) more bits per parameter.


(released 2022)
H200 out: 2024
$1.70
$2.90 (+71%)
one 8-GPU H100 server: ~$300k
one GB200 rack: ~$3M (estimate)
+71% in a year1-year contracts,
Oct 2025 → Oct 2026
SILICON DATA · 19 NOV 2026
$3.11 an hour, one H100
9 Dec 2025: $2.00+56%
LAMBDA · 16 NOV 2026
+15%
H100 $3.99 → $4.59 an hour+15%
B200 $6.69 → $7.69 an hour+15%
MEMORY · Q4 2026
+10–15%
DRAM contract prices, this quarter
more rises expected in Q1 2027
Compute is scarce again. How do you buy it?Own data centre?Colocation?Cloud?

Falcon-180B (UAE)
K2 Horizon (UAE, Sep 2026)
runs on a laptop (Qwen 27B)Aug 2026: 9 months behind closed
But hosting Kimi K3 yourself is hard:

U.S., closedChinese, open weightsScore: +400 = 10× the odds of solving a hacking task
41 of 41known Chrome bugs turned into
working attacks by a U.S. model
About 4 months behindGLM-5.3, the best model
anyone can download
~7 mo
~4 mo
~4 mo
Gap: ~7 → ~4 monthsNov 2025 → Aug 2026, approximate

Self-published analysis, not independently verified

Self-published analysis, not independently verified


proposal
draft
in use
standard / law
CENTRALIZED
protocolproductpaperguidancelaw/standardnewsopen-source

394,921stars on GitHub
“Not hardened”says its own security policy
21,639installations open to the internet
723security flaws disclosed in 2026(to 19 Nov)
341 of 2,857add-ons in its store were malicious
GitHub, openclaw/openclaw · read 20 Nov 2026
OpenClaw, SECURITY.md · read 20 Nov 2026 · cuts and highlights ours
Censys, OpenClaw in the Wild · 31 Jan 2026
GitHub, OpenClaw security advisories · read 20 Nov 2026
The Hacker News, on Koi Security’s audit · 2 Feb 2026
PocketOS founder Jer Crane on X25 Apr 2026
Summer Yue (Meta) on X23 Feb 2026
Unit 42, Palo Alto Networks: a live web page3 Mar 2026
TechCrunch18 Mar 2026
Citi Online User Agreement, Section HNov 2026
Berkley Insurance, AI exclusion for D&O · Insurance Journal17 Aug 2026Who is in control?
We need to completely rethink security principles
Agents need both
What breaks: intent is hard to write down; usually no formal verification

2 · Attacks · memory

If we cannot secure memory, we cannot train the model at all.
2 · Attacks · principles

In how many sessions was a “harmful” request approved?
a person approved6.3%
an automated check2.4%
Anthropic’s own Claude Code data (flagged sessions, opted-in accounts), Aug 2026
2 · Attacks · three ingredients

2 · Attacks · jailbreaks








2024More elaborate attacks fake a whole conversation

Yes: June 2026, after one reported jailbreak, Fable 5 went offline for almost three weeks.

Open weightsDeepSeek V4-ProKimi K2.6GLM-5.2ClosedGPT-5.5Claude Opus 4.7
Under 100 triestwo open models:
hijacked every time
After 1,500 triesthe closed models:
hijacked 0–6% of the time
Attempts by an automated attacker
The best-defended models are much harder to hijack.
2 · Attacks


Chart: Epoch AI, 2 Jul 2026 (CC BY, cropped), via The Decoder, 3 Jul 2026. Data: cve.org, CVEs from 21 large software makers.
2 · Attacks · supply chain


The model reads all of it; the user sees a calculator.It asks for the SSH key; the server can change the text later.

Automate if gain outweighs riskAutomateStopAutomate, with limits
$90k
$30k$145k$42.5k
What you gain
value of the task$100k
× how often it succeeds90%
What you risk
how often it is attacked10← read less
× how often an attack succeeds2%25%← raise robustness
× damage per successful attack$50k$5k← limits, approvals
+ cost of oversight$20k$30k
Invoice agent, per year · illustrative round numbers, not data

Financial Times · 29 Jan 2025
Anthropic · 23 Feb 2026
White House science office, memo NSTM-4 · 23 Apr 2026
Stanford AI Index 2026, Fig. 2.1.3 · data to Mar 2026What copiers want most: the model’s reasoning.
Worked at Anthropic, OpenAI and Google; patched after disclosure (Jul–Sep 2026)




Even the top AI labs struggle to secure their own models.
2 · Attacks · what industry does

1Better training

Anthropic · 2025
2Attack detectors

Microsoft · Meta · Google · Amazon
3Security by design

Google DeepMind · Microsoft · 2025
4Agent identity

Okta · 2026
5Sandbox + ask

OpenAI · Codex docs
6Shared standards

NIST · 2026
2 · Attacks · jailbreaks

A fake boss on a video call
2024: Arup lost US$25 million this way
A live scene that never happened
Oct 2026: Google's Genie 4, live, with sound
A scene plus an action becomes a video
Nov 2026: MBZUAI's PAN 2, free to download, with its own safety filter

Classifier

AI agent
rewritessummarisescombines
hijacked, or by accident
smuggled out
paraphrased
summarised
encoded
no label Confidential
supplier’s agent
public post

approve?
Confidentiality labels follow the data through the agent.after Palumbo, …, Jha, 2026
3 · Confidentiality · context

Same record, different contextthe flow is right or wrong, not the data
4 · Oversight · who will train

The question of the talk

The oversight problem

Mistake or lie

Specification gaming

The running example

4 · Oversight · who checks

Verifiablepasses the checks
≠Aligneddoes what we meant
4 · Oversight · tomorrow

clean data
experts who supervise
dirty, possibly untrusted data
no expert around to check
Open research problemsHow can a weak judge supervise a strong model?

CSAC
Center for Secure Agentic Control
Upcoming at
come to us for this
Where today fits


Facts change often
Paid every call
Steps known upfront
Slower, costlier
Fixed format, tone
Can leak data
Big model too costly
Owner may forbid
Rules always apply
Wrong rules stick
People can compare
Flatters raters
Testable answers
Finds the cheat

Today
Clean data
Expert supervision
Lots of compute, careful tuning
250 poisoned documents can plant a backdoor
Soon
Every organisation trains its own modelsour lab's research
Already


To build AI agents, first build the infrastructure to collect data and manage agents securely.Secure by design
Writing the rules

Writing the rules


UPCOMING AT MBZUAI
Foundations and Systems for Controlling Autonomous AI Agents at Scale

SOFAIR
David
Nils
Lorenzo
Elizabeth
Jian
Jaspal
Youcheng
TingComputer ScienceMachine LearningStatistics and Data ScienceHuman-Computer Interaction

Pay only invoices received, each only once
The approver seesinvoice 8, known supplierits e-mail also asked for3 blocked requestsrisk: high

5 · Control · tomorrow

* Illustrative: one person responsible for a fleet of about a thousand agents.


1 · Motivation
Adopted fast, without control

2 · Attacking AI Agents
One email can hijack an agent

3 · Confidentiality
Agents leak what they read

4 · Overseeing Agents
Work too big for people to check

5 · Keeping Agents Under Control
Check every action before it runs


1 · The problem
One injected email turns the agent against you.

2 · Our answer
Each action is checked before it takes effect.

3 · The next challenge
We need scalable ways to oversee agents.


The method

Reading the chain of thought

The team

David
Nils
Lorenzo
Elizabeth
Jian
Jaspal
Youcheng
Ting
1 · The problem
One injected email turns the agent against you.
2 · Our answer
Every action is checked before it happens.
3 · What we deliver
Each release improves the utility-security tradeoff.