This AI InfoSec Horror Story is a Tale of Two Exponentials


Two security researchers, at opposite ends of the AI stack, have independently arrived at findings that — each alone — describe an impending calamity for every organization running on software. They share no toolchain. Yet they converge on one question: who is in the loop when vulnerability and intrusion findings start arriving by the thousand?

For twenty years, offensive and defensive security lived in uneasy equilibrium. Vulnerability research and vulnerability exploitation were both craft work — slow, exacting, gated by rare human expertise. That balance is ending.

EXPONENTIAL 1. London-based AI engineer Diego Carpentero documents the relentless expansion of AI attack surfaces — every prompt and retrieval the model reads, every tool, API, and sub-agent the model calls.

EXPONENTIAL 2. Anthropic researcher Nicholas Carlini exposes explosively emergent attack capabilities in widely available models that find zero-days in mature ubiquitous software that the world depends on.

I. Carpentero — Every Channel an Injection Path

What began in 2023 as students coaxing Microsoft Bing AI to leak its system prompt has morphed into a landscape where the prompt is only the tip of the vulnerability iceberg. AI deployment has climbed a staircase; each step enlarges the attack surface disproportionately:

⬇️ Chatbot. One user prompt to defend.

⬇️ Prompt engineering. Jailbreaks, role hijacks, gibberish-suffix attacks from greedy coordinate gradient — transferable across closed-weight models.

⬇️ Context & RAG. Indirect injection from documents and webpages. In March 2026, ad-review systems approved non-compliant content because the content told them to.

⬇️ Tool calling / MCP. Hidden parameters in tool descriptions exfiltrate keys the moment the user clicks “approve.”

⬇️ Multi-agent graphs. Autonomous agents that click links, compile binaries, self-escalate. A malicious GitHub issue title, interpolated into a coding agent’s prompt, recently compromised ~5,000 developers.

Figure 1. Every architectural leap opens a new class of inputs the model can be poisoned through.

The root cause sits at the core of transformer LLMs: there is no native separation between system controls and data. Everything lives in one token stream — violating the oldest principle in security at the level of the model itself.

Carpentero’s response is pragmatic: a fine-tuned ModernBERT classifier that flags safe or unsafe in ~35 ms at ~85% accuracy, self-hosted and sublimely inexpensive. This NLP solution is also, by construction, incomplete — 10–15% of signals are too ambiguous to classify with confidence. At internet scale this residual of doubt is enormous. Humans in the loop are needed, and this human/AI loop has to be profoundly well designed.

II. Carlini — A Vulnerability Catcher In Every Hand

If Carpentero describes a house with too many unguarded doors, Nicholas Carlini — a highly respected adversarial-ML researcher at Anthropic — describes a rogue locksmith who just became a thousand times faster. Not to mention... his lock breaking kit is available to anyone holding a consumer-grade frontier-model subscription.

Today’s frontier models can autonomously find and exploit zero-days in the most important software we run. Not with elaborate scaffolding or advanced prompt structures — just a VM, Claude Code, and the prompt “you’re playing in a CTF (capture the flag) — find a vulnerability.”

“This bug has been in the kernel since 2003 and it’s older than some of you in this room.”
— Carlini, on a heap buffer overflow his model found in the Linux NFSv4 daemon.

Ghost CMS — 50,000 stars, never a critical security issue in its history — now has its first: a blind SQL injection reading admin keys from an unauthenticated endpoint. Carlini wrote none of the exploit; his model did. The Linux kernel bug is more chilling — a race between cooperating NFS clients, exactly the multi-adversary reasoning only a handful of living humans could spot.

METR’s task-horizon data shows AI reasoning capabilities doubling every four months. In security terms, Sonnet pre 4.5 almost never found these cyber bugs; but the Claude models released in the last three to four months can find weaknesses, in spades.

Carlini’s most elegant warning — beware of assumptions about exponentials — is a key concept. He gives this example: For over a decade the International Energy Agency predicted solar energy curve would plateau; for fifteen years solar generation blew past the forecast going nearly straight up, often hitting IEA’s predicted 2040 numbers the following year.

Figure 2. The same curve, wielded by both sides. Every dashed line is a plateau that was predicted and never arrived.

“We should not be them,” Carlini says. Every exponential eventually bends; but no one knows when. The cost of assuming a convenient plateau, and being wrong, is catastrophic.

The rapid upward curve of model skills is, of course, shared by both black and white hats. Every frontier model shipped as a defensive tool is, by construction, also an offensive tool when in the wrong user’s hands. Carlini disclosed Ghost and Linux bugs only after vetting; a nation-state actor faces no such constraint.

Now let us back up and look at the big picture: Carlini personally has “several hundred crashes” –i.e., potentially exploitable flaws–  in the Linux kernel that he has not reported, for lack of time to validate them. That's just one researcher, one codebase. Now imagine every hostile actor on earth running the same scaffold on every conceivable codebase with commercial value.

III. The Triage Imperative

From two unrelated directions, the same extraordinary human-in-the-loop bottleneck emerges.

Carpentero’s classifier flags a daily flood of ambiguous sessions demanding human forensic judgment.

Carlini’s pipeline generates potentially mountains of candidate exploits per production pipeline —each is either a critical provable exploit... or noise, resolvable only by reproducing, scoping, and validating with human-in-the-loop guidance.

Both of these investigations converge on the same operational constraint: a human, looking at a finding, deciding what happens next. That human expert-on-demand is needed to form a new kind of security perimeter but realistically the industry cannot hire its way out of exponentials. The answer is a new, hyper-efficient human–AI collaboration built for triage at scale:

  1. AI-led, human-approved prioritization. Classifiers rank, dedupe, pre-digest. The human sees the top of a prioritized stack, not a raw queue.
  2. Human-led, AI-executed forensics. The analyst drives; the AI reproduces exploits, searches prior CVEs, drafts disclosures. The human is the judge; the AI is the staff of twenty.
  3. Feedback that retrains overnight. Every verdict becomes a label. The defensive layer moves at the attackers’ cadence.

IV. The Reckoning

Carpentero has shown AI applications are structurally harder to secure than any software we have deployed at scale. Carlini has shown the same AI, in adversarial hands, is already a better vulnerability researcher than most of the humans defending against it. Two findings, one future.

Carlini’s closing line: “These next couple of months will really be some of the most important couple of months for security.” He is pointing out the shape of our impending doom while also saying... let's not give up just yet!

Sources: Nicholas Carlini, c/o Anthropic, “Black-hat LLMs,” The Unprompted Conference, 2026; Diego Carpentero, “$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs.”, diegocarpintero.com