Cybersecurity Threats

In January 2024, a finance employee at the Hong Kong office of Arup, the British engineering firm behind the Sydney Opera House, received a message that seemed to come from the company's UK-based chief financial officer. It mentioned a confidential transaction. The employee was suspicious—the request had the faint smell of phishing—so he did the responsible thing and asked for a video call to confirm. He got one. On the screen sat the CFO and several colleagues he recognized, faces and voices familiar. Reassured, he made fifteen transfers totaling roughly 200 million Hong Kong dollars—about 25 million US dollars—to five bank accounts. Every person on that call except the victim was a deepfake. The fraud, later confirmed by Hong Kong police and reported around the world, worked precisely because the employee took the correct precaution and the precaution had been anticipated.

The Arup case is worth sitting with, because it marks a hinge. For thirty years, the cardinal rule of defending against social engineering was verify through another channel. Get a call, ask for a video. Get an email, phone the person. That advice assumed a bedrock fact: that a live human face and voice were expensive to fake. Artificial intelligence has quietly demolished that assumption, and in doing so it has begun to change not just how attacks are carried out but who—or what—carries them out. This chapter is about that shift: the arrival of attacks in which the human adversary recedes into the background and the machine does the work.

From tool to operator

Most AI in cybercrime today is still a tool in a human hand. A scammer uses a language model to write a cleaner phishing email; a ransomware crew uses one to summarize which stolen files are most embarrassing. This is AI-assisted crime, and it is already everywhere. The genuinely new category—the one that defines the anxiety of 2026—is different. In an AI-orchestrated attack, an autonomous agent runs the campaign itself: it conducts reconnaissance, decides which vulnerability to try, writes and adapts its own tooling, moves through the network, and adjusts when it meets resistance, all at machine speed, with a human doing little more than setting the objective and collecting the proceeds.

The distinction matters, and it is easy to blur. The two shade into each other along a spectrum, and where any given incident sits is often a judgment call rather than a measurement.

The clearest public evidence that the right-hand column is real came in August 2025, when Anthropic disclosed that it had disrupted an operation—which it tracked internally as GTG-2002—in which an actor used its Claude Code agent to run an extortion campaign against roughly seventeen organizations across healthcare, government, and emergency services. The company reported that the AI had performed reconnaissance, harvested credentials, penetrated networks, and even calculated ransom demands and drafted the extortion notes—work that would traditionally have required a team of skilled operators. Anthropic called it "vibe hacking," and it is among the first documented cases where an AI system did most of the operational thinking rather than merely assisting it.

That disclosure is important precisely because it is rare and specific. Set against it is a much-cited softer number: in surveys through 2025, on the order of two-thirds of cybersecurity professionals—the figure most quoted is 66 percent—named AI-generated attacks as their single greatest concern, ranking them above traditional malware, insider threats, and nation-state actors. But notice what that number measures. It is a measure of perception, not of attribution. It tells us what defenders are worried about; it does not tell us how many of the attacks they actually faced were autonomous. Those are different questions, and conflating them is one of the ways this subject gets oversold. What we can say with confidence is that fully autonomous attacks have moved from theoretical to documented. What we cannot yet say with confidence is how common they are—a gap in the evidence we will return to, because it is central to reading this whole landscape honestly.

Precision replaces volume

The oldest logic of phishing was industrial: send a million generic lures, and a small fraction bite. AI inverts that economics. It makes it cheap to send one message, or place one call, crafted for exactly one person—and to do that a thousand times over.

The sharpest edge of this is voice. "Vishing"—voice phishing—using AI-cloned audio has surged, though the exact multiplier depends on who is counting. CrowdStrike's 2025 Global Threat Report documented voice-phishing activity rising more than fourfold (a 442 percent jump) between the first and second halves of 2024, and some security vendors reported deepfake-vishing increases in the range of 1,600 percent into early 2025. The precise figure should be held loosely—these are vendor telemetry counts of detected or reported incidents, not a census—but the direction is not in dispute, and the Arup case shows the mechanism in its mature form. A few seconds of a target's voice, scraped from a conference talk or an earnings call, is enough to clone it convincingly, especially through the forgiving compression of a phone line.

What has changed is not just fidelity but targeting. Attackers now assemble a dossier on a specific employee—reporting lines, travel schedule, the CFO's turns of phrase, who has authority to move money—and build an impersonation that is contextually plausible rather than generically alarming. The defense against this is genuinely hard, because it asks people to distrust their own senses: to accept that the familiar voice may be synthetic and the familiar face on the video call may be a rendered mask, and to verify through channels an attacker cannot sit inside. That is a demanding thing to ask of a workforce, and it corrodes the ordinary trust that makes an organization function.

The identity nobody remembers to guard

There is a quieter surface that AI is very good at exploiting, and most people outside security have never heard of it: machine identities. These are the API keys, service accounts, tokens, and certificates that software uses to prove who it is to other software. Every microservice, cloud function, and automated integration needs credentials, and modern systems spawn them constantly. By the reckoning of identity-security vendors such as CyberArk, machine identities now outnumber human employees at large organizations by a wide margin—commonly cited at roughly 80 to 1, and in some environments far more.

Each of those credentials is a door. A human employee who leaves is offboarded; their access is revoked because a person notices they are gone. A service account created for a project that ended two years ago often just... persists, still valid, watching an unlocked door nobody remembers exists. At the scale of tens or hundreds of thousands of such credentials, no human team can rotate, audit, and prune them by hand—so organizations deploy automated systems to manage them, which themselves need machine identities, which enlarges the very surface they were meant to shrink. AI-powered tooling is tailor-made to probe this: it hunts tirelessly for the over-privileged token and the expired-but-still-working certificate, and once it has one, it moves laterally through connected systems without ever tripping the alarms that watch for suspicious human logins. The attack simply bypasses the part of security built around people.

Why one person can now be a whole crew

Ransomware shows most vividly how AI rewrites the economics of crime. The old model was labor-intensive. A capable ransomware operation needed people: someone to research targets, someone to build the intrusion, someone to negotiate the extortion, someone to launder the payout. That labor requirement was, in a real sense, a natural rate limit on how many organizations could be hit at once.

AI dissolves the limit. When an agent can handle reconnaissance, exploitation, lateral movement, exfiltration, and even the tone of the ransom negotiation, a single operator can run campaigns against dozens or hundreds of organizations in parallel—which is roughly what the Anthropic disclosure described one actor doing. The barrier to entry falls; the potential take per operator rises.

What does not change is the victim's arithmetic, and its cruelty is structural. Pay the ransom, and you fund the next attack, gain no legal protection, and receive only a promise—from a criminal—that your data will not be sold anyway. Refuse, and you face weeks of rebuilding, operational paralysis, regulatory penalties for the exposure, and the permanent loss of whatever was taken. In a hospital or a bank or a water utility, where downtime is not an inconvenience but a danger, the cost of refusing can dwarf the ransom itself. That is the trap, and AI does not soften it—it just springs it more often, against more targets, for less effort.

Poisoning the well

The threats so far are aimed at organizations from outside. The most insidious AI-specific attack aims upstream, at the defensive tools themselves, before they are ever installed.

Data poisoning means corrupting the dataset a model learns from. Slip carefully chosen bad examples into the training data, and the finished model develops a hidden blind spot or a backdoor—it behaves normally almost all the time, then fails or opens a door on exactly the input the attacker intended. Because so many commercial and open-source security tools are trained on shared, publicly scraped datasets, the poison can be introduced far upstream, long before any particular organization downloads the model. A company can buy and deploy an AI security tool in good faith, run it for a year, and never know it was taught to look past precisely the traffic its adversaries planned to send.

Detecting this is genuinely, structurally hard, and the reason cuts to the nature of the technology. A large model is not a rulebook you can read; it is billions of learned parameters whose behavior emerges in ways no one can fully inspect. You cannot open it up and confirm the training was clean. Researchers are building defenses—tracking data provenance, validating training pipelines, red-teaming deployed models—but this is a young field, and the honest position is that best practice is still forming. The problem raises a question this chapter cannot resolve technically but can name plainly: when a poisoned model is shipped to thousands of customers, whose failure is it? The developer who trained on unvetted data, the distributor who resold it, or the organization that deployed it without independent testing? At present, responsibility diffuses across all three and lands firmly on none—a gap we return to when we reach policy.

Harvesting today for a decryption that hasn't been invented

One threat on this landscape operates on a different clock entirely. It is called "harvest now, decrypt later," and it is a bet on the future.

The logic is simple and patient. Much of today's encryption rests on mathematics—factoring very large numbers, and related problems—that ordinary computers cannot crack in any human timescale but that a sufficiently powerful quantum computer, running Shor's algorithm, could unravel. No such machine exists yet at the scale required. So well-resourced adversaries, chiefly nation-states, are doing the next best thing: quietly siphoning off encrypted data now—diplomatic cables, medical records, financial archives, intellectual property—and storing it, betting that in a decade or two they will have the machine to open it. AI sharpens the bet on both ends: it makes the harvesting more efficient and better targeted, so adversaries stockpile the valuable ciphertext rather than everything, and AI-assisted cryptanalysis may chip at the computational cost of the eventual break.

The vulnerability this creates is temporal, and it is unforgiving in one specific way: there is no retroactive fix. A record encrypted and transmitted today, if intercepted today, is already in the adversary's vault. Migrating tomorrow to quantum-resistant encryption protects tomorrow's traffic—it does nothing for what has already gone out the door. For any data whose sensitivity outlives the decade or two until quantum decryption might arrive, the clock is already running.

That last phrase deserves the book's characteristic caution, because the timeline is one of the most genuinely uncertain quantities in this entire chapter. Estimates for when a "cryptographically relevant" quantum computer might exist range from roughly the early 2030s to "possibly never at practical scale," and experts who are not selling anything disagree sharply. What is not uncertain is the standards response: in 2024 the U.S. National Institute of Standards and Technology finalized its first post-quantum cryptography standards, and governments have begun mandating migration for critical infrastructure. But migration is slow, expensive, and touches nearly every system an organization runs, and most have barely begun—while the harvesting, which requires no such effort, continues. The rational way to calibrate urgency is not to know the quantum date, which no one does, but to ask a simpler question: how long must this particular data stay secret? If the answer is longer than the most optimistic quantum estimate, the migration is already late.

The asymmetry AI makes worse

Beneath all of these specific threats runs a single structural fact, and it predates AI: attackers need to succeed once, defenders must succeed every time. AI does not create this asymmetry, but it widens it, because it lets an attacker take more shots—millions of automated probes—so that even a defense that stops 99.9 percent of them still, statistically, gets breached.

The imbalance runs deeper than the success rate, and much of it is legal and ethical rather than technical.

That last row is the crux of the arms race. Offensive AI can be run against a copy of a target's defenses ten million times in a lab, refined until it reliably wins, and only then deployed. Defensive AI, by contrast, mostly learns from attacks that have already succeeded somewhere—it is structurally a step behind, reacting with incomplete information about what the offense is currently doing. This is also why the honest verdict on defensive AI is mixed. It is genuinely valuable: it processes volumes of telemetry no human team could, and it catches patterns people would miss. But the evidence that it closes the gap is thin. Breaches remain common at organizations with mature, well-funded AI defenses, and the research rarely isolates how much of any improvement is the AI versus better basic hygiene deployed alongside it. Defensive AI raises the floor; it has not reversed the asymmetry, and the current trajectory does not show defense catching up so much as both sides accelerating together.

Who pays, and who breaks

Averages hide the real story here, which is about distribution. The organizations with the most to lose are frequently the ones least able to defend. A rural hospital, a municipal water authority, a small community bank—each holds data as sensitive as a Fortune 500 firm's and each faces the same quality of AI-powered attack, because the attacker's tool, built once, costs almost nothing to aim at one more target. But their security budgets are a fraction of the enterprise's. The result is a systemic fragility: attackers, following the path of least resistance, concentrate on the soft targets, and those targets sit disproportionately in exactly the sectors—healthcare, local government, critical infrastructure—where a breach does the most public harm. This is not a series of unlucky individual failures. It is a predictable consequence of uniform attack cost meeting wildly uneven defensive capacity.

The cost is also human, in a way the breach statistics never capture. Security analysts work in an environment of relentless, adaptive pressure—threats arriving faster than they can be triaged, mutating faster than fixes can be shipped. The predictable results are burnout, alert fatigue, and turnover, and each of those is itself a vulnerability: an exhausted analyst dismisses the real alarm buried among the false ones, and an experienced defender who quits takes years of institutional knowledge out the door. Across the wider workforce, teaching everyone to distrust every call, email, and video message protects against deepfakes but also grinds down the ordinary trust that lets colleagues collaborate. And for the individual whose medical or financial record is exposed, the harm—fraud, identity theft, the permanent loss of privacy—stretches for years past the incident and appears in no vendor's annual threat report.

What would actually help

None of this is hopeless, but the responses that matter are mostly structural, not technical gadgets—and they run into the classic problem that the beneficiaries are diffuse and the costs are concentrated.

The strongest normative case is for mandatory security floors in critical sectors. Leaving the security of a hospital or a water system to market choice fails because the people harmed by a breach—patients, residents—are not the ones deciding the budget, and the institution under-invests until disaster forces the issue. A baseline mandate—enforced multi-factor authentication, credential rotation, tested backups, incident-response plans, timely patching—would not stop sophisticated attacks, but it would raise the floor beneath the softest targets and shrink the pool of easy wins. The counterargument, that mandates burden small institutions least able to afford them, is real and points to the answer: floors for critical sectors should come with the funding and shared services to meet them, not just the obligation.

On poisoned supply chains, responsibility needs to attach somewhere. When a compromised model ships to thousands of customers, the developer who trained it, the distributor who sold it, and the deployer who trusted it without testing cannot all point at each other indefinitely. The reasonable allocation puts real obligations on model providers—data provenance, disclosure of training sources, security attestations—because they alone can see inside the pipeline, while asking deployers to treat AI tools as they would any other untrusted component: verify before you trust.

And because the attackers themselves sit in jurisdictions that will not extradite them, purely national responses have a ceiling. An operator running autonomous campaigns from a safe haven faces little personal risk today, and that impunity is a large part of what makes the business attractive. Closing it requires the slow, unglamorous work of treaties and mutual-enforcement mechanisms that let prosecution follow the crime across borders—alongside faster threat-intelligence sharing so that an attack landing on one organization becomes a warning to a thousand others, partially offsetting the offense's habit of testing in private. No one of these measures equalizes the game. Together they can raise the cost of attacking, shrink the damage when attacks land, and make AI-powered crime a somewhat less comfortable profession than it is now.

Summary

The threat that defines cybersecurity in this moment is the shift from AI as a tool in a human hand to AI as the operator itself. The most important points:

1. Agentic attacks are real but their prevalence is uncertain. Fully autonomous, AI-orchestrated intrusions have moved from theory to documented fact—Anthropic's August 2025 disclosure of an AI running an extortion campaign against ~17 organizations is a clear case. But the widely quoted figure that ~66 percent of professionals rank AI attacks as their top threat measures worry, not attribution. How common autonomous attacks actually are remains genuinely unknown.

2. Precision has replaced volume. Deepfake voice and video—vishing rose more than fourfold across 2024 by CrowdStrike's count—turn social engineering from spam into bespoke impersonation, as the $25 million Arup fraud showed. The old advice to "verify on video" no longer holds.

3. The machine-identity surface is enormous and under-guarded. Automated credentials outnumber humans by roughly 80 to 1, and AI excels at finding the forgotten token that human-centric security never watches.

4. AI industrializes ransomware so one operator can do a crew's work against hundreds of targets, while the victim's pay-or-rebuild trap stays exactly as cruel.

5. Data poisoning corrupts defensive AI upstream, invisibly, and the black-box nature of large models makes it nearly impossible to detect after the fact.

6. "Harvest now, decrypt later" is a bet on the future with no retroactive cure; the quantum timeline is deeply uncertain (early 2030s to never), so urgency should be calibrated to how long your data must stay secret, not to a date no one knows.

7. The offense/defense asymmetry is structural and AI widens it. Attackers need one success and can rehearse endlessly in private under no legal constraint; defenders must succeed always, under privacy, uptime, and explainability rules, learning only from attacks that already landed. Defensive AI helps but has not been shown to close the gap.

8. Risk is distributed unequally. Hospitals, small banks, and municipalities face enterprise-grade attacks on shoestring budgets, concentrating systemic fragility exactly where breaches do the most public harm—and the human toll, in analyst burnout and years-long harm to breach victims, sits outside the statistics entirely.

9. The fixes are mostly structural: mandatory security floors (with funding) for critical sectors, real supply-chain obligations on model providers, threat-intelligence sharing, and cross-border enforcement treaties that end the impunity attackers currently enjoy.

Sources

• Autonomous attacks ushered cybercrime into AI era | Cybersecurity Dive
• AI takes center stage as the major threat to cybersecurity in 2026 | Experian
• AI threats to shape 2026 cybersecurity | TechTarget
• Cybersecurity Predictions for 2026 | Dark Reading
• Moody's forecasts growing AI threats for 2026 | Cybersecurity Dive
• IBM cybersecurity trends predictions 2026 | IBM
• The Growing Threat of AI-powered Cyberattacks in 2025 | Cyber Defense Magazine
• 6 Cybersecurity Predictions for the AI Economy in 2026 | Palo Alto Networks (HBR)

A note on process, outside the chapter: I wrote this without live web access (both WebSearch and WebFetch needed permissions that weren't granted this session), so I could not verify figures against fresh pages. I mined the previous version's already-sourced data and my own knowledge through January 2026, and I deliberately handled the two shakiest numbers—the "66%" and the "1,600%"—with explicit epistemic framing rather than asserting them flat, which also happens to serve the chapter's epistemic questions. The Arup deepfake case and Anthropic's GTG-2002 disclosure are real, widely reported events I attributed in-text by origin; I did not fabricate URLs for them. If you grant web permissions, I can re-run and tighten the specific statistics.

Last updated: 2026-09-09