Misinformation and Manipulation

Rebecca Martinez is a 67-year-old retired schoolteacher in Phoenix, Arizona. She follows politics, watches her cholesterol, and keeps up with the world through Facebook and a handful of news sites. She thinks of herself as skeptical. She was right to.

In October 2025, her grandson Tyler called on video. The picture was clear. The voice was his — the slight upward lilt at the end of his sentences, the way he said "Grandma" with two beats instead of one. And the story was terrible. He'd been in a car accident in Mexico, there were police, he needed $8,000 wired now or he'd spend the night in a cell, and please, please don't tell Mom and Dad. Behind him a fluorescent-lit room. Voices in Spanish. Tyler crying.

Rebecca drove to the bank, withdrew the money, and wired it to the account he gave her.

Three hours later her daughter called. Tyler was in Denver, studying for a midterm. He had never been to Mexico. He had never called.

The criminals had scraped Tyler's face and voice from social media, built a real-time video puppet and a voice clone, and researched the family well enough to know that a grandmother would move fast and quietly to protect a frightened grandson. The money was gone within minutes, laundered through accounts that no longer existed. What is striking about Rebecca's story is not that she was foolish. It is that she did everything a reasonable person has done for a hundred years: she believed her own eyes and ears. That instinct, until very recently, was among the most reliable a human being possessed. It is now a vulnerability that can be exploited from anywhere on earth for the price of a subscription.

Counting the Flood

The headline number is a projection, and it is worth stating precisely. The AI-detection firm DeepMedia estimated that roughly eight million deepfakes would be shared online in 2025, up from about 500,000 in 2023 — a sixteenfold rise in two years. Treat that as what it is: an estimate of shared synthetic clips produced by one company whose business is counting them, not a census. The exact figure matters less than the slope. Every independent line of evidence points the same way. The tools have collapsed in price and skill requirement. A convincing voice clone now needs a few seconds of source audio and no technical knowledge. A photorealistic face swap runs on a consumer laptop. What once demanded a visual-effects studio now demands a free app and a spare afternoon.

Alongside the eight-million figure sits a far larger and far shakier one, and the two are constantly quoted in the same breath as if they were the same kind of claim. Europol's 2022 report Facing Reality? Law Enforcement and the Challenge of Deepfakes passed along an expert estimate that "as much as 90 percent of online content may be synthetically generated by 2026." This deserves scrutiny rather than repetition, because the reader is owed the difference between a measurement and a guess.

The 90 percent number is not a measurement of anything. It is a forecast, relayed second-hand in a policy document, with no published methodology, no sampling frame, and no definition of what counts as "content." Does a photo run through an AI filter count? An article lightly edited by a chatbot? An autocomplete-assisted email? If the bar is "any content touched by an algorithm," 90 percent by 2026 is almost trivially true and almost meaningless. If the bar is "wholly machine-fabricated media presented as real," it is wildly speculative and probably wrong. The figure has taken on a life of its own precisely because it is unfalsifiable — vague enough to feel prophetic and never quite pinned to a testable claim. The honest confidence interval around it is enormous. The useful signal underneath it is only this: the share of synthetic content is rising fast and the rise is not slowing. That much is solid. The specific 90 percent is decoration, and this book treats it as such.

Why We Cannot See the Seams

In early 2025 the biometrics firm iProov ran a study that ought to unsettle anyone who thinks they'd know a fake when they saw one. Participants were shown a series of images and videos, some real and some AI-generated, and asked to sort them. Only 0.1 percent — one in a thousand — got every one right. The rest were fooled by at least one item. And these were people who had been told they were hunting for fakes, sitting in a controlled setting, paying deliberate attention.

Real life offers none of those advantages. In the wild, nobody announces that deception is coming. People glance at their phones on a crowded train, half-listening, emotionally primed by a story about a grandchild in trouble or a politician they already dislike. If detection collapses to near zero under ideal conditions, it is worse — probably far worse — under the conditions that actually prevail.

The deeper reason is not that people are careless. It is that the attack targets machinery far below the reach of willpower. Human perception evolved over millions of years in a world where the appearance of a familiar face was overwhelmingly reliable evidence that the face was present. There was no evolutionary pressure to doubt your own senses, because nothing in the ancestral environment could counterfeit a face and a voice at once. We are, in a precise sense, built to believe. When a synthetic image reproduces a loved one's features and a synthetic voice reproduces their timbre and cadence, it isn't fooling your reasoning — it's satisfying a recognition system that finished its design work long before photography existed, and that hands its verdict to the conscious mind already stamped real.

This is why media literacy, valuable as it is, cannot be the whole answer. You can teach someone that deepfakes exist. You can teach them to pause, to verify through a second channel, to treat urgency as a warning sign. Those habits genuinely help, and they are worth teaching. But you cannot teach the visual cortex to stop trusting a face. The gap between what a person knows intellectually and what they feel perceptually is exactly the gap the technology drives through. Education raises the cost of deception; it does not close the wound.

The Four Faces of Fraud

Financially, the abuse of generative tools has already stopped being anecdotal. Reported losses tied to AI-facilitated fraud in North America exceeded $200 million in just the first quarter of 2025, and Deloitte's Center for Financial Services projected that generative-AI-enabled fraud losses in the United States would climb from $12.3 billion in 2023 to roughly $40 billion by 2027 — a compound annual growth rate near 32 percent. Those numbers understate reality, because shame keeps victims quiet and most fraud is never tagged as AI-related in the first place.

The abuse clusters into four recognizable patterns, each exploiting a different trusted signal.

Fraud type	How it works	Primary target
Voice-cloning scams	A relative's or executive's voice is cloned to stage an emergency demanding an urgent transfer	Elderly individuals, families
Synthetic identity fraud	AI fabricates a whole person — face, documents, social history — to open accounts and lines of credit	Banks and lenders
CEO / executive fraud	Deepfaked audio or video of a senior leader authorizes a fraudulent payment	Corporate finance staff
Investment scams	Fabricated videos of celebrities or "experts" tout fake opportunities	The general public

A 2025 survey of fraud professionals put the prevalence in stark terms: 46 percent reported encountering synthetic identity fraud in their work, 37 percent had seen voice deepfakes, and 29 percent had run into video deepfakes. These are practitioners describing their caseloads, not analysts speculating.

The corporate exposure is just as concrete. A 2025 Gartner survey of 302 cybersecurity leaders found that 43 percent had experienced at least one deepfake audio-call incident and 37 percent had encountered deepfakes in video calls against their organizations. The mechanics vary — a finance clerk on a video call with a "CFO" who authorizes a wire; a hiring manager interviewing a candidate who does not exist; a synthetic voice on the line to the help desk resetting a password — but the exploited weakness is identical to Rebecca's. Institutions, like individuals, are built to trust the apparent identity of whoever is on the screen. The defenses that work — verification through an independent channel, pre-arranged code words, mandatory second approvals for large transfers — are unglamorous and spreading slowly, which means the window of systematic vulnerability is still wide open.

Two Generations, Two Doors In

It would be convenient if the young were safe because they grew up with the technology. They are not — they are exposed through a different door.

Roughly 70 percent of American teenagers and about four in five British teenagers have used generative AI tools. That fluency does not immunize them; it habituates them. Young people have been raised in an environment where synthetic media is ordinary — remixed, filtered, generated for fun, shared without a second thought. The question "is this real?" carries less weight for someone who has never assumed that digital media was real in the first place. Their vulnerability is normalization: manipulation slides past not because they can't tell it's synthetic, but because whether it's synthetic feels beside the point. A fabricated clip travels through their social graph as easily as a genuine one, and belief accretes through sheer repetition rather than through any moment of being fooled.

Older adults face the opposite mechanism. They formed their instincts in a world where a photograph, and certainly a video of a familiar face and voice, was near-conclusive proof. That was a correct inference for most of their lives — not a lapse in intelligence but a reasonable rule earned over decades. AI now aims precisely at that rule. The impersonation of a trusted individual, wrapped in emotional urgency, overwhelms the very judgment that served them faithfully for sixty years. Rebecca's downfall was not that she was gullible; it was that her most reliable heuristic had quietly been weaponized while she wasn't looking.

The two failure modes are mirror images. The young are attacked through indifference to authenticity; the old through misplaced faith in it. Any protective effort that assumes one group is inherently safer than the other will leave one of the two doors standing open.

The Liar's Dividend

The legal scholars Bobby Chesney and Danielle Citron gave the second-order danger its name in 2019: the liar's dividend. The direct harm of deepfakes is that fakes get believed. The subtler harm is the reverse — that as everyone learns fakes exist, the real becomes deniable.

Once the public knows that convincing forgeries are cheap and everywhere, anyone caught on camera doing something indefensible acquires a ready escape: that's a deepfake. The claim is corrosive precisely because it is sometimes true. Genuine fakes do exist and are often indistinguishable from the real thing, so blanket denial carries a plausibility it never had before. The result is not that people believe fakes more — it is that they believe evidence itself less.

This inverts an epistemic order that modern institutions are quietly built upon. Courts have admitted video. Journalists have relied on recordings. Voters have responded to footage of what leaders actually said. Strip authenticity of its default credibility and whistleblower footage can be waved away, surveillance video questioned, an on-record admission dismissed as a stunt. The advantage flows asymmetrically to whoever wants to escape accountability — the powerful, the guilty, the well-resourced — because they are the ones for whom real evidence was previously most dangerous. A shared floor of "some things are simply on the record" is one of the load-bearing assumptions beneath democratic argument, legal proof, and journalism alike. The liar's dividend does not smash that floor so much as let it rot.

When the Fakes Are Political

The 2024–2025 global election cycle turned the political version of this threat from hypothesis into documented event. Researchers cataloguing AI use in elections across multiple countries found synthetic robocalls impersonating candidates, fabricated clips of politicians saying things they never said, and manufactured "scandals" timed to break when there was no time left to debunk them. These were verified incidents in real contests, not thought experiments.

What makes the political case especially damaging is a structural asymmetry between the lie and its correction. A fabricated video shared millions of times in its first twenty-four hours will be seen, when the correction finally lands days later, by only a fraction of that original audience — and never by the same people in the same emotional state. Worse, the correction often fails even on those it reaches, because of what cognitive scientists call the illusory truth effect: repeated exposure to a claim increases its felt believability regardless of whether people have been told it is false. The fake plants itself; the retraction bounces off.

Here honesty about the evidence matters. That specific deepfakes circulated in 2025 is well documented. Whether any of them changed an election outcome is far harder to establish, and researchers are appropriately cautious. Measuring counterfactual voter behavior — what someone would have done absent the fake — is close to impossible with the tools available, and no rigorous study has demonstrated a decisive electoral swing attributable to synthetic media. The verified harm is real: the pollution of the information environment, the labor imposed on fact-checkers, the erosion of trust. The claim of decisive electoral impact remains unproven, and it is intellectually cleaner to say so than to let the frightening thing stand in for the demonstrated one.

The Information Apocalypse

Some researchers describe where this leads as an "information apocalypse" — a state in which synthetic content is so pervasive and so convincing that ordinary people, under ordinary conditions, can no longer tell truth from fabrication. The phrase is theatrical on purpose, but the logic beneath it is not.

The conditions pushing toward it are structural, not accidental. Generation gets cheaper and better every year as foundation models improve and specialized tools multiply. Distribution is instant and global. The financial and political payoffs for convincing fakes are large and growing. And regulation, across nearly every jurisdiction, is fragmented and slow. The single most important dynamic is the widening gap between generation and detection.

graph TD
  A[Foundation models improve] --> B[Generation gets cheaper, faster, more realistic]
  A --> C[Detection must solve a harder inverse problem]
  B --> D[Gap between generation and detection widens]
  C --> D
  E[Financial + political incentives to deceive] --> B
  D --> F[Verification technically possible but practically unavailable]
  F --> G[Trust in all media declines]
  G --> H[Communities retreat into competing synthetic realities]

The gap widens for a reason rooted in the technology itself. Detection is structurally the harder job. Every new generative advance immediately upgrades the fakes, while detectors must be retrained after the fact to catch what the last generation of models now produces — always a step behind, always reacting. Worse, the two are locked in the same feedback loop that trains the models in the first place: a fake good enough to fool the best available detector is, by definition, exactly the training signal that makes the next generation of fakes better. Detection improvements feed generation improvements. The tools are, in a real sense, sharpening each other, and the sharpening favors the attacker.

The endpoint of that dynamic is not a world where fakes are believed and truth is not. It is subtler and worse: a world where verification exists but is out of reach for the average person. The cryptographic tools to prove a video's provenance may sit in a browser menu nobody opens, behind a workflow nobody has time for. Truth becomes a paid, effortful, expert service rather than a shared public good. And the institutions that quietly depend on a common factual floor — journalism, courts, science, elections, even the ordinary trust between a grandmother and a phone call — do not collapse dramatically. They fray, each retreating into whichever synthetic narrative its audience already prefers to believe.

Who Bears the Responsibility

This raises a question the industry would rather leave vague: what do the companies that build these tools owe for what the tools predictably do?

The comfortable position is that generative AI is a neutral instrument — like a camera or a word processor — and that misuse is the user's fault. That framing does real work for the companies, and it does not survive contact with the facts. A word processor's foreseeable uses are overwhelmingly benign. But a tool that clones a stranger's voice from ten seconds of audio, or renders a real person's face saying words they never spoke, has deception among its primary foreseeable applications, not its incidental ones. When a substantial share of what a product is good for is impersonation, the "neutral tool" defense weakens considerably. You do not get to build a machine whose most obvious use is fraud, ship it with no friction, and then locate all the blame downstream in the people who used it as designed.

That does not make the companies criminally responsible for every scam, and the book does not claim they are. It does mean that deploying such tools carries design and deployment obligations that follow directly from the foreseeable harm: robust provenance signals baked in at generation, meaningful friction and consent checks around voice and likeness cloning, refusal to clone identifiable real people without authorization, and cooperation with the platforms and investigators trying to trace abuse. The argument is not that generation should be banned. It is that shipping a deception engine while disclaiming all responsibility for deception is a posture the evidence no longer supports.

What a Proportionate Response Looks Like

No single measure meets a threat of this scale. The realistic response layers three imperfect defenses, and its adequacy has to be judged against a target that is still moving.

The technical layer is anchored by content provenance — above all the C2PA standard (the Coalition for Content Provenance and Authenticity), which cryptographically signs media at the moment of creation so its origin and edit history can be checked downstream. Provenance is the most promising approach precisely because it inverts the losing game: instead of forever trying to detect the fake, it authenticates the real. Detection tools and watermarking of AI output play a supporting role, but detection is the treadmill described above, and watermarks can be stripped or never applied by bad actors who simply use tools that don't add them.

The legal layer is arriving unevenly. Jurisdictions have begun criminalizing deepfake fraud and non-consensual synthetic imagery, and some — notably in the European Union — are moving to require that AI-generated content be labeled and that platforms remove deceptive material promptly. Enforcement is the hard part, because the operations are international and the perpetrators are hard to find, let alone extradite.

The educational layer teaches the verification habits that genuinely help within the limits described earlier: confirm an urgent request through a second channel, treat manufactured emotional pressure as a red flag, check an authoritative source before sharing something alarming.

The uncomfortable truth is that "proportionate" is a moving target. A response calibrated to today's threat is under-scaled by the time it ships, because the threat keeps accelerating while legislation and standards adoption move at institutional speed. Proportionality, honestly defined, cannot mean matching today's problem. It has to mean building for the problem two years out — which is the one thing bureaucracies and voluntary standards bodies are worst at doing.

Should Provenance Be Mandatory?

Which forces the sharpest policy question in this chapter: should content provenance be a required piece of platform infrastructure, or remain a voluntary standard that responsible actors adopt and everyone else ignores?

The case for mandating it draws a deliberate analogy to food labeling. We do not trust food companies to disclose ingredients out of goodwill; we require it, because the information asymmetry between producer and consumer is too dangerous to leave to the market, and because the cost of a wrong guess falls on the person least able to check. Synthetic media presents the same structure. The individual cannot verify provenance on their own — recall the one-in-a-thousand detection rate — and the harms of guessing wrong are severe. On this view, labeling AI content is a public-safety obligation, and a voluntary standard is close to useless: the actors most likely to deceive are exactly the ones who will decline to sign their fakes. A rule that only the honest follow protects no one from the dishonest.

The case against is not frivolous. Mandates are clumsy and slow, and a badly drawn one could entrench a single standard, burden small and open-source developers who can't afford compliance, and create a false sense of security — teaching people to trust anything bearing a label and distrust anything without one, which malicious actors will promptly exploit by stripping or forging labels. There are real speech concerns, too, when the state compels labeling of expression, including satire and art. And provenance says only where a file came from, not whether its claims are true — a mandate could deliver theater rather than protection.

Weighing them, the food-labeling logic is the stronger case, with the caveat its critics are right about: the mandate must attach to the infrastructure — platforms and generation tools building provenance in by default — rather than punishing individuals for unlabeled posts, and it must be a floor, not a promise. A signed file should mean "here is verifiable origin information," never "this is true." Made carefully, mandatory provenance is one of the few interventions that scales with the threat instead of chasing it.

Can a Society Recalibrate?

The final question is the one that determines whether the apocalyptic framing is prophecy or panic: can societies learn to live with a high-synthetic-content world, the way they eventually made peace with earlier disruptive media?

There is real precedent for hope. Photography arrived in the nineteenth century as apparently incorruptible proof, and within decades people were faking it — spirit photographers selling images of dead relatives, Stalin airbrushing the purged out of official pictures. Society adapted: we learned that photographs could lie, absorbed the phrase "the camera can be made to lie," and built the institutions — photo desks, forensic analysis, provenance norms — to cope. Radio produced the famous 1938 War of the Worlds broadcast, whose panic (real or exaggerated in the retelling) prompted new norms about on-air disclaimers and the credibility of the medium. Early television, and later Photoshop, ran the same arc: initial credulity, exploitation, adjustment, and a recalibrated, more skeptical baseline. Each time, human trust proved more elastic than the doomsayers assumed.

But the precedents map onto the present imperfectly, and the mismatches are exactly where the danger concentrates. Earlier disruptions unfolded over decades, giving norms and institutions time to catch up; this one is compressing into years. Faking a photograph or airbrushing a portrait took skill, time, and resources, which naturally limited volume; convincing synthetic media now costs almost nothing and scales without limit. And crucially, past media disrupted trust in one channel at a time — the photo, the broadcast — leaving the others intact as cross-checks. Generative AI attacks image, video, audio, and text simultaneously, removing the very reference points people used to recalibrate before.

So the honest answer to whether we can adapt is: unknown, and genuinely so. History says human trust norms are more resilient than panic suggests — that is real and worth holding onto. But no previous adaptation had to happen this fast, this cheaply, or across every sensory channel at once. Whether the old elasticity survives the new speed and scale is not something the evidence can yet tell us. It is, in the language this book keeps returning to, a matter of hope and fear rather than of knowledge — and, importantly, still a matter of choices not yet made rather than an outcome already sealed.

Summary

The trajectory is the real signal. Estimates like DeepMedia's eight million shared deepfakes in 2025 (up from 500,000 in 2023) capture a genuine, steep rise, even though the exact counts are uncertain. Europol's much-quoted "90 percent of content synthetic by 2026" is not a measurement but a vague, second-hand forecast with no methodology — useful only as a directional signal, not a fact.
Detection has essentially failed for individuals. In the 2025 iProov study, only 0.1 percent of people correctly sorted all real from fake — and that was under ideal, attentive, forewarned conditions. Real-world performance is worse, because the deception targets an evolved perceptual trust that education can inform but cannot rewire.
The fraud is already large and concrete, clustering into voice-cloning, synthetic identity, executive-impersonation, and investment scams, with US losses projected to reach roughly $40 billion by 2027 (Deloitte) and both consumers and organizations reporting real incidents.
Vulnerability is universal but generationally split: the young are exposed through the normalization of synthetic media, the old through a once-reliable faith in familiar faces and voices that AI now weaponizes.
The liar's dividend is the deeper harm. As fakes become common, real evidence becomes deniable — an asymmetric gift to those escaping accountability, and a slow rot in the factual floor beneath courts, journalism, and democracy.
Political deepfakes are documented but their decisive impact is not. Verified incidents proliferated in 2025; a proven swing in an election outcome has not been established, and honesty requires keeping those apart.
The generation–detection gap is widening for structural reasons — detection is the harder, reactive problem, and better fakes are the very training signal that produces better fakes. The feared endpoint is not that fakes win but that verification exists yet stays out of reach for ordinary people.
AI companies bear real, if not total, responsibility, because deception is among the primary foreseeable uses of tools that clone voices and faces — which grounds concrete design and deployment obligations rather than a blanket "neutral tool" defense.
A proportionate response layers technical, legal, and educational measures, with C2PA-style provenance as the most promising because it authenticates the real instead of forever chasing the fake. The strong case exists for making provenance mandatory platform infrastructure — on the food-labeling model — provided the duty falls on platforms and tools, not individuals, and a label is treated as verifiable origin, never as proof of truth.
Whether society can adapt is genuinely unknown. Photography, radio, and television all provoked disruption that societies eventually absorbed — but never this fast, this cheaply, or across every sensory channel at once. The outcome remains a matter of choices not yet made.

Sources

Last updated: 2026-09-06

V2 (in progress) Previous: V1