Authoritarian AI Governance

In 2018, a story swept through Western newspapers, TED talks, and dinner-party conversations. China, it was said, had built a single all-seeing machine that assigned every citizen a number. Jaywalk, and the number fell. Buy too many video games, and it fell further. Befriend the wrong people, and their low numbers dragged yours down with them. The number decided whether you could board a train, rent an apartment, or send your child to a good school. It was Orwell rendered in code — a nation of 1.4 billion people ranked, sorted, and disciplined by an algorithm.

The story was gripping. It was also, in almost every specific, wrong.

When researchers who actually read the Chinese-language policy documents — Jeremy Daum at Yale's Paul Tsai China Center, Rogier Creemers at Leiden, the analysts at the Berlin think tank MERICS — went looking for the unified citizen score, they could not find it. What they found instead was a sprawl of overlapping, often incompatible programs: court blacklists of people who had defaulted on judgments, municipal pilot schemes in a few dozen cities, industry compliance databases aimed mostly at companies rather than people, and a handful of commercial products like Ant Financial's Sesame Credit that were voluntary, financial, and never actually folded into any government system. There was no national score. There was no master algorithm. The most famous municipal experiment, in the small city of Rongcheng, ran partly on retirees with clipboards writing down whether residents had swept their doorsteps.

This gap — between the myth of a finished digital panopticon and the messier, more fragmented reality — is the right place to begin. Because the myth, it turns out, describes the architecture rather well. It is the blueprint. And the blueprint matters, because pieces of it are real, more pieces are being built, and its underlying logic is spreading to countries that have far fewer institutional brakes than China itself.

The Five-Part Blueprint

The design that most people mean when they say "China's Social Credit System" rests on five components. None is science fiction; each exists somewhere in partial form. Their power lies less in any one part than in how they are meant to reinforce one another.

The first is comprehensive surveillance data integration — the ambition to pool everything into one view. Facial-recognition cameras, internet activity, financial transactions, transit records, workplace logs, health data: individually unremarkable, collectively a near-complete portrait of a life. The technical promise of AI here is fusion. A camera that recognizes a face is a tool; a system that ties that face to a bank account, a chat log, and a location history is a form of governance.

The second is AI-generated behavioral scoring — reducing that portrait to a signal that predicts conformity. The third is automated enforcement: consequences applied by software rather than by an official, so that a penalty follows a violation in seconds and at population scale. The fourth is the social-network effect — making a person's standing depend partly on the standing of their associates, so that people police one another and the compliant cluster together while the flagged are quietly shed. The fifth is gamification: visible rankings, points, notifications, and status tiers borrowed from behavioral psychology, so that citizens come to pursue a good score the way they pursue any game score — as a goal they have internalized rather than one imposed on them.

The interaction is the whole point. Integration feeds the score; the score triggers automated enforcement; enforcement plus the network effect makes low standing socially contagious; gamification converts the entire apparatus from something done to people into something they do to themselves.

graph LR
  A[Comprehensive data integration] --> B[AI behavioral scoring]
  B --> C[Automated enforcement]
  C --> E[Anticipatory self-policing]
  B --> D[Social-network effects]
  D --> E
  F[Gamification] --> E
  E -->|behavior becomes new data| A

The last arrow is the crucial one. A system like this closes a loop: the compliance it produces becomes the data it feeds on, so it tightens over time without anyone tightening it. That feedback is why the design is worth taking seriously even where it remains a blueprint.

But we should hold the epistemic caveat firmly. As of the mid-2020s, China's system was not a single integrated score for individuals; it was strongest against companies (compliance and tax records) and against a specific category of people — those a court had already found in default. A draft national Social Credit Law circulated in 2022, aiming to bring order to the patchwork, but it described boundaries and legal bases more than it described a universal behavioral score. The honest summary is that the components are real and unevenly deployed, the integration is aspirational, and the finished dystopia of the 2018 news cycle does not yet exist. The danger this chapter examines is not that the blueprint is complete. It is that it is coherent, exportable, and being assembled piece by piece — and that the place where the most complete version was actually built is not a pilot city but a region under emergency rule.

Where the Blueprint Became Real: Xinjiang and the Predictive Turn

The most fully realized version of AI governance China has built runs not in Shanghai but in Xinjiang, against the Uyghur population. Human Rights Watch reverse-engineered the phone app used by police there — part of the Integrated Joint Operations Platform (IJOP) — and documented what it collected and flagged: electricity usage that seemed too high, a car registered to someone else, use of a back door, "not socializing with neighbors," donating to a mosque with "excessive enthusiasm," using a VPN, gathering more than a normal amount of food. The system aggregated these signals, generated lists, and prompted officers to investigate — and, in tens of thousands of cases, to detain.

Xinjiang matters analytically because it demonstrates the difference between the reactive and predictive phases of authoritarian AI governance — and this distinction runs through the entire subject. A reactive system watches for an offense and punishes it afterward. A predictive system tries to identify who is likely to offend, or dissent, and intervenes before anything has happened. The IJOP did not wait for a crime. It manufactured a category of suspicion out of ordinary behavior and acted on it pre-emptively.

The pre-emptive interventions that political prediction enables are specific and, from a regime's point of view, elegant. Rather than crush a protest — which creates martyrs, images, and grievances — a predictive system can dissolve the conditions for one. It can flag individuals whose network of associations and communications resembles that of past organizers, and then quietly reassign their jobs, deny them travel, adjust the risk score that governs their daily frictions, or simply send an officer to their door to signal that they have been seen. Nothing visible happens. No crowd forms. The strategic advance over old-style repression is that reactive suppression fights dissent after it becomes legible and dangerous; predictive governance aims to prevent it from becoming legible at all. It redefines the state's problem from defeating opposition to preventing its formation.

Whether the softer, score-based interventions imagined for the general population are anywhere near as capable as the marketing suggests is genuinely uncertain. Predicting who will jaywalk is easy; predicting who will become a dissident is not, and false positives are endemic to this whole class of systems. But the direction of travel — from watching-and-punishing toward forecasting-and-pre-empting — is unmistakable, and it is the feature that makes AI governance qualitatively different from the surveillance states of the twentieth century.

Why Authoritarians Move Faster

Authoritarian states enjoy real structural advantages in deploying these systems — advantages that have nothing to do with any technical genius and everything to do with the absence of friction. This is a causal claim worth taking seriously, because it predicts that the gap between authoritarian and democratic deployment will widen, not close, absent deliberate choices.

The first advantage is the absence of civil-liberties constraints. The single hardest technical problem in the blueprint — integration — is in democracies mostly a legal and political problem. Privacy law, data-protection regimes, judicial warrants, corporate refusal, and press scrutiny keep datasets fragmented. An authoritarian state can simply mandate that the tax system, the health system, the banks, the telecoms, and the cameras all talk to one another. The comprehensive data environment that AI models need to work well is, for them, a policy decision rather than a decade-long negotiation.

The second is uninterrupted planning horizons. China could announce a multi-year surveillance buildout and pursue it across a decade without an election reversing course. Democratic technology policy is reactive and reversible by design; each administration can undo the last. Sustained, coordinated, long-range deployment is structurally easier where power does not change hands.

The third is the ability to compel rather than persuade. When an authoritarian state adds a new data source, every citizen is enrolled the moment the switch is flipped. A democratic equivalent needs enabling legislation, public consultation, opt-in mechanisms, and the constant possibility of legal challenge — friction that can add years or kill a program outright.

The fourth is the freedom to run population-scale behavioral experiments without consent. Refining a governance algorithm requires testing what actually changes behavior. Democracies face research-ethics review, disclosure requirements, and liability; an authoritarian state can treat its population as a live laboratory and iterate. These advantages do not make authoritarian AI governance good — they make it fast and comprehensive. Speed and reach are precisely the qualities that matter in a diffusion contest, which is why the model travels.

Sixty Countries, Three Depths

That travel is the second half of the story. The most careful count remains Steven Feldstein's Carnegie Endowment survey, which found AI surveillance technology in use in at least 75 countries, with Chinese firms — Huawei above all, alongside Hikvision, Dahua, and ZTE — as the leading suppliers, present in roughly half of them. The vehicle is China's Digital Silk Road, the technological arm of the Belt and Road Initiative, which bundles fiber-optic cable, data centers, and "safe city" surveillance packages into infrastructure-and-financing deals. Huawei's safe-city marketing alone has named dozens of deployments. Ecuador's nationwide ECU-911 camera network, Zimbabwe's facial-recognition agreement with the firm CloudWalk, camera systems across Kenyan and Pakistani cities, and monitoring platforms in Central Asia and the Gulf are the concrete cases journalists and researchers have documented.

What does "over sixty countries" actually mean in practice? Not sixty replicas of the Chinese blueprint. Adoption comes in three tiers of depth. At the deepest tier sit a handful of states — China itself, some Gulf monarchies, several Central Asian governments — pursuing integrated surveillance tied to behavioral scoring across many domains. At an intermediate tier, a larger group of developing states has installed the hardware — camera networks, facial recognition, monitoring platforms — and begun scoring in narrow domains like policing or welfare, without the full integration. At the broadest and shallowest tier are countries, many of them democracies, that have adopted individual components — predictive policing, biometric borders, welfare-fraud algorithms — that build authoritarian capability while still operating inside democratic law. The tiers are not a ladder every country climbs, but they describe how the same underlying toolkit shows up at radically different intensities.

Why does the model spread? Partly because it is sold as turnkey: a government need not build AI capacity, only buy a platform and switch it on. Partly because Belt and Road financing makes the surveillance layer part of the price of the road, the port, or the grid. Partly because the effectiveness claims — less crime, smoother services — have enough empirical truth to be persuasive to a resource-strapped government. And partly because, in a decade when democracies have looked chaotic and gridlocked, the appearance of authoritarian competence has itself become a soft-power asset.

The Lock-In Problem — and What We Don't Know About It

Once installed, these systems are hard to remove, and the reasons compound. There is technical dependency: proprietary systems that require the vendor for maintenance, updates, spare parts, and training, so that ripping them out means losing a working capability and starting over. There is the debt structure of Belt and Road financing, which ties the recipient to Beijing financially and raises the cost of any move that would sour the relationship. And there are the political relationships built around the technology — the police forces trained on it, the officials whose authority now runs through it, the ministries that depend on the data. Together these favor deepening over dismantling. The path of least resistance, having bought in, is to buy more.

Here honesty requires care, because this is where evidence is thinnest. The claim that adopting Chinese surveillance infrastructure produces durable political alignment with China is intuitive but hard to prove. Countries that take Chinese surveillance kit tend to be countries already inclined toward China for trade, ideology, or geopolitics — so the surveillance may be a symptom of alignment rather than a cause of it. Sheena Chestnut Greitens and others who have examined the demand side argue that recipient governments are often active buyers pursuing their own domestic-control goals, not passive victims of Chinese strategy. Untangling the lock-in effect of the technology itself from all the other reasons a country stays close to Beijing has not been done cleanly. The lock-in is real at the technical and financial level; its translation into lasting geopolitical alignment is plausible but under-evidenced, and the chapter should not overstate it.

The Psychology of Living Inside It

The most consequential thing about these systems may not be the hardware but what they do to the people inside them. And the striking, uncomfortable finding is that populations under pervasive surveillance frequently do not experience it as oppression. Three mechanisms explain why.

The first is anticipatory compliance. When a system is legible — when you can see how your behavior maps to consequences — you begin to optimize for it in advance, adjusting not because you were caught but because you have internalized the map. The behavior becomes a background habit, like checking the speedometer near a known camera. Compliance comes to feel like prudence, even like personal virtue, rather than submission.

The second is normalization. When everyone is scored, conforming stops being remarkable and becomes simply what one does. Deviance is not just costly; it is lonely, because the surrounding population is oriented toward the same targets. The pressure that enforces the norm is largely social, applied by peers, not by the state directly.

The third, for those old enough to remember an alternative, is learned helplessness — the phenomenon Martin Seligman first described in the 1960s, in which repeated experience of futile resistance produces a settled belief that resistance is pointless. When challenging the system proves costly and ineffective, it becomes psychologically cheaper to reframe it as benign or even good. The reframing is helped along by real material benefits: cleaner services, lower street crime, functioning infrastructure. It is less painful to be grateful than to stay permanently, uselessly angry.

This is not merely a Western surmise. Genia Kostka's survey research on Chinese attitudes toward social credit found approval levels around 80 percent, with support highest among the more educated and affluent — the opposite of what the "oppressed masses awaiting liberation" narrative would predict. Respondents framed the systems as tools for building trust and improving conduct in a low-trust society, not as instruments of control. Survey work under authoritarian conditions carries obvious caveats — people may not answer freely — but the consistency of the pattern is hard to dismiss entirely.

The Generational Lock and the Early-2040s Threshold

The deepest source of stability is generational, and it is worth stating precisely because it drives the temporal argument. A person who remembers life before the cameras experiences them as an addition — a constraint laid over a prior freedom. A person who has never known anything else experiences them as furniture — a feature of the world as unremarkable as streetlights. The discomfort of being watched, the sense of violation, the felt value of privacy: these turn out to depend heavily on having an experiential reference point for the alternative. Remove the reference point, and the objection loses its grip. You cannot mourn a freedom you never had, and you struggle even to imagine it — it becomes not just undesirable but illegible.

This is why the arithmetic of cohorts matters. China's comprehensive surveillance buildout took off in the mid-2010s. The children who have grown up entirely inside it begin reaching adulthood — voting age, working age, the age of forming settled political values — in the late 2030s and early 2040s. That threshold is significant not because anything mechanical happens on a particular date, but because it marks the point when the demographic center of gravity shifts toward people whose sense of legitimate governance was formed within the system. When that cohort becomes the majority of adults, the system no longer has to suppress a memory of alternatives; there is no memory to suppress. Political scientists studying regime durability have long observed that systems which normalize their core mechanisms among the generation raised under them become markedly harder to reverse. The window for reversal, if one is ever open, narrows as those cohorts grow — which is the strongest reason to treat the governance choices of this decade as more consequential than they feel. This is a projection, not an established fact; but the mechanism is well-grounded and the demographic timing is arithmetic.

Are Democracies Converging?

The pressure on democracies is real, and the honest answer to whether they are converging is: partly, and unevenly, and mostly below the level at which anyone has actually decided to.

The pressures are economic, security, and public-safety. Police forces want the crime-fighting edge that facial recognition and predictive tools promise. Border agencies want biometric throughput. Welfare agencies want fraud detection. Each adoption is defended on its own narrow merits, by an agency solving a specific problem, without anyone assessing the cumulative direction. That is what "gradual convergence below the threshold of democratic deliberation" means: the individual steps are small, technical, and rarely debated as what they collectively are, so a society can travel a long way toward pervasive surveillance without ever holding the vote that would be about pervasive surveillance.

The evidence cuts both ways. On the convergence side: London's Metropolitan Police deployed live facial recognition despite years of civil-liberties objections; U.S. departments used predictive-policing products like PredPol and risk-assessment tools like COMPAS, and cities like Detroit ran facial recognition that produced documented wrongful arrests; Clearview AI scraped billions of faces and sold matches to hundreds of law-enforcement agencies. On the divergence side: the European Union's AI Act, which entered into force in 2024, does something no Chinese law does — it prohibits social scoring by public authorities outright, and it sharply restricts real-time remote biometric identification in public spaces. Several U.S. cities banned government facial recognition entirely. Courts, journalists, and legislatures pushed back, sometimes successfully.

So the defensible reading is not that democracies are quietly becoming China. It is that they are accumulating the components of authoritarian capability under democratic control, and that the safeguards which distinguish the two — transparency, contestability, judicial review, the possibility of banning a tool — are real but under strain. The architectural difference persists. Whether it is durable, or is being hollowed out one procurement decision at a time, is exactly the open question.

Where Legitimate Governance Ends

If democracies are going to use AI in government at all — and they will — the important line is not "surveillance versus no surveillance" but the line between accountable AI-assisted governance and authoritarian AI governance. Four criteria draw it usefully.

The first is accountability: is there a specific, identifiable human or institution answerable for each decision, and a real path to contest it? A traffic camera you can appeal to a court is categorically different from a score adjusted by a model no one will explain. The second is scope limitation: is the tool bounded to a defined purpose, or does it aggregate across every domain of life into a single judgment of the person? Purpose-bound systems remain tools; totalizing ones become governance of the person as such. The third is reversibility: can the system be switched off, its data deleted, its decisions undone — and is that capacity real rather than nominal? The fourth is consent and legal basis: was the deployment authorized through legitimate, public, revisable processes, or simply imposed? A society can use AI to detect tax fraud, triage emergency calls, or manage traffic and remain free, provided these four hold. It becomes something else when scope goes total, accountability disappears, reversal becomes impossible, and consent is neither sought nor grantable.

The Normative Knot

This brings us to the hardest question, and it is worth refusing the easy answer. If people under these systems report satisfaction — if Kostka's 80 percent is real, if the generation raised inside feels no violation — does that dissolve the objection?

It complicates it; it does not resolve it. The liberal-philosophical objection to these systems was never primarily that they make people feel bad. It is that they make people unfree — that a life optimized for a score set by an unaccountable authority is diminished even when it is comfortable, because the capacity to think, associate, and dissent otherwise has been engineered away. The absence of subjective suffering is not evidence of freedom; it can be evidence of adaptation, of preferences quietly shaped to fit what is permitted. People come, over time, to want what they can have. That a system succeeds in producing contentment is, from this angle, not a defense but part of the indictment — it means the machinery reaches all the way into what people are able to want. And yet the satisfaction is not nothing. It should make us humble about the assumption that these populations are simply waiting for liberation, and honest that the case against the systems rests on a conception of freedom that their own citizens may not share. The objection survives. It just cannot pretend to be speaking for people who do not experience themselves as harmed.

Can the Effects Be Undone?

The final open question is whether any of this is reversible — and here we mostly do not know. If a country dismantled its surveillance apparatus tomorrow, would anticipatory compliance and normalization lift, or would the habits of mind persist? The most relevant evidence comes from transitional societies that removed pervasive surveillance: post-1989 East Germany after the Stasi, post-authoritarian states in Latin America and the former Soviet bloc. The fragmentary lesson from that literature is that infrastructure can be dismantled far faster than the psychology it produced — that self-censorship, distrust, and habituated caution can outlast the regime that instilled them by a generation. But those cases predate AI governance, and none of them removed a system that had shaped an entire cohort from birth. Whether the specific effects of algorithmic governance are more or less reversible than analog surveillance is genuinely unstudied. It is one of the most important unknowns in this whole field, and it bears directly on the window-for-reversal argument: if the psychology is sticky, then even removing the hardware may not restore the capacity to want something else.

Three Trajectories

Where does this go? Three trajectories are worth naming, each favored by different conditions.

In the democratic-alternative trajectory, democracies build and export AI governance tools that deliver the public-safety and efficiency benefits without the total surveillance — bounded, accountable, reversible systems — and offer partner countries a real choice against the Chinese package. This is favored by coordinated democratic investment, credible financing that competes with Belt and Road terms, and enforceable standards that make the alternative attractive rather than merely preachy. It is the least likely on current trends and the most dependent on deliberate action.

In the gradual-convergence trajectory, democracies drift toward authoritarian capability one unremarked procurement at a time, until the global distinction between the two models blurs without any decisive, contestable moment. This is favored by exactly the dynamics now visible: agency-by-agency adoption, public ambivalence, security and economic pressure, and the absence of any forum that assesses the cumulative direction. It is, uncomfortably, the path of least resistance.

In the entrenched-authoritarian-norm trajectory, the model becomes so widespread and so generationally normalized across large populations that reversal ceases to be practically available within relevant political horizons. This is favored by continued diffusion, deepening lock-in, and the cohort arithmetic reaching its threshold before any counterweight is built. Which trajectory prevails is not fixed. But the three are not equally probable from where we stand, and the two darker ones require only that we keep doing what we are already doing.

Summary

The famous "citizen score" is largely a myth, but the blueprint behind it is real. China's Social Credit System is fragmented — court blacklists, municipal pilots, corporate compliance databases — not a unified national score. Yet its five-part design (data integration, behavioral scoring, automated enforcement, network effects, gamification) is coherent, and it forms a self-reinforcing loop in which compliance becomes the data that tightens the system.
The clearest full deployment is Xinjiang, where the IJOP platform demonstrated the shift from reactive punishment to predictive pre-emption — dissolving the conditions for dissent before it forms rather than crushing it afterward.
Authoritarian states deploy faster for structural reasons: no civil-liberties friction on data integration, uninterrupted planning horizons, the power to compel rather than persuade, and freedom to experiment on populations. Speed and reach, not superior technology, are the advantage.
The model has diffused to 60-plus countries via the Digital Silk Road, in three tiers of depth from full integration to isolated components. Lock-in — technical dependency, debt, and political ties — favors deepening over dismantling, though the claim that it produces durable alignment with China is under-evidenced.
Populations often do not experience these systems as coercive, through anticipatory compliance, normalization, and learned helplessness; survey evidence shows high approval. Generational normalization is the deepest lock: the cohort raised entirely inside the system reaches adulthood in the early 2040s, and lacks the reference point that would make alternatives even imaginable.
Democracies are partly converging — facial recognition, predictive policing, biometric borders — mostly below the threshold of explicit deliberation, even as the EU AI Act's ban on social scoring shows the architectural difference is being defended.
The line between acceptable and authoritarian AI governance turns on accountability, scope limitation, reversibility, and consent. The normative objection to these systems survives citizen satisfaction: contentment engineered by the system is part of the problem, not a refutation of it.
Two big unknowns remain: whether the psychological effects are reversible once the hardware is removed, and which of three trajectories — democratic alternative, gradual convergence, or entrenched authoritarian norm — prevails. The darker two require only that current trends continue.

Sources

Last updated: 2026-09-12

V2 (in progress) Previous: V1