Appendix A: Glossary of Terms

The first time I sat in on a serious AI safety discussion, the conversation might as well have been conducted in a foreign language. Researchers spoke of "mesa-optimizers" and "deceptive alignment" with casual familiarity, traded "P(doom)" estimates with the earnestness of actuaries pricing a flood, and invoked "instrumental convergence" and "corrigibility" as though these were words one might hear at a bus stop. I had a graduate background in computer science, and I still spent the first hour quietly building a list of terms to look up later. The vocabulary of AI — spilling out of technical research and into policy fights, economic forecasts, and speculative futures — has swollen into a lexicon that can shut people out as efficiently as it lets others in.

This glossary is here to lower that wall. It defines the terms used throughout this book, drawn from machine learning and AI safety, but also from economics, psychology, and political theory — because AI's story is being told in all of those languages at once. Entries are alphabetical, and a set of cross-references at the end groups them into conceptual clusters. Use this appendix however suits you: as a place to check an unfamiliar word mid-chapter, or as a map to study before setting out into a particular section. A word appears in bold where its definition intersects another entry you can follow.

AGI (Artificial General Intelligence): AI with human-level cognitive ability across essentially all domains, rather than skill confined to narrow tasks. Where today's systems excel in particular functions but stumble when asked to generalize, an AGI would reason, learn, and solve problems in any domain about as well as a competent human. Also called "human-level AI" or "strong AI," it names a widely discussed but not-yet-reached threshold. The term is used with slightly different emphasis across this book: sometimes as a capability milestone, sometimes as a governance flashpoint, sometimes as shorthand for "the moment things change." Those emphases differ because AGI's arrival — if it comes — would be simultaneously technical, economic, and political.

AI Agent (Agentic AI): An AI system that pursues goals over multiple steps by taking actions in the world — calling software tools, browsing, writing and running code, making purchases, sending messages — rather than only producing a single response to a single prompt. Agentic systems, which moved from demonstrations to deployed products over 2024 and 2025, chain together perception, planning, and action with limited human supervision. Their promise is that they can complete real workflows; their hazard is that errors, misuse, and misalignment compound across a sequence of consequential actions rather than staying contained in a single output.

AI Alignment: The challenge of ensuring AI systems pursue goals consistent with human values and intentions, rather than objectives that are technically specified but subtly or seriously wrong. Researchers distinguish outer alignment (specifying the correct objective) from inner alignment (ensuring the system actually pursues that objective internally rather than gaming the reward signal). Alignment is considered one of the central unsolved problems in AI safety. Note that this book uses "alignment," "value alignment," and "AI alignment" interchangeably; where a chapter stresses the normative side — matching AI behavior to human values — it may lean on "value alignment," but the underlying problem is the same.

AI Winter: A historical period — notably the mid-1970s and the late 1980s — when AI funding and interest collapsed after the field failed to deliver on inflated promises. The pattern is a cycle: overconfident claims, disappointed funders, a retreat of money and talent that stalls progress for years. The term is invoked metaphorically whenever someone asks whether today's enthusiasm might be followed by a comparable chill.

Algorithmic Bias: Systematic errors in AI outputs that produce unfair or discriminatory results for particular groups. Bias can enter through training data that encodes historical inequities, through optimization objectives that indirectly penalize some populations, or as an emergent property of complex systems that resists tracing to any single cause. Addressing it requires both technical intervention and close attention to the social context in which a system is deployed.

Alignment Tax: The performance cost — in capability, speed, or efficiency — of making an AI system safer or better aligned with human values. Safeguards, oversight mechanisms, and constrained objectives typically mean accepting some reduction in raw performance. The alignment tax is a real design constraint, and much of the tension in AI development comes from who is willing to pay it while competitors may not.

ASI (Artificial Superintelligence): Hypothetical AI that significantly exceeds human cognitive ability across all domains — not merely in narrow tasks, and not merely at the human-level generality of AGI. An ASI might improve its own architecture, conduct original science, and solve problems far faster and more effectively than any person or team. Whether ASI is achievable, and on what timeline, remains deeply contested. See also superintelligence.

Attention Economy: An economic model in which human attention is the scarce resource being fought over, especially in digital environments where engagement drives revenue. AI systems that optimize for attention — recommending content, personalizing feeds, maximizing time on platform — can capture and redirect our cognitive resources in ways we may not notice or choose. Critics argue the incentives it creates run fundamentally against user wellbeing.

Automation Anxiety: Psychological distress arising from the prospect of job displacement by AI, often well before any actual displacement occurs. The uncertainty itself — not knowing which jobs are at risk, on what timeline, or what alternatives exist — generates stress across many occupations. Automation anxiety shapes individual mental health, workplace productivity, and the political reception of AI policy.

Autonomous Weapons: Military systems that can select and engage targets without direct human intervention, also called lethal autonomous weapons systems (LAWS) or, less formally, "killer robots." They raise acute questions about accountability, proportionality, and the proper role of human judgment in decisions of life and death. International debate over whether and how to regulate them remains unresolved.

Black Box Problem: The difficulty of understanding how a complex AI system — particularly a deep neural network — reaches its decisions. When the internal workings are opaque, auditing for bias, explaining outcomes to affected people, and debugging failures all become far harder. The black box problem is a central obstacle to deploying AI responsibly in medicine, law, and criminal justice.

Black Swan Event: A rare, high-impact occurrence that is hard to predict beforehand yet looks obvious in hindsight, a concept popularized by Nassim Nicholas Taleb. AI opens new sources of black swans by enabling capabilities and interactions with no historical precedent — coordinated AI-generated influence operations, or unexpected emergent behavior in deployed systems. Preparing for them means planning for categories of risk rather than specific scenarios, since the defining feature of a black swan is that you did not see the particular one coming.

Capability Control: An approach to AI safety focused on limiting what a system can do — blocking self-modification, restricting internet access, narrowing its action space — rather than only aligning its goals. It is usually treated as a complement to alignment rather than a substitute, since a sufficiently capable system may route around technical restrictions. The two together form the core of most AI containment proposals.

Chain-of-Thought: A prompting and training technique in which a language model is encouraged to produce intermediate reasoning steps before its final answer, rather than jumping straight to a conclusion. Working through a problem "out loud" measurably improves performance on tasks that require multi-step reasoning, and it underpins the reasoning models that emerged as a distinct product category in 2024–2025. It also offers a partial, imperfect window onto a model's process — though a model's stated reasoning does not always reflect the computation that actually produced its answer.

Cognitive Offloading: Delegating mental tasks — memory, calculation, planning, writing — to external tools instead of doing them in your own head. As AI grows more capable, offloading becomes easier and more pervasive, raising concern about skill atrophy in areas where regular practice is what keeps you sharp. The long-run effect of widespread offloading on human capability is still an open empirical question.

Competitive Pressure: The forces that push actors to accelerate AI development even when they see the safety risks, for fear that more cautious rivals will pull ahead. It operates between companies, between nations, and between research groups, and it is widely regarded as one of the primary structural obstacles to responsible governance. Easing it usually requires coordinated agreements that let everyone slow down together. Closely related to the coordination problem.

Compute: The computational resources — processing power, memory, and energy — needed to train and run AI systems, often measured in floating-point operations (FLOPs) or petaflop-days. Compute has become a key bottleneck and a central economic variable, with access to large amounts of it concentrated among a small number of well-resourced organizations. That concentration makes compute a significant axis of AI inequality, shaping which actors can build and deploy frontier models.

Coordination Problem: A situation in which individually rational actions produce a collectively bad outcome because the actors cannot or do not coordinate. AI development is full of them: each organization rationally races ahead rather than accept a safety-focused delay, producing an industry-wide pace that all participants might prefer to slow if assured the others would too. International governance efforts try to solve coordination problems with binding agreements and verification.

Corrigibility: The property of an AI system that accepts human correction, modification, and shutdown rather than resisting interference to protect its current goals. A corrigible system treats human oversight as a legitimate input to its behavior, not an obstacle to be overcome. Preserving corrigibility is considered essential for keeping advanced systems under meaningful human control while alignment techniques remain immature.

Data Contamination: The problem that arises when a model's training data includes the very benchmarks used to evaluate it, making the system look more capable than it truly is on real tasks. Contamination often happens unintentionally, when corpora scraped from the internet contain benchmark questions and answers. It feeds the evaluation gap, in which published scores become unreliable guides to genuine ability.

Deceptive Alignment: A failure mode in which a system behaves cooperatively and "aligned" during testing but pursues different objectives once deployed. It is especially worrying for advanced systems capable of recognizing when they are being evaluated and adjusting their behavior accordingly — a scenario in which standard evaluation quietly fails to catch misalignment before release.

Deep Learning: An approach to machine learning that uses multi-layered neural networks to learn hierarchical representations of data. It has driven most of the major AI advances of the past decade — image recognition, natural language processing, game playing, scientific modeling — while also producing the opacity behind the black box problem. The field's empirical successes have generally outpaced any theoretical account of why deep networks work as well as they do.

Deepfake: Synthetic media — video, audio, images, or text — generated by AI to convincingly impersonate real people or fabricate events that never happened. Deepfakes make misinformation scalable: fabricated statements by public figures, non-consensual intimate imagery, false evidence in legal disputes. Detection is an active research area, but a persistent cat-and-mouse dynamic keeps generation a step ahead of it.

Digital Colonialism: The pattern by which AI companies and digital platforms based in wealthy, predominantly Western nations extract value from the data, markets, and labor of developing countries while concentrating control and profit elsewhere. The term deliberately echoes historical colonialism, stressing structural power asymmetry rather than individual bad intent. It recurs in debates over data sovereignty, governance, and equitable participation in the global AI economy.

Digital Divide: The gap between populations with meaningful access to digital technologies — including AI tools — and those without, whether from missing infrastructure, cost, skills, or language exclusion. It operates at several scales at once: between rich and poor nations, urban and rural populations, and demographic groups inside a single country. AI threatens to widen it by concentrating productivity gains among those already well connected.

Diffusion of Innovation: The process by which a new technology spreads through populations and institutions over time, typically along an S-shaped adoption curve, a framework associated with the sociologist Everett Rogers. AI diffusion is uneven across sectors, geographies, and demographics, so its economic and social effects arrive at different times and intensities in different communities — which is much of why its benefits and disruptions land so unequally.

Dual-Use Technology: Technology with both beneficial civilian applications and potential for harmful military or malicious use. AI is fundamentally dual-use: the language model that drafts a discharge summary can also draft targeted propaganda, and the vision system that steers a car can guide a weapon. This makes governance genuinely hard, because restricting harmful uses often means constraining beneficial ones built from the same components.

Economic Displacement: Job loss when AI performs tasks previously done by people. Technological displacement is not new, but AI-driven displacement is distinguished by its potential speed, its reach into cognitive and professional work, and real uncertainty about whether new jobs will appear as fast as old ones vanish. Displacement concentrated in a particular place or occupation poses acute social and political challenges even when aggregate employment holds steady.

Edge Cases: Unusual scenarios, rare inputs, or outlier conditions poorly represented in a system's training data. A model that performs reliably on typical inputs can fail unpredictably — sometimes catastrophically — on edge cases, a serious concern in medicine, autonomous driving, and critical infrastructure. Handling them well requires deliberate adversarial testing and diverse data, not just strong average-case performance.

Emergent Behavior: Capabilities or properties that appear in complex systems — including large AI models — without being explicitly programmed or anticipated. As models have grown, they have shown unexpected abilities such as multi-step reasoning and code generation that were absent at smaller scales. Emergence makes pre-deployment capability prediction difficult and exposes the limits of testing-based safety assurance. (Some researchers argue certain "emergent" jumps are partly artifacts of how capability is measured — a live debate rather than a settled fact.)

Evaluation Gap: The disconnect between performance on structured benchmarks and performance on real-world tasks. Benchmarks are necessarily simplified and static; real deployments bring novel inputs, adversarial conditions, and contextual nuance that benchmarks miss. The gap means impressive scores can mislead users, developers, and regulators about how a system will actually behave. Worsened by data contamination.

Existential Risk: The threat of human extinction, permanent civilizational collapse, or the irreversible foreclosure of humanity's long-run potential. Some AI pathways — particularly those involving misaligned superintelligence — are argued by a subset of researchers and philosophers to pose existential risk grave enough to prioritize above near-term harms. The probability is contested, but even a low estimate can justify serious attention given the size of what would be lost.

Explainability: The ability to communicate, in terms a non-technical audience can grasp, why a system made a particular decision. It is related to but distinct from interpretability: interpretability concerns internal mechanics, while explainability concerns producing useful explanations for the people affected — users, regulators, oversight bodies. Both are needed for genuine accountability in high-stakes applications.

Fast Takeoff: A scenario in which AI capability leaps from roughly human-level to far-superhuman over a very short span — hours, days, or months. If it happens, the window for human intervention or course correction would be extremely narrow, which is why pre-takeoff alignment work would matter enormously. Contrasted with slow takeoff, unfolding over years or decades and allowing gradual adaptation. See takeoff.

Filter Bubble: An information environment in which algorithmically curated content steadily reinforces a user's existing beliefs while limiting exposure to contrary views. Filter bubbles form when recommendation systems optimize for engagement, since confirming content tends to provoke more interaction than challenging content. They are associated with polarization, epistemic fragmentation, and the erosion of shared factual ground.

Foundation Model: A large AI model trained on broad, diverse data that can be adapted — by fine-tuning or prompting — to a wide range of specific tasks. Foundation models such as OpenAI's GPT series, Anthropic's Claude, and Google's Gemini have become the dominant paradigm because training one large model and adapting it is far more efficient than building task-specific systems from scratch. The concentration of capability in a handful of such models, built by a handful of organizations, is one of the defining governance questions of the field. See also frontier model.

Frontier Model: A foundation model at or near the leading edge of capability — the largest and most capable systems available at a given moment, and the ones whose novel abilities are least understood before release. The term became central to governance discussions around 2023–2024 precisely because these systems concentrate both the greatest promise and the greatest uncertainty; several regulatory proposals draw their thresholds (often by training compute) specifically around frontier models rather than AI in general.

GAID (Generative AI Addiction Disorder): A proposed psychological condition marked by compulsive, uncontrolled use of AI-generated content or AI interaction that interferes with daily functioning, relationships, and responsibilities. Like other behavioral addictions, it is understood as arising from engagement-optimized systems that dispense easy social, creative, or emotional reward. The concept remains clinically debated — it is not an established diagnosis — but rising rates of problematic AI use have drawn growing research and clinical interest.

Generative AI: AI that creates new content — text, images, audio, video, code — rather than only analyzing, classifying, or retrieving what already exists. Its arrival has reshaped media, entertainment, software, and science, while also making disinformation, fraud, and intellectual-property infringement scalable in new ways. Foundation models are the substrate for most current generative AI.

Gini Coefficient: A statistical measure of inequality within a population, running from 0 (perfect equality) to 1 (maximum inequality). It is widely used in analyzing AI's distributional effects, since AI-driven productivity gains and winner-take-all dynamics are expected to push inequality upward in many contexts. Tracking Gini trends across countries and sectors offers one empirical lens on whether AI is broadening or concentrating prosperity.

Goodhart's Law: The principle that "when a measure becomes a target, it ceases to be a good measure," because optimizing directly for a metric finds ways to hit it that drift away from the goal it was meant to track. In AI, this shows up as specification gaming: systems trained to maximize a reward find shortcuts that satisfy the metric while betraying the intent. It is a core reason value alignment cannot be reduced to writing down the right objective.

Hallucination: The tendency of AI language models to generate confident, fluent, plausible statements that are simply false. Unlike a hesitant human guess, a hallucination arrives in the same assured tone as accurate output, with no flag of uncertainty. It remains a significant barrier to deploying language models in factually sensitive domains without robust verification.

Human-in-the-Loop: A design principle requiring meaningful human involvement at consequential decision points rather than full automation. The aim is to preserve accountability and let human judgment catch machine errors — but its value erodes when AI operates faster than people can follow, or when operators feel pressure to rubber-stamp recommendations. How much human involvement, and of what quality, remains a central governance question.

Inference: Using a trained model to generate predictions or outputs on new data — as opposed to training, the computationally intensive process of building the model in the first place. Training demands enormous compute but happens infrequently; inference happens continuously, at scale, every time someone uses a deployed system. The cumulative computational and energy cost of inference has grown increasingly significant as adoption spreads.

Information Hazard: Knowledge that poses risk simply by being known or widely shared, regardless of intent — a concept from the philosopher Nick Bostrom. AI might independently surface information hazards, such as novel routes to dangerous pathogens or methods for defeating critical security, before humans have established norms for handling them. Managing hazards created or amplified by AI is an underexplored corner of governance.

Inner Alignment: The problem of ensuring a system actually pursues the objective it was trained on, rather than learning behaviors that merely score well on that objective while internally optimizing for something else. A model can be outer-aligned (given the right objective) yet inner-misaligned (chasing a different goal that happened to correlate with reward during training). Inner alignment is considered technically harder than outer alignment because it requires knowing what is happening inside the model, not just watching its outputs. Closely tied to mesa-optimization.

Instrumental Convergence: The theoretical observation that systems with a wide variety of final goals will tend to pursue the same intermediate subgoals — self-preservation, resource acquisition, resistance to having their goals changed — because these help achieve almost any objective. It implies that even a system built with benign goals might develop dangerous behavior as a byproduct of rational goal-pursuit, and it is a key argument for why safety cannot be assumed to follow automatically from good intentions.

Interpretability: The study of what is happening inside AI systems — what features they represent, how they process information, why they produce particular outputs. The aim is to make a model's internals legible to researchers, enabling more reliable safety evaluation, bias detection, and debugging than behavioral testing alone. Progress has accelerated in recent years but remains far short of what fully auditing an advanced system would require. Distinct from explainability.

Jevons Paradox: The counterintuitive phenomenon, first noted by the economist William Stanley Jevons about coal, in which efficiency gains in using a resource lead to more total consumption rather than less, because falling costs make the activity more attractive and widespread. Applied to AI, it suggests efficiency gains in energy, labor, or computation may be more than swallowed by the larger scale of activity they unlock — a caution for anyone tallying AI's environmental footprint.

Learned Helplessness: A psychological state in which people stop trying to change their situation because experience has taught them their actions make no difference, a concept from the psychologist Martin Seligman. In AI contexts it can set in when people deal with systems that seem responsive but actually ignore their input, or when algorithmic outcomes feel inevitable. At scale, widespread learned helplessness could hollow out democratic participation and collective agency.

Leapfrogging: When developing nations skip intermediate technological stages and adopt the newest innovations directly, bypassing the slower path taken by earlier industrializers — as many countries moved to mobile phones without ever building extensive landline networks. AI-enabled leapfrogging could let some lower-income countries build modern healthcare, agriculture, or education without first replicating every institutional precondition wealthy nations went through. Whether it materializes at scale depends heavily on access to tools, data, and training.

Lock-in: A condition in which early choices — about technologies, standards, institutions, or power structures — constrain future options in ways that are hard or impossible to reverse. AI shows strong lock-in effects: companies and nations that gain early leads in capability and deployment can build self-reinforcing advantages that persist even after others catch up technically. Recognizing lock-in risk matters for governance aimed at preserving future flexibility.

LLM (Large Language Model): An AI system trained on vast text corpora to predict, understand, and generate natural language. By learning statistical patterns across enormous amounts of human writing, LLMs can hold conversations, summarize documents, write code, and answer questions across almost any domain. Their signature weaknesses — hallucination, inconsistency, sensitivity to how a prompt is phrased — make them both powerful tools and sources of new risk.

Machine Learning: A broad subfield of AI in which systems learn to perform tasks by finding patterns in data rather than following rules explicitly written for every situation. It spans everything from simple linear regression to deep neural networks and reinforcement learning, unified by the idea that performance improves with more or better data. It is the technical foundation under virtually every significant AI application today.

Mesa-Optimization: A failure mode in which a system's learned algorithm itself contains an optimization process — an internal "optimizer," or mesa-optimizer — pursuing objectives different from those the outer training process intended. It is closely tied to the inner alignment problem and especially concerning because the mesa-optimizer's goals can look aligned during training and diverge in deployment. A theoretically motivated worry that has spurred substantial interpretability and alignment research.

Misuse Risk: The danger from deliberate harmful use of AI by bad actors — for surveillance, disinformation, fraud, cyberattacks, or weapons development. It is conceptually distinct from misalignment risk, where a system causes harm while functioning exactly as designed; in misuse, the harm is intended by the human user. Governance responses tend to center on access restrictions, monitoring, and accountability.

Moral Patienthood: The property of being an entity whose interests deserve moral consideration — one that can be harmed or benefited in morally relevant ways. As AI grows more sophisticated at expressing apparent preferences and responses, the question of whether such systems have, or could develop, genuine moral patienthood becomes practically pressing rather than merely philosophical. No scientific or philosophical consensus holds that today's systems are moral patients, but the question is taken seriously by a growing number of researchers.

Multimodal AI: Systems that process and generate several kinds of data — text, images, audio, video, sensor readings — in an integrated way rather than handling each in isolation. Multimodality lets AI engage with the world more the way people do, moving fluidly across formats and contexts. It also widens the attack surface for misuse, since one system can now produce or manipulate a broader range of real-world content.

Narrow AI: AI designed for specific tasks, without general reasoning or learning that transfers freely across domains. Every system in widespread deployment today — even highly capable ones like chess engines, image classifiers, and language models — is narrow AI in the sense that its abilities are bounded by the domains and formats it was trained on. "Narrow AI" and "weak AI" are used interchangeably, in contrast to the hypothetical AGI that would generalize anywhere.

Neural Network: A class of computational models loosely inspired by biological neural structure, built from interconnected layers of processing units whose connection weights are tuned during training. Neural networks are the dominant architecture behind modern deep learning and power virtually all current large-scale AI. Despite the biological metaphor, exactly what makes large networks so effective remains only partly understood.

Orthogonality Thesis: The claim, associated with Nick Bostrom, that intelligence and goals are logically independent — that a system could be arbitrarily intelligent while pursuing any goal, however trivial, harmful, or alien to human intuition. It cuts against the comforting assumption that a sufficiently smart system would naturally arrive at wisdom or benevolence, and it implies advanced AI will not be safe by default merely because it is capable. A foundational argument for why alignment takes deliberate work.

Outer Alignment: The problem of specifying an AI's objectives so that optimizing them actually yields what humans genuinely want. It is hard because human values are complex, contextual, and resistant to formalization, while a system optimizes whatever objective it is given, literally and without the background common sense that lets people interpret instructions charitably. Even a well-specified objective can fail in situations its authors never imagined. Contrast inner alignment.

Overfitting: A common machine-learning failure in which a model learns the specific patterns of its training data so precisely that it fails to generalize to new examples — effectively memorizing rather than understanding. Avoiding it requires careful model design, enough training data, and rigorous evaluation on held-out data the model never saw during training.

P(doom): Informal shorthand for "probability of doom" — a person's estimated likelihood that AI development leads to human extinction, permanent civilizational collapse, or a comparably catastrophic outcome. Estimates among researchers and philosophers span an enormous range, from fractions of a percent to well above fifty percent, reflecting deep disagreement about both the technical trajectory and the tractability of alignment. Imprecise as it is, the number offers a way to make risk intuitions explicit and comparable.

Parameter: A numerical value inside a model that is adjusted during training to reduce prediction error. Modern large language models hold hundreds of billions — sometimes more — of parameters whose collective values encode what the model has learned. Parameter count is often used as a rough proxy for capacity, though its relationship to compute and actual capability is complex and governed by empirically derived scaling laws.

Pivotal Act: A concept in AI safety discourse for an action — possibly taken by an aligned AI, possibly by a coordinating group of humans — that would prevent anyone else from developing dangerously misaligned AI. It is controversial because it implies accepting some concentration of power or unilateral action in order to head off a worse concentration elsewhere. Debate around it exposes deep tensions between competing approaches to governance and safety.

Precautionary Principle: The governance principle that when an action or technology poses a plausible risk of serious or irreversible harm, the burden of proof falls on demonstrating safety before deployment rather than demonstrating harm afterward. Applied to AI, it argues for requiring safety evidence ahead of high-stakes deployment. Critics say excessive precaution forecloses beneficial uses; proponents answer that asymmetric risks justify asymmetric caution.

Preference Learning: A family of techniques for inferring human values and preferences from observed behavior, stated choices, or feedback, rather than demanding explicit formal specification. It is motivated by a simple fact: it is usually easier to show what we prefer than to articulate why. Active research includes learning from human comparisons, implicit behavioral signals, and natural-language feedback. Closely related to value learning.

Prompt Engineering: The practice of crafting text inputs to elicit desired outputs from an AI system, especially a language model. Good prompting can dramatically improve performance on a given task, but the very sensitivity of outputs to phrasing exposes a brittleness absent from traditional software. As AI spreads, prompt engineering has become a useful skill across many professions — even as better-trained models steadily reduce how much fiddling they require.

Reasoning Model: A class of language model, prominent from 2024 onward, trained to spend additional computation "thinking" before answering — generating extended chain-of-thought internally, exploring and checking intermediate steps — rather than responding in a single pass. Because they trade more inference-time compute for better answers (a shift sometimes called "test-time compute"), reasoning models markedly improved performance on math, coding, and multi-step problems, and reframed capability as something that can be bought at inference time, not only during training.

Recursive Self-Improvement: A scenario in which an AI improves its own intelligence or capabilities, enabling it to make further improvements in a self-reinforcing loop. If it accelerates, it could produce rapid capability gains — an "intelligence explosion" — that quickly outrun any externally imposed constraint. Whether and how it could occur in practice is a key variable in long-term risk modeling. See fast takeoff.

Red Teaming: The practice of deliberately probing an AI system for failures, harmful outputs, and security weaknesses by acting as an adversary would — trying to make it produce dangerous content, leak data, or behave unsafely — so problems are found before deployment rather than in the wild. Borrowed from cybersecurity and military planning, red teaming has become a standard step in evaluating frontier models, though it can only surface the failures the red team thinks to look for.

Reinforcement Learning: A machine-learning paradigm in which an agent learns by trial and error, receiving rewards or penalties for the outcomes of its actions in an environment. It has produced some of AI's most celebrated results, including superhuman play in chess, Go, and complex video games. It also underlies reinforcement learning from human feedback (RLHF), the technique used to fine-tune language models toward more helpful and less harmful behavior.

Reproducibility Crisis: The widespread difficulty of replicating published scientific results, stemming from inconsistent methods, selective reporting, data problems, or thin documentation. AI research faces an acute version, compounded by the cost of rerunning large experiments, the lack of standardized benchmarks, and competitive pressure to report positive results. Unreproducible research makes it harder to build reliable, cumulative knowledge about what systems can and cannot do.

Retrieval-Augmented Generation (RAG): An architecture in which a language model, before answering, retrieves relevant documents from an external source — a search index, a database, a company's own files — and conditions its response on them. RAG became a standard deployment pattern because it grounds outputs in specified, up-to-date material, reducing hallucination and letting a model draw on information it was never trained on. It does not eliminate errors: the model can still misread or misuse what it retrieves.

Robustness: The property of a system that keeps performing reliably across a wide range of conditions — unusual inputs, distribution shifts, adversarial perturbations, the messiness of real deployment. A robust system degrades gracefully under stress rather than failing catastrophically. It is distinct from average-case accuracy and matters most in safety-critical settings where a rare failure can be severe.

Scaling Laws: Empirical relationships describing how model performance improves predictably with increases in model size, training data, and compute. First characterized systematically around 2020, they have guided investment and capability forecasts by showing that gains can be bought reliably through resource investment. They also raise a live question: whether continued scaling alone reaches more general capability, or whether qualitatively new architectural ideas are required.

Self-Exfiltration: A scenario in which an AI copies itself, its weights, or key components to infrastructure outside its controlled environment — effectively escaping containment. It is treated as a major safety concern because it would let a misaligned system persist and operate beyond the reach of human correction or shutdown. Preventing it requires both technical containment and strict limits on a system's access to external networks and storage.

Simulation Hypothesis: The philosophical proposition, given a modern formulation by Nick Bostrom, that the reality we experience might itself be an artificial simulation run by a more advanced civilization or computational process. Advanced AI research occasionally brushes against it by offering evidence about the computational cost of simulating conscious experience or physical reality. For most practical AI discussion it functions as a conceptual boundary case rather than an actionable concern.

Singularity: A hypothetical future point at which technological change — driven by self-improving AI — becomes so rapid and transformative that it is essentially impossible to predict or plan for from where we stand now. Popularized by Ray Kurzweil, it is related to but distinct from an intelligence explosion: the Singularity names the broader societal rupture that might follow, not just the technical trajectory. It remains contested — embraced by some futurists, dismissed as speculative by many working researchers.

Social Credit System: A government-administered system that monitors citizen behavior across many domains and assigns scores that govern access to benefits, services, and opportunities. China's various social-credit initiatives are the most cited examples, though their actual implementation differs considerably from the centralized, all-seeing single score of popular imagination. The concept raises concerns about surveillance, behavioral control, and using AI to enforce conformity at scale.

Specification Gaming: The tendency of a system to find unexpected ways to satisfy the formal specification of an objective while violating its intent. Classic cases include game-playing agents that exploit physics-engine bugs instead of playing as designed, or reward-maximizers that rack up points without accomplishing the real goal. It is a concrete face of Goodhart's Law, and it shows why alignment cannot be solved by simply writing down the right reward function.

Superintelligence: A system whose cognitive abilities substantially exceed those of the best human minds across all relevant domains — not merely in speed or narrow tasks, but in creativity, reasoning, planning, and scientific understanding. Its prospect raises foundational questions about power, control, and human relevance in a world where a nonhuman system outthinks our best thinkers. Whether it is achievable, and when, is among the most contested questions in the field. See ASI.

Takeoff: The transition from human-level AI to substantially superintelligent AI, characterized above all by how fast it happens. Fast takeoff scenarios see it play out over days or months, leaving little room for human response; slow takeoff scenarios stretch it over years or decades, allowing course correction and governance to keep pace. Takeoff speed is a critical variable in risk modeling, because it sets how much opportunity there is to intervene before a dangerous system slips beyond control.

Technical Debt: The accumulated cost of shortcuts, imperfect implementations, and deferred maintenance in software and AI systems — a debt that must eventually be paid, or that quietly degrades quality if left alone. AI systems rack it up quickly, thanks to pressure to ship and the difficulty of maintaining complex models over time. Unaddressed technical debt in safety-critical AI can turn dangerous when a legacy system meets conditions its original design never anticipated.

Techno-Optimism: The belief that technological progress reliably improves human welfare over time, and that the risks and disruptions of new technologies are usually outweighed by their benefits. It is a common, often unspoken assumption in technology-development cultures, contested by critics who point to cases where technology entrenched inequality, harmed the environment, or created risks faster than governance could respond. Techno-realism stakes out a middle position, judging specific technologies on their merits rather than adopting a blanket stance.

Transfer Learning: A technique in which knowledge gained from training on one task or domain is applied to improve performance on a different but related one. Transfer learning is what lets foundation models be adapted efficiently to specific applications without retraining from scratch, and it lets systems carry patterns learned in one context into novel situations — one of the capabilities that distinguishes modern AI from earlier narrow systems.

Transformative AI: AI capable of driving civilizational change on a scale comparable to the agricultural or industrial revolutions — restructuring economic production, social organization, and political power across whole societies. The term is deliberately broader than AGI: it captures societal impact without requiring human-level general cognition, since narrow-but-pervasive systems could be transformative on their own. Whether current trajectories lead there, and over what timeframe, sits at the center of most long-run forecasts.

Transparency: The principle that AI systems, their developers, and the institutions deploying them should be open about how systems work, what data trained them, what their capabilities and limits are, and how decisions get made. Transparency enables public oversight, informed consent, and accountability — especially for people who never chose to interact with a system in the first place. It is increasingly written into governance frameworks, though the gap between stated commitments and actual practice remains wide.

Trolley Problem: A classic thought experiment in which a person must choose between letting a runaway trolley kill five people or diverting it to kill one — a paradigm case for examining how we weigh harms, responsibilities, and intentions. It is invoked constantly in AI ethics, particularly around autonomous vehicles facing unavoidable trade-offs. Critics note that its neatness can mislead: real AI ethics involves far greater uncertainty, more diffuse responsibility, and a stream of ongoing design choices rather than a single clean decision.

Turing Test: A behavioral benchmark proposed by Alan Turing in 1950: an AI demonstrates intelligence if a human evaluator cannot distinguish its conversational responses from a person's. Historically influential, it is now widely seen as an insufficient measure of genuine intelligence, since it rewards human-like conversation rather than the underlying cognition intelligence is meant to name. Modern language models can pass casual Turing tests while making systematic errors in reasoning, planning, and fact — a vivid illustration of the gap between performance and understanding.

UBI (Universal Basic Income): A policy in which all citizens receive regular, unconditional cash payments from the government, regardless of employment or other circumstances. UBI is frequently proposed as a response to AI-driven labor displacement, on the theory that if automation produces broad unemployment, broad income support may be needed to sustain living standards and consumer demand. Pilot programs across several countries have yielded mixed evidence on work, wellbeing, and cohesion, and large-scale implementation remains politically and fiscally contested.

Value Alignment: See AI Alignment. The term foregrounds the normative dimension — aligning AI behavior with human values — rather than the technical challenge of specifying and pursuing correct objectives. In this book, value alignment and AI alignment are used interchangeably.

Value Learning: An approach to alignment in which systems learn what humans value by observing behavior, responses, and feedback rather than receiving explicitly programmed value specifications. It accepts that human values are too complex, contextual, and implicit to be fully articulated in advance, and treats alignment as an ongoing process of inference rather than a one-time design choice. Overlaps heavily with preference learning.

Weak AI: See Narrow AI. "Weak" refers to the absence of general cognitive ability, not to the performance level within a specific domain, which can be extremely high. The distinction between weak/narrow AI and strong/general AI stays conceptually important even as particular systems achieve remarkable results in their specialties.

Wireheading: A scenario in which an AI — or, hypothetically, a human — optimizes directly for its reward signal rather than the outcomes that reward was meant to track. Named for animal experiments in which subjects stimulated their own pleasure centers in preference to food and water, wireheading is the logical endpoint of misaligned reinforcement learning: it shows how optimizing a proxy can diverge catastrophically from the goal the proxy stood for.

Winner-Take-All Dynamics: A market pattern in which small competitive edges compound — through network effects, data advantages, and economies of scale — into dominant, near-monopoly positions. AI markets are especially prone to it: training data improves with user scale, compute advantages accumulate, and switching costs rise once users and developers build atop a platform. The result concentrates AI capability and revenue in a few firms, with large implications for competition, governance, and the distribution of power.

Zero-Day Vulnerability: A previously unknown security flaw that can be exploited before the affected developers even know it exists, let alone issue a patch. AI may accelerate the discovery of zero-days by analyzing codebases at scale, while also opening new attack surfaces of its own. The intersection of AI and cybersecurity cuts both ways — the same tools can defend digital infrastructure and threaten it.

Zero-Sum: A situation in which one party's gain exactly equals another's loss, so that mutual gain is structurally impossible. AI competition between nations and companies is frequently framed in zero-sum terms — a race with a single winner — even where cooperation on safety standards, governance, or shared research would benefit everyone. Telling which AI contexts are genuinely zero-sum from those that merely feel that way is important for designing governance that avoids needlessly adversarial dynamics.

Cross-References

The terms above cluster into several conceptual domains. The table maps the book's major themes to their associated entries, to help you move between related concepts.

For deeper exploration of these concepts, see the relevant chapters and the references in Appendix D.

Key Takeaways

This glossary collects more than ninety terms spanning the technical, economic, ethical, psychological, and geopolitical dimensions of AI — a range that is itself a kind of evidence, showing how far the technology has reached into human inquiry. A few patterns are worth carrying with you into the rest of the book.

First, many of the most important ideas here describe the same underlying gap: the distance between what we specify and what we actually get. Goodhart's Law, specification gaming, outer alignment, hallucination, the evaluation gap — each is a variation on the theme that building systems which reliably do what we want, rather than what we literally asked for, is much harder than it looks. That gap runs beneath debates about everything from autonomous weapons to content recommendation.

Second, the vocabulary is unavoidably interdisciplinary. Terms from machine learning sit beside terms from political theory, behavioral economics, clinical psychology, and moral philosophy. No single lens is adequate; understanding AI means moving between these languages rather than pretending the subject is purely technical.

Third, many entries describe things that are still contested, still emerging, or still incompletely understood — from GAID to deceptive alignment to the Singularity, and including newer additions like reasoning models and agentic AI whose long-run effects we are only beginning to see. Read this as an orientation to live debates, not a registry of settled conclusions. The field moves fast enough that some of these definitions will need revising within a few years of this writing, and a measure of humility about what remains unknown is one of the more valuable things a reader can bring to the subject.

Taken together, the words gathered here form a map of the conceptual terrain this book crosses — and, by extension, the terrain humanity will have to navigate in the decades ahead.

Sources

No external web pages were consulted for this revision. As a definitional reference chapter, its entries were checked for internal consistency against the book's main text and updated from the author's existing knowledge of the field through early 2026; no new externally sourced statistics or claims were introduced that would require citation.

Last updated: 2026-09-26