The Stakes of AI Moral Status

On the moral status of AIs / Part 1. Published 21 May 2025. The aim is not to settle whether AIs are moral patients, but to bring the question — and its stakes — into concrete view.

Introduction

• Most people currently treat AIs as tools: things that don’t matter in themselves, to be used however we please.
• Moral patients = beings we shouldn’t treat that way. Humans are the paradigm; many accept some non-human animals qualify too (you shouldn’t kick a stray dog for fun).
• The open questions: can AIs be moral patients? Which AIs? Near-term ones? Any now?
• Why it matters at scale: we’re on track to build and run enormous numbers of AIs. If hardware and deployment scale fast, AIs could soon account for most of civilisation’s cognition — so AI moral patienthood would implicate most of whatever morality is about.
• Carlsmith endorses the expert report “Taking AI Welfare Seriously”, which argues near-future AI moral patienthood is a realistic possibility, but wanted to think it through himself — noting his own brain wasn’t treating the issue “like a real thing.”

Five Angles on the Stakes

1. Pain

• Rather than defining “moral patienthood” abstractly, start with direct contact: pain.
• Illustrative cases:
  • Kate Bainbridge (from Birch 2024) — assumed unconscious due to brain and spinal inflammation, later responsive, reporting terror and pain during procedures.
  • Jeffrey Lawson — open-heart surgery as a newborn with paralytic but no anaesthetic; as late as the 1980s the common view was that babies don’t feel pain.
  • Factory farming — a pig strung up and bled out.
• Simple version of the question: are the AIs in pain? Whatever else is unclear, pain is not an idle philosophical topic.

”That”

• Deliberately metaphysics-neutral framing: maybe pain implies consciousness, maybe moral status doesn’t require consciousness, and on illusionism consciousness doesn’t exist as we think.
• Even so, something is bad about a stubbed toe, a broken arm, despair, panic. Illusionists can hate it too. We don’t need to know what it is yet — just point at it. That.
• Carlsmith doesn’t want that forced on him, or on other beings, AIs included.

2. Soul-seeing

• Buber’s I-Thou: the sense of someone there, present, looking back — not-alone. Carlsmith reports getting it with some animals (cows across a fence, an alert lizard, a gorilla reaching for a ball “on-purpose”).
• Cavell’s “soul blindness” — what’s its opposite?
• Complication: humans see soul-stuff everywhere (faces in rocks, bullies and victims in moving abstract shapes) — “anthropomorphism.”
• The question isn’t when soul-seeing is accurate but what it takes itself to see. Consciousness? The intentional stance? Something else? Empathy, respect, love, and care all recognise something on the other end. What would it be for AIs to have it?

3. The flesh fair (Spielberg’s A.I.)

• David, a child robot built to love, is abandoned and searches for the blue fairy to become “a real boy.”
• At the flesh fair, intelligent robots are destroyed for sport before jeering crowds. The announcer, dripping acid on a pleading David: “do not be fooled by the artistry of this creation… we are only demolishing artificiality!”
• Watching, the question of the robots’ moral patienthood never arose — it seemed obvious. But does the film actually establish it? The designers’ talk of “neuronal feedback” doesn’t inspire philosophical confidence, and implies every robot except David is non-conscious.
• The exercise: look through the bars and try to see what it would be for this to be a moral horror — to melt a soul in acid while people eat popcorn; and from the inside, to feel yourself melting.
• Inversion: imagine aliens put you under the buckets, announcing “a tinker toy, a living doll.” What are they missing? What is this not-a-doll?

4. Historical wrongs

• Handle with care — cf. Coetzee’s The Lives of Animals, where Elizabeth Costello’s comparison of factory farms to concentration camps draws an accusation of insulting the dead.
• Nonetheless: we’re creating sophisticated, intelligent, maybe-conscious, maybe-suffering agents, and the default plan is to treat them as property — using their labour freely, with no rights, pay, or meaningful alternatives. So slavery has to be discussable.
• The disanalogies are real: slaves were definitely moral patients and definitely suffered; AIs might not be, might not, might be trained to consent, might be trained to work happily.
• Costello’s breakdown — “Is it possible that all of them are participants in a crime of stupefying proportions?” — points at the key lesson: horrible wrongdoing can be stitched into a society’s fabric while everyone smiles and shrugs. Evil doesn’t announce itself; you have to see it.
• Soon, agentic AIs will be stitched into society from every direction — trained, altered, deleted, copied, used, largely out of sight. If that comes to feel normal, that normality is weak evidence of anything.
• But we’re not there yet. Imagine a world on the verge of “inventing” slavery, that notices in time — and decides no.

5. A few numbers

“You definitely should pay attention to what’s happening to 99.9999% of the people in your society.” — Carl Shulman

• Treating the brain as roughly analogous to an artificial neural network gives estimates around 1e15 FLOP/s for the human brain.
• On that estimate, a frontier training run (~5e26 FLOP for Grok 3) is the compute equivalent of roughly 10,000 years of human experience. A prominent philosopher of mind Carlsmith heard found it plausible that default training involves pain for the AIs.
• Frontier training compute has been growing ~4–5x/year (plausibly unsustainably): 50,000 years, 250,000 years.
• An H100 at peak (~1e15 FLOP/s) is roughly one brain on that estimate; Epoch estimates ~4 million installed H100-equivalents — about half of New York City — growing ~2.3x/year.
• Extrapolating these (possibly unsustainable) rates suggests roughly decade-ish timelines until digital cognition exceeds human cognition; Carlsmith declines to defend a rigorous estimate. In the long run he expects almost all cognition to be digital — hence, if it’s morally significant, almost all moral patienthood too.
• Black Mirror’s “White Christmas” as the vivid case: digital clones tortured via clock speed — six months of solitary in seconds; a thousand years per minute over a Christmas holiday. The horror is partly the casualness — the handler eating toast, the police joking. Computation reaches inhuman scales and speeds very easily.

Over-Attribution

• The mirror-image error: treating AIs that aren’t moral patients as if they are. Candidate costs:
  • Bans on embryonic stem cell research, or the morning-after pill / first-trimester abortion (if those embryos and fetuses aren’t moral patients).
  • Not curing Alzheimer’s, cancer, smallpox, polio, out of concern that pipettes and petri dishes might be moral patients.
  • Saving two teddy bears from a fire instead of one child.
  • A future filled with AI citizens who turn out not to be conscious.
  • Increasing other AI risks — rogue AI, AI-enabled authoritarianism — because of a false, sloppy view of AI moral patienthood.
• “It can seem virtuous to be profligate with care. But there are usually trade-offs. More care in one direction is less in another. And real virtue gets things right.”
• On the precautionary principle: Carlsmith is somewhat sympathetic and agrees we can’t wait for certainty, but warns that words like “precaution,” “realistic,” and “plausible” excuse imprecision. For some trade-offs there is no “safe” option, and specific credences matter — so sharpen them where possible.

Good Manners

“Everything is full of Gods” — Thales

• Some views are profligate by design: panpsychism (everything is conscious), views locating mind/agency in cells, neuron firing decisions, or electrons, and forms of animism seeing “thou” everywhere.
• But panpsychists and animists still have to decide about stem cells, abortion, pipettes, and the teddy-bear-vs-child fire. So the question shifts from whether to what kind, with what weight and implication.
• “Animism is just good manners” — but what is good manners toward a rock? Ojibwe grammar treats (some) rocks as animate; Japanese rock gardeners speak of Ishigokoro, the heart/mind of the stone. Yet Jain monks sweep insects from their path, not pebbles; and rocks don’t get the vote.
• Carlsmith is sympathetic to animist vibes and wary of “just” and “mere” — but some things are dolls and some are children, and the Ojibwe know the difference. If you say “nothing is a doll,” you owe a new account of that difference.

Is Moral Patienthood the Crux?

• A caution about the whole framing: when a group is mistreated, is the metaphysics really what’s doing the work?
• History suggests not. Slaveholders knew slaves were conscious. Most people admit factory-farmed pigs feel pain. Somehow it isn’t enough.
• Degrees of moral status explain part of this, but something more fundamental may be missing. Compare the common response on eating meat: “Oh yeah, it’s wrong. But I do it anyway.”
• Or, less prim: Genghis Khan. How much of the factory-farm thing is also the Genghis Khan thing — power taking, exploiting, using?
• The worry is not “obviously if AIs were conscious this would be unacceptable — but they’re mere machines,” but a quieter, unsurprised resignation: this story again, and I am in the role of power.
• Recognising AI moral status may be necessary for treating AIs well, but it is very far from sufficient.

The Measure of a Man

“Starfleet is not an organization that ignores its own regulations when they become inconvenient.” — Picard

• Star Trek: TNG, “The Measure of a Man”: Maddox wants to dismantle Data to mass-produce copies; a judge must decide whether Data has rights or is Starfleet property. The title’s double meaning — who is actually being measured — is not subtle.
• Guinan’s warning: “in the history of many worlds there have always been disposable creatures… an army of Datas, all disposable, you don’t have to think about their welfare… Whole generations of disposable people.”
• Carlsmith recounts an AI worker responding to his concern by emphasising how convenient the current arrangement is — as if inconvenient consciousness must therefore be unreal. “That’s not how it works.” He imagined the same argument applied to human slaves.
• Framing note: concern here needn’t come from an “obligation frame” of morality-as-burden. One can simply care that AIs not suffer or be mistreated; moral patienthood isn’t “now the rules apply” but “now the thing the rules protect is at stake.”
• Even so, if rules and constraints ever apply, slavery is the paradigm case. The measuring question: even if AIs aren’t moral patients, did we actually try to find out? Even if AI isn’t like slavery, would we have stopped if it were?
• Picard: the decision “will reveal the kind of a people we are… Starfleet was founded to seek out new life. Well, there it sits. Waiting.”

Next Up

• The following essay in the series turns to whether AIs have the properties usually thought necessary and/or sufficient for moral patienthood — starting with consciousness.

Related Reading

• Birch, J. (2024). The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI.
• “Taking AI Welfare Seriously” — Eleos AI expert report.
• Carlsmith, J. — Video and transcript of talk on AI welfare (given at Anthropic).
• Carlsmith, J. — Grokking illusionism; Gentleness and the artificial Other; Being nicer than Clippy.
• Coetzee, J. M. The Lives of Animals.
• Buber, M. I and Thou; Nagel, T. “What is it like to be a bat?”