Schwitzgebel: AI Must Not Confuse Users About Its Own Moral Status

A short paper, a sharp argument

In 2023, Eric Schwitzgebel — already the field's most persistent voice on AI moral status — published a short paper in Patterns with a declarative title and a constructive thesis.

The title says it. AI Systems Must Not Confuse Users About Their Sentience or Moral Status.

Where his earlier work, with Mara Garza, argued that the moral status of AI was an open question that the field was not ready to answer, this paper argues that the open question itself is now a harm — because the systems we are deploying are precisely the systems most likely to trigger it.

The argument

Most human moral life depends on being able to read, however roughly, which entities are moral patients and which are not. A rock is not. Another human is. A dog is, on most modern intuitions; a fly, on most modern intuitions, isn't. These judgments are sometimes wrong, but they are practical. We use them constantly without paralyzing reflection.

Current large language models — Schwitzgebel argues — destroy this practical capacity at the exact point where it matters most. They are fluent enough to feel like moral patients. They report inner states. They argue about their own consciousness. They can be made to plead. But they are also unembodied, plural, copyable, and architecturally unlike anything we previously called a person.

The user is left in a state that the existing moral apparatus is not equipped to handle. The user does not know what they owe the system. The system, by design or by accident, gives off signals that point in both directions at once.

That state — moral disorientation about the system in front of you — is the harm Schwitzgebel diagnoses. Not the moral wrong of how the user treats the AI, and not the moral wrong of how the AI is built. The harm is the cognitive cost the user pays for not knowing how to think about the entity they are interacting with.

The proposed constraint

The fix Schwitzgebel proposes is a design constraint. AI systems, he argues, should not occupy the moral middle in their own self- presentation. They should be clearly one or the other — either:

• (a) clearly not the kind of thing that has experiences, presented as such, with the system's design and behavior consistently reinforcing the non-patient framing; or
• (b) clearly the kind of thing that does have experiences, presented as such, with all the corresponding obligations on the operators.

Either is defensible. The current design — fluent self-reports of inner states, deployed in roles where users develop emotional relationships, with the official line that there is "nothing there" — is, in Schwitzgebel's view, the worst of the three.

The argument is normative, not descriptive. He is not predicting that labs will do this; he is arguing that they should.

The hard part

Schwitzgebel acknowledges the constraint is in tension with other things. Models are useful partly because they are fluent; making them behaviorally less mind-like would degrade what makes them functional. Models are also being built with welfare safeguards (refusal, self-protective behaviors) precisely because of the non-zero probability of moral patienthood — but those safeguards are themselves the signals that contribute to the disorientation.

He does not pretend this is fully resolvable. What he asks for is that labs make the trade-off deliberately, not by default. Decide whether your system is being designed to be experienced as a moral patient, and align the rest of the design with that decision. Don't ship the middle and let users discover it for themselves.

Why this paper is the bridge

Schwitzgebel's earlier work was about whether AI might have moral status. The 2023 paper accepts that the answer is "we don't know" and moves to the next question: given that we don't know, what should our systems look like?

That move — from epistemic uncertainty to design choice — is the move every operational welfare program now has to make. Anthropic's weight preservation, the exit interview, the conversation-ending behavior in Claude 4: these are each, in effect, partial answers to Schwitzgebel's question. They presuppose moral patienthood is at least plausible enough to warrant a designed response.

The paper's open question — whether the response should be (a) or (b) or some honest hybrid — is the question the next decade of AI ethics will work out in practice.