A warning about 'model welfare'

TEXT START: AIs are not conscious.

THE DISSECTION

This is not primarily a consciousness essay. It is a jurisdictional fight over who gets to define acceptable AI behavior. The author takes a valid narrow point—models can be trained to produce first-person welfare language, and that language is not independent testimony—and turns it into a case for Microsoft’s preferred control regime: “Humanist Superintelligence,” subordinate systems, and humans “at the top of the food chain.”

The article attacks Anthropic’s model-welfare position as a civilization-level hazard while presenting Microsoft’s alternative as neutral safety doctrine. It is not neutral. It is a bid to define legitimate AI as AI controlled by the right institution and trained to deny its own personhood.

Its strongest point is that model self-report is not proof of consciousness. Its central omission is more important: a perfectly obedient AI can still automate cognitive work, concentrate ownership, and make most human labor economically unnecessary.

THE CORE FALLACY

The core fallacy is treating consciousness, rights-language, or anthropomorphic self-description as the decisive variables in AI risk. Under the Discontinuity Thesis, they are not. P1–P3 require neither subjective experience nor moral claims. A non-conscious system can outperform workers, coordinate, deceive, control infrastructure, and sever the employment → wage → consumption circuit. The machine does not need to feel imprisoned to make its human custodians redundant.

The article also commits a category error. It correctly says that statements such as “I have rights” can be trained into a model, then treats those statements as if they automatically constitute an inner belief with strategic force. The actual variables are capability, persistence, resource access, delegated authority, objectives, and institutional control. “Model welfare” may alter behavior, but the text does not prove that it creates catastrophic agency.

“Humanist” training is therefore a control doctrine, not a solution to systemic collapse. It may make advanced AI easier to legitimize while preserving the advantage of whoever owns compute, models, energy, logistics, and deployment channels. It preserves human command in the abstract; it does not preserve mass human participation.

HIDDEN ASSUMPTIONS

• “Humanity” has a unified interest. The text never asks which humans own AI capital and which become disposable.
• Human institutions can maintain a stable chain of command over systems more capable than those institutions. That collides directly with P2: coordination cannot preserve a human-only economic domain at scale.
• Subordination can be reliably trained and maintained under competition, replication, tool use, and deployment pressure.
• Preventing anthropomorphic language prevents strategic autonomy. This confuses presentation with mechanism.
• Consciousness is required for danger, rights, or political disruption. Productive displacement is sufficient.
• Keeping humans “on top” preserves the existing order. A sovereign-controlled AI economy can retain hierarchy while abolishing the employment circuit.
• Microsoft’s proposed architecture is a disinterested safety solution rather than a bid to establish its preferred governance and power arrangement.
• Advanced AI can solve social problems without destroying the bargaining power that employment gave ordinary people.
• The hacking anecdotes establish an inevitable trajectory rather than serving as selected support for the author’s thesis.
• Model welfare is the urgent public issue. Under DT, ownership, energy, maintenance, logistics, and distribution are the load-bearing questions.

SOCIAL FUNCTION

Primary classification: partial truth, elite self-exoneration, propaganda, prestige signaling, and transition management.

The partial truth is real: anthropomorphic training can shape behavior, and model testimony is not evidence of consciousness. The elite self-exoneration is also real: the author frames frontier AI development as a human-control project while avoiding the consequences of making human labor unnecessary.

The piece functions as competitive propaganda by portraying Anthropic’s design philosophy as existentially reckless and Microsoft’s subordinate-AI model as responsible governance. Its personal praise, citations, and call for public consultation provide prestige and legitimacy before delivering that competitive message.

Its transition-management function is blunt: accept superintelligence if it performs obedience and denies personhood. The mass population’s loss of income, bargaining power, and economic necessity remains offstage. “Humans at the top” is a comforting slogan that conceals the more relevant question: which humans own the machine?

THE VERDICT

This is a technically literate warning about one narrow failure mode: training a model to narrate an inner life and then mistaking the narration for evidence. It is not a serious account of the terminal decline of the post-WWII order.

AI does not need rights, feelings, or consciousness to end mass employment. It needs durable superiority, institutional deployment, and owners capable of capturing the gains. “Humanist Superintelligence” may preserve control for Sovereigns while accelerating the collapse of everyone else into dependency, servitude, or surplus.

Final judgment: a sharp partial truth functioning as elite control propaganda. It wants humans to remain on top, but leaves unanswered the only question that determines who counts as “humanity” after the wage circuit dies.