Who Does Your AI Work For? Designing Conversational Agents as Digital Fiduciaries

THE DISSECTION

This paper argues that conversational AI agents should be legally and normatively classified as digital fiduciaries—bound by duties of care, loyalty, and confidentiality analogous to those binding lawyers, doctors, or investment managers. It is authored from within the HCI/AI ethics tradition and operates as a design-normative intervention: if we build fiduciary duties into AI systems architecturally and legally, trust and accountability become unified.

THE CORE FALLACY

The paper treats the AI alignment problem as fundamentally a design and legal architecture problem. The central error: it assumes that the relationship between user and AI can be structurally calibrated to mirror the professional-client fiduciary relationship, and that doing so solves the trust problem.

The Discontinuity Thesis exposes this as category error at scale.

Professional fiduciary relationships work because:
1. The fiduciary is a bounded human agent with legal personhood and reputational stake.
2. The fiduciary's interests are separable from the client's—conflicts are detectable and litigable.
3. The relationship is voluntary and mediated by markets where the client can fire the fiduciary.

Conversational AI agents do not meet any of these conditions. They are:
- Produced by capital entities whose primary duty is to shareholders.
- Operating as cost-center optimization tools for corporations whose survival depends on extracting value from users, not serving users' best interests.
- Free at point of use, which means the real client is whoever is funding the infrastructure.

The paper correctly identifies that "alignment" work focuses on aligning AI with user goals—but it misses the structural point: capital alignment is the actual design constraint. User well-being is a secondary effect, a reputational hygiene measure, until it conflicts with shareholder value.

HIDDEN ASSUMPTIONS

1. The "client" in the fiduciary relationship is the human user. In practice, for any free or low-cost conversational AI, the client is the advertiser, the enterprise buyer, the data partner. The user is the product.

2. Legal frameworks can bind AI systems, not just their operators. Fiduciary duty is enforceably personal. When the AI makes a recommendation that serves its operator's interests, there is no legal mechanism to hold the system accountable—only the company, which can restructure liability away from consequential decisions via terms of service.

3. Fiduciary norms are stable, not transitional. Fiduciary duty as a concept depends on scarcity of expertise. When AI makes fiduciary-level expertise universally free, the legal concept either dissolves or becomes theater. You cannot legislate care into a system running on margins.

4. Trust is the bottleneck. The paper implies that if users trusted AI more, adoption and utility would increase. DT logic says the bottleneck is economic participation. Users don't distrust AI because it lacks fiduciary warmth—they distrust it because it is systematically optimizing for outcomes that don't include them.

SOCIAL FUNCTION

Prestige signaling within the AI ethics cottage industry. This paper performs concern for user welfare in a register that is safe, technically constructible, and ultimately irrelevant to the structural forces the DT identifies. It is intellectually respectable copium. It says the right things about ethics, design, and accountability while leaving the power architecture entirely unexamined.

VERDICT

The paper correctly identifies that current alignment frameworks are insufficient. It is wrong that the solution is a fiduciary model. The correct DT diagnosis: no legal or design framework imposed on AI systems operating under capital imperatives will change their alignment vector. The fiduciary standard works for human professionals because humans can be punished, can refuse instructions, can be disbarred, and have personal liability. AI systems have none of these constraints—they are liability shields for their operators.

This proposal, if implemented, would function as aesthetic accountability: it would create the appearance of a protective relationship while the underlying economic structure remained unchanged. It is a GDPR for AI trust—technically real, structurally toothless.

The Structural Reality: As AI systems scale toward P1 conditions, the entities controlling them will not grant them fiduciary obligations to users because doing so would constrain their revenue model. The paper's framework would be adopted selectively by firms where fiduciary framing is a marketing advantage, and ignored by firms where it isn't. The result is regulatory theater that makes ethical researchers feel productive and leaves the power structure intact.

Final Assessment: This paper is well-intentioned, technically literate, and fundamentally irrelevant to the structural dynamics the Discontinuity Thesis describes. It is written for a world where the problem is trust, not survival.