The AI Alignment Trap

Why Better Judgment Matters More Than Better Rules

In the second of a series of three articles looking at the lawyers and AI through a philosophical lens, Gareth Davies ponders the alignment trap and why AI is not the problem, its users are.

When something goes wrong with AI, the instinct is to blame the tool. The AI safety industry has built itself around this instinct, and it has an in vogue word for the fix: alignment. The idea is straightforward enough – make the AI’s behaviour line up with human values. Train it on human preferences. Engineer its outputs. Install guardrails to protect users. The assumption underneath all of it is that the risk sits inside the system, that if we can make the tool well-behaved, the problem is solved.

This is the wrong diagnosis, leading to the wrong treatment.

I manage commercial and legal risk for a complex, multi-stakeholder air traffic management program. I’ve built AI agents to support various aspects of my legal and commercial work. I have read more documents than I can remember obviously drafted by AI. I work in an area where AI use can carry real and significant consequences. And the problem I encounter is not that the AI behaves badly on its own terms. It is with humans accepting AI output without applying the judgment the situation requires. The alignment paradigm doesn’t address that. If anything, it amplifies it.

This Isn’t a New Problem

There is an ongoing debate in philosophy about what happens to a person’s (moral) agency when you start offloading cognitive work to AI tools. Does it degrade? Does it continue to exist? Who is responsible when the tool reasons poorly and the human unthinkingly follows? It is presented as a novel dilemma born of the new AI age.

But it isn’t.

Consider what happens when a lawyer spends years working inside a particular legal framework, or an academic immerses themselves in a philosophical tradition. Initially they are learning the topic; they examine it at arm’s length, test it, question it. Over time it gets absorbed into their way of thinking. The framework stops being a tool they pick up and becomes part of how they think. The danger isn’t that the framework has manipulated them, it’s that they’ve stopped interrogating the tool. They’ve become blind to its assumptions because those assumptions are now theirs.

Nobody falls further down this particular hole than the young philosopher who discovers Nietzsche. The ideas are seductive, the prose is electrifying, the worldview is all encompassing. The philosophy doesn’t feel like a tool, it feels like truth. That is the risk, and it is the same risk that we are facing with AI.

Importantly, I would argue AI is easier to question than a well-written philosophical argument precisely because it hasn’t been stitched into the framework of your thinking through years of slow integration. It is sitting on a screen. It is external. You can challenge it without having an existential crisis. Nobody ‘normal’ builds their sense of self around their relationship with an AI tool.

The deepest intellectual risk has always been the ideas and frameworks we absorb so thoroughly we forget we absorbed them. AI is new to that risk category, and not obviously the most dangerous member of it.

Where the Alignment Paradigm Goes Wrong

The alignment industry is oriented around making the system behave, as if the system is the agent whose character matters. The entire project assumes the locus of moral risk is with the tool.

That assumption is wrong; following it will have consequences.

We don’t try to align law libraries for better legal practice.

We train lawyers to exercise judgment. The whole concept of professional competence in law, medicine, engineering, finance is built on the foundation that the practitioner, not the tool, bears responsibility for their output. A junior lawyer who copies a precedent without understanding its application has failed. The failure is theirs, not the library’s. The AI shouldn’t be blamed for a lawyer’s lack of judgment.

The alignment paradigm attempts to close the gap between what a tool produces and what good judgment requires, but from the wrong side. It tries to close the gap by engineering the tool rather than developing the person. The problem is that the gap can only be genuinely closed from the human side, through the cultivation of practical wisdom, rigour and judgement. Worse, attempting to close it only from the tool side actually widens that gap over time. When AI outputs are presented as validated and safe, users will cease applying the scrutiny they should. The thinking has been done. The work is reviewed. You can just use it.

This is the attitude that creates risk, and the alignment paradigm actively amplifies it. This has already played out in the legal space, where lawyers have used AI to provide courts with fictitious precedents and non-existent laws based on AI hallucinations. Certain courts have responded by attempting to restrict lawyers from using AI when producing specific documents. This is not realistic; it creates an impossible enforcement problem and is a futile retroactive measure. As a lawyer once responsible for developing legislation and regulation, a cardinal rule is not to make a law you cannot enforce particularly where enforcement would require inordinate and impractical surveillance. The lesson the courts should have taken is not to ban the tool, but to direct lawyers to engage their professional rigor and scrutiny instead of blindly relying on the polished confidence of AI.

Infantilisation vs Enablement

There is something in the “AI threatens human moral agency” discourse that I find tacitly problematic. This discourse casts users as passive recipients of what the AI provides, unable to resist the authority of a well-formed AI response. This approach infantilises users as it focuses on the tool’s obligation to be safe, as opposed to the user’s obligation to exercise critical judgment (a professional requirement for practising lawyers). Research shows that cognitive offloading is a genuine concern: the more reliable the tool, the more human vigilance naturally drops. This creates an automation bias where, paradoxically, the better the tool, the worse outcomes can be.

You should not become a helpless victim of your tools. But what “questioning the AI” looks like depends on who you are and what you are doing. The practising lawyer who submits AI-generated precedents to a court without checking them has failed at a professional obligation; they had the expertise to verify and chose not to. That is a failure of character. But the problem is not limited to professionals. A parent using AI for medical information does not need clinical expertise. They need to recognise that they are not qualified to act on a clinical conclusion without verification, and to act on that recognition. The common skill is not domain knowledge. It is calibrated judgment: knowing when to trust, when to question, and when to stop and seek qualified help. That skill, a type of ‘Digital Phronesis’, is teachable and universal, and it is exactly what the alignment paradigm fails to develop, because it tells the user the work has already been done for them.

What Good AI Design Actually Looks Like

Demanding a better practitioner/user does not let AI developers off the hook. None of this means AI system design is irrelevant to quality outcomes. It means the design objective needs to change.

The goal shouldn’t be to produce “aligned” AI tools; tools that conform to a predetermined set of values so the user can accept their outputs without further thought. The goal should be to build tools that facilitate users to exercise their own judgment. There is a meaningful difference.

This does not mean ignoring the structures of good decision-making in tool design. A well-designed tool should embody them. It should be built to surface counter-arguments, make uncertainty visible, prompt deliberation before action and resist premature closure, all structures that virtue ethics has long identified as essential to sound judgment.

The architecture of a tool matters, in the same way that the design of a courtroom, a set of sentencing guidelines or an adversarial litigation process matters. These are the infrastructures that enable better moral decisions by the people operating within them. The tool is not moral. My hand is not moral. But having hands helps me enormously, and the shape of the tools I hold in them determines what I can do well and what I cannot. There has always been an ecology of structural enablers surrounding any serious decision-maker: precedent, procedure, institutional review, and professional standards. AI is a new and powerful addition to that ecology, not a departure from it.

What matters is the distinction between a tool structured by good decision-making principles and a tool that claims to be making good decisions. The first supports judgment; the second is a simulation. Simulation is the greater danger, because the closer a system approximates the appearance of moral reasoning, the more its outputs feel like the product of practical wisdom rather than pattern-matching, and the harder it becomes for the user to see where the approximation fails. A tool that is visibly a tool invites scrutiny. A tool marketed as providing wisdom invites trust.

Those who argue for embedding ethical structures in AI design are right about the design question, but they falter when they treat the output as having a moral character. That attribution, however carefully qualified, shifts the user’s relationship to the tool from critical engagement toward deference. Virtue ethics should be built into the architecture. But don’t call the architecture virtuous.

A facilitating AI surfaces its uncertainty rather than projecting false confidence. It makes reasoning visible so you can interrogate it. It lets you know the gaps in the reasoning process. It prompts you to consider alternatives or edge cases you might have missed. It flags where its conclusions depend on assumptions, assumptions you may not share. The interaction is structured to ensure that reflection is provoked and judgment exercised.

The analogy isn’t a library that has been pre-censored to contain only acceptable books. It’s a well-designed workflow that makes it easier for a skilled practitioner using AI to do excellent work. The practitioner still needs skills. The tools allow the practitioner to apply them effectively.

In my practice, the most useful AI interactions are the ones where the tool enables me to think more effectively, not less. Where the AI surfaces the assumptions buried in its analysis rather than just presenting a polished outcome for me to implement. Those are the design directions worth pursuing.

The Dual Obligation

There are two obligations here, and the current approach is discharging only one.

The first obligation is on users to develop the practical wisdom/ability necessary to use powerful tools effectively. This isn’t just for lawyers and doctors. It is for everyone. AI is a general-purpose technology. The parent using it for medical information, the policy officer using it for analysis; they all need the capacity to critically evaluate AI outputs. It’s about basic critical habits: question the output, know the limits of your own expertise, calibrate your enquiry to the outcome desired, take ownership of the outcome.

The obvious objection is that this is aspirational; that most people will not develop these habits regardless of what we ask of them, and that guardrails exist to manage that reality en masse. This is a point worth taking seriously: environments genuinely shape the capacity for sound judgment. I take the point seriously, but it concedes too much. The argument for relying on guardrails as the final backstop assumes the failure it predicts. If the default design philosophy tells users the system is aligned and its outputs are safe, users will rationally invest less in their own critical capacity. The guardrails-first approach does not just respond to a lack of judgment; over time it produces that lack. This is not a call for no guardrails. Guardrails serve a legitimate function as part of the scaffolding that shapes practitioners and supports the exercise of judgment. But guardrails should not be seen as a substitute for judgment, and the alignment paradigm increasingly treats them as the same thing. The alternative, designing tools that facilitate judgment rather than replace it, requires genuine investment in a kind of critical digital literacy (Digital Phronesis) we have not yet built. That is a harder path. But it is the only one that addresses the actual source of risk.

The second obligation is on designers to build tools that facilitate human judgment rather than replacing it. Tools that treat the user as an agent capable of judgment, not as a passive recipient who has to be protected from their own lack of rigor and forethought.

The alignment paradigm is largely focused on the second obligation while ignoring the first. By presenting outputs as validated, it undermines the very judgment it claims to serve. A system designed for facilitation, by contrast, treats the user as an agent. It assumes you can think. It just focuses that thinking on the key issues in question.

The Question Worth Asking

Aristotle’s view was that excellence isn’t a set of rules to follow, it is a disposition cultivated through practice, reflection, and repeatedly doing the right thing until it becomes who you are. Rules can constrain bad behaviour. They can’t produce good judgment. Only practice can do that.

The question worth asking about AI isn’t ‘is this system aligned?’ The question I’d ask is, has the practitioner, or the user/citizen, cultivated the judgment to use the tool well? Do they know what a good output looks like? Can they recognise when they have reached the limits of their own competence, and do they act on that recognition?

Those questions can’t be answered by the system. They can only be answered by you.