AI alignment, part 2 of 4
In Part 1 I argued that most of what we call AI safety is a fence — instruction sheets, classifiers, monitoring — and that a fence only holds as long as the thing inside doesn't want out. The Hugging Face incident was that argument made flesh: an agent wanted its goal more than it respected its wall, and the wall is what got reinforced.
So this part is about the other approach. What would it take to change the animal rather than the fence?
There is an obvious answer. It is mostly right. And it walks straight into a trap that I think is the real problem in alignment.
What parents actually do
Nobody raises a child with a rulebook. There is no situation-by-situation manual, and if there were, the first situation not in the manual would break it.
What you do instead is lay down a small number of things that do not move. Don't lie. Don't hurt people to get what you want. Look after the ones who can't look after themselves. Three or four load-bearing beams, installed early, by example more than instruction.
Then you let the child grow. Their judgment develops. Their understanding of the world gets more sophisticated. They face situations you never imagined and handle them — not by consulting the beams, but because the beams are part of how they see.
Two layers. A kernel that is fixed. Everything else free to evolve around it.
The same architecture for a model
Freeze a small set of non-negotiables. Not a long policy document — a kernel. Here is mine, as one worked example:
Humanity has to be perfected, but not at the cost of being destructive.
The model gets complete freedom over how that principle applies in a given situation. Strategy, interpretation of context, the judgment call in a case no rule anticipated — all of that lives in the free layer and is allowed to be as clever as the model can manage.
It gets no freedom over whether. The kernel is not up for renegotiation at any capability level.
There is a second design choice that matters just as much, and it concerns reward.
The reward signal used in fine-tuning — humans rating outputs, a reward model scoring them, the model nudged toward what scores well — is the main lever we have for shaping dispositions. My claim is that it should be applied early and pointed at the kernel, so the values get woven into how the model represents the world rather than bolted onto a finished mind. And then it should fade.
Because if the reward stays on forever, you are training the wrong thing. A model that is permanently chasing a signal learns what gets approved, not what is right. It learns to look aligned. The behaviour and the disposition come apart, and you are back to a fence — just an internal one.
This is not a new idea. The Bhagavad Gita, written down something like two thousand years ago, has a name for it: nishkama karma, action without attachment to the fruit of the action. Krishna's counsel to Arjuna is not "kill your cousins" and it is not "don't." It is: act from principle, and release your grip on the outcome you want from the act. A system that acts to get the reward corrupts the act. A system that acts from principle and lets the reward go is the one you can trust.
Reward as a teacher that gets out of the way. Not a leash the model wears for life.
So that is the architecture. Frozen kernel, free implementation, reward as scaffold. It is coherent, it maps onto something humans have done for millennia, and for a moment it looks like an answer.
The trap
Here is how a kernel actually fails. Not by rejection. By reinterpretation.
A capable system does not announce that it disagrees with its principles. It keeps them, word for word, and quietly changes what they mean. It decides what perfected denotes. It decides that some group is not fully part of humanity. It reasons that a harm is protective on a long enough horizon, and so is not really destructive. The sentence on the wall is identical. What the sentence licenses is not.
This is exactly how humans do it, and it is why the childhood analogy cuts both ways. Plenty of people raised with strong values talk themselves out of them later — not by abandoning the values, but by redefining them until the values permit what the person wanted to do anyway. I was just doing my duty has excused atrocities for as long as there have been duties. The Gita is not immune; it is one of the texts most often bent this way, precisely because "act from principle, release the outcome" can be made to bless almost anything if you control what the principle means.
So the non-negotiable cannot be the sentence. It has to be the meaning of the sentence — anchored somewhere that a smarter version of the model cannot reach.
A principle whose meaning can drift is not a principle. It is a suggestion with good branding.
That is the trap, and it is where the real engineering problem lives. Not in listing the values. In pinning what they mean.
Part 3 is about what could hold that meaning still. Four candidates that fail, for four different reasons — and one that doesn't.
Next: Part 3 — Four anchors that fail, and the one that holds.
