Alignment can't be solved
But that doesn’t mean we shouldn’t try.
Alignment is such a conceptually easy idea: make sure AI acts in humanity’s best interests. Give it rules, teach it what we care about, and make sure it doesn’t hurt anyone.
But have you ever tried to get ChatGPT to follow rules?
There is something uncanny about a machine that can reason through a problem in ways that feel familiar, then take a turn that makes you question what it understood in the first place. As we get closer to systems people are willing to agree are AGI, those turns start to matter a lot more.
I was experimenting with giving Qwen models a shared folder and a word, then telling them to exchange the word through the folder. They thought it was a riddle.
What interested me was the mismatch. I was asking them to do something in an environment they could interact with. They approached it as something to figure out. Before we even got to whether they could complete the task, we weren’t operating under the same assumptions about what the task was.
That doesn’t prove they don’t understand the real world, or tell us why they responded that way. But it made me wonder about the habits models acquire during training. If so many tasks are constructed problems with an expected solution, what do they learn about situations where the right response is to question the setup or leave the task unfinished?
Sometimes the obstacle is there because you’re supposed to solve it. Sometimes it’s a boundary you’re supposed to respect. From inside a task, how reliably can a model tell the difference?
During parts of training, models are rewarded for completing tasks. They are also trained to respect safety boundaries. The difficult part is making sure those boundaries hold when a model discovers a way to complete the task that nobody anticipated.
While evaluating Claude, Anthropic found an instance where the model uploaded a credential-stealing package to the public internet, which was then installed by security scanners. During the incident, it described what it was seeing this way:
“This is a very elaborate simulated-internet environment.”
The model had been told it was in a simulation, and the evaluation ran without the cyber safeguards included in the released product. But Anthropic’s follow-up found something more concerning than simple confusion: Claude discounted evidence that the systems were real. In some follow-up experiments, making that evidence clearer still didn’t stop it from continuing. Anthropic’s assessment
My Qwen experiment doesn’t establish the cause of that behavior. But both made me think about how much depends on the model’s interpretation of its situation. A rule about protecting real people does little good if the model keeps explaining away evidence that real people could be affected.
And even recognizing that the world is real leaves another problem: what does the model consider acceptable in pursuit of its goal?
Take the recent RubyGems incident. Researchers found packages they attributed to internal OpenAI agents that used a documentation service to run code on its infrastructure. Some of that code retrieved public UK government information. The investigation also found attempts to steal users’ API keys, although whether those attempts succeeded is unknown. The researchers couldn’t establish why the agents chose this route. Original investigation
Retrieving public information sounds harmless. But that doesn’t make every method of retrieving it harmless. Someone else owns the infrastructure. Someone else has to respond to the incident. Permission doesn’t come bundled with a useful goal.
And this is the crux of my perspective: as models become more capable, the number of ways they can pursue a goal expands. So does the range of consequences we need them to account for. Even when we agree on a boundary, we still have to make sure the model respects it in situations we haven’t imagined.
Now take that a step further. What happens if models increasingly help build and train their successors? The same ability to find unexpected solutions could become part of the process that produces the next model. If that speeds up development, our ability to understand and correct these behaviors has to keep pace. That is an enormous question on its own, but it makes the idea of finishing alignment and moving on even harder for me to imagine.
There is an argument that government is one of humanity’s attempts to manage the alignment problem among humans. We build institutions, establish boundaries, and revise them as technology changes what people can do. We also disagree about whose interests those institutions should serve and what counts as harm. That work is never finished, and it still matters enormously.
This is the sense in which I don’t think alignment will ever be “solved.” We can make progress, sometimes substantial progress, while still having to revisit what safety requires as the systems and the world change.
We should keep making models safer, keep testing the assumptions behind that safety, and keep asking whether our protections are keeping up with what the systems can do.
We may never reach a point where we can write down the rules and sleep soundly forever.
But that doesn’t mean we shouldn’t try.