There Is No Reward Function for Meaning
We Only Celebrate Progress Once
A century ago, sending word to another continent meant waiting weeks for a ship to cross the ocean. Today, we complain when a video call lags for half a second. For some reason, we celebrate a breakthrough once, then it quietly becomes the baseline.
Smartphones condensed cameras, maps, encyclopedias, music players, and libraries into devices that fit in our pockets, yet we’re quicker to notice a low battery than to appreciate all other aspects of convenience they bring to our lives. Once something remarkable becomes routine, we stop noticing it at all.
I think we’ve fallen into the same pattern with AI. The “strawberry” meme is a good example: we ridiculed language models when they confidently miscount the number of r’s in the word strawberry. It became a simple, memorable way to argue that AI wasn’t really intelligent. At the same time, those same models could explain advanced mathematics, summarize research papers, write working software, and help millions of people learn new skills. Somehow the mistake became more remarkable than the capabilities.
But the joke is on us. We normalize competence almost instantly and grow increasingly sensitive to failure. Once something consistently exceeds our expectations, we stop rewarding it for doing so. We simply raise the bar and wait for the next mistake.
Intelligence Isn’t the Impressive Part
After a year of working with coding agents, I’ve come to appreciate that their most interesting quality isn’t raw intelligence. It’s their willingness to iterate.
They produce code that won’t compile, misread requirements, reach for the wrong approach. Then they read the error, revise, rerun the tests, and try again. Then again. Sometimes dozens of times before creating something of value. Watching the process has made me realize that progress often depends less on getting the first answer right than on continuously incorporating feedback.
Humans aren’t always as good at this. Once we’ve invested time in an idea, it becomes difficult to abandon it, and our pride unconsciously folds hypotheses into identities. We defend decisions because they are ours rather than because they are correct.
Machines don’t have that problem. They don’t care whether yesterday’s idea survives the afternoon; if the feedback says the approach isn’t working, they move on. There’s no ego to protect, and from the outside, it looks a lot like intellectual humility.
The Importance of Having an Objective
Working with coding agents has also changed how I think about intelligence itself.
I’ve found that their performance depends surprisingly little on clever prompting and surprisingly much on the quality of the objective I give them. If I clearly define the problem, explain the important tradeoffs, enumerate the edge cases, and provide a reliable test harness, they can often complete ninety percent of the implementation with very little intervention. In those situations the bottleneck usually isn’t the model. It’s my ability to specify what success actually looks like.
The remaining ten percent is almost never about correctness. It’s about judgment. Should this abstraction live here or somewhere else? Should the implementation optimize for readability or performance? Is documentation more valuable than self-explanatory code? Where should the single source of truth live? None of these questions have objectively correct answers. They are decisions shaped by experience, context, and instinct.
That distinction has become one of the most important lessons AI has taught me. Coding agents thrive because software engineering provides unusually clear feedback: every compilation, test suite, benchmark, and code review tells the system whether it’s moving closer to or farther from the objective, and the objective itself is rarely in dispute. I don’t think this is unique to programming, either. Mathematics, engineering, and much of science share the same property. Success is measurable, and reality answers with a clear “yes” or “no.” Those are exactly the environments where optimization shines.
But Life Doesn’t Work That Way
Then I think about the questions that bother us and have a long-term impact on our lives. Whether to forgive someone who never apologized. Whether to leave a good job for one that might matter more. When, or whether, to have children. None of these are hard for lack of information. They are hard because there isn’t a universally agreed definition of success. There is no compiler error for purpose, no benchmark for wisdom, and no test suite that tells us whether we’re becoming the person we hoped to be. Instead, we navigate through experience, relationships, culture, memory, and intuition. Two thoughtful people can examine the same situation and arrive at different conclusions about what matters, and neither is necessarily wrong, because the disagreement isn’t about facts. It’s about values.
It made me reconsider something I’d taken for granted: maybe intelligence isn’t only about solving problems, but about deciding which problems are worth solving in the first place.
There Is No Reward Function for Meaning
Maybe That’s the Point
People often ask whether we’ll eventually build artificial general intelligence (AGI). I suspect we’re Even the most ambitious new architectures run into the same limit. Yann LeCun’s world models, for instance, try to build a machine’s internal map of reality so it can reason about cause and effect and plan ahead. It’s a real step beyond next-token prediction. But the model still learns the only way these systems can: by minimizing a loss, the gap between what it predicts and what it then observes. Make that map richer and you get a better predictor, not a system that knows which futures are worth wanting. A more advanced model doesn’t escape optimization; it just makes the objective harder to see. Whatever the system, reward or loss, the target is still something someone had to define.
And understanding what is doesn’t tell us what ought to be. A perfect prediction of every possible future still leaves open which future we should prefer. My experience with coding agents keeps bringing me back to the same point: optimization only works once the objective is defined. Programming gives us that objective. Nature often gives one to scientists. Life usually doesn’t.
That may be the wall, not just for today’s language models but for tomorrow’s world models too. Fundamentally, many of the questions we care about most aren’t optimization problems waiting for better algorithms. Consciousness, beauty, purpose, morality, and meaning aren’t things we discover the way we discover the speed of light. They are things we continually negotiate through experience, reflection, and relationships.
Maybe That’s the Point
People often ask whether we’ll eventually build artificial general intelligence (AGI). I suspect we’re asking the wrong question.
I’m at a point in life where I am less interested in perfection and more interested in judgment. Some of my favorite lessons don’t provide answers. They leave room for interpretation. For me, that’s where the meaning is. I find it in the quirks and diversity of human thought, and in the matters of taste and opinion each of us has to weigh for ourselves. The people I admire most are thoughtful not because they’re always right, but because they’re willing to wrestle with ambiguity. Much of what makes life meaningful exists precisely because reasonable people can disagree about it.
I don’t think those ambiguities are flaws in human intelligence. I think they’re part of what makes life worth living.
AI will almost certainly become an extraordinary partner for discovering objective truth. It may become our greatest scientific collaborator, our most capable engineer, and our most tireless researcher. But the questions that define a human life—what to value, what beauty is, why consciousness exists, and how to spend the finite years we’re given—may remain stubbornly resistant to optimization. Not because machines lack intelligence, but because there is no reward function for meaning.