There Is No Reward Function for Meaning

When Progress Becomes Ordinary

A century ago, sending word to another continent meant waiting weeks for a ship to cross the ocean. Today, a livestream lags for half a second and we complain. Smartphones put a camera, a map, and a library in our pockets, yet a low battery gets our attention more readily than any of those capabilities. For some reason, we celebrate a breakthrough only once before it quietly becomes the baseline.

I see the same pattern in how we talk about AI, especially in the “strawberry” meme. Language models confidently miscounted the r’s in strawberry, and the mistake became an easy way for people to dismiss their intelligence. Since then, AI systems have disproved a longstanding mathematical conjecture, helped reduce protein synthesis costs, and uncovered security flaws in Firefox. Yet discussions of their intelligence can still return to that spelling mistake rather than what they have since become capable of doing.

Over the past year, I caught myself doing the same thing with coding agents. As they became more reliable at finishing long-horizon tasks, my expectations rose too, making it more frustrating when they misunderstood requirements or overlooked established guidelines.

Still, I find the way agents handle failure compelling. An agent will read an error, revise its approach, and rerun the tests, sometimes repeating that process dozens of times before producing anything useful. The results from each attempt help guide what the agent tries next.

Monitoring that loop up close has made me appreciate how much problem-solving depends on iteration. For us humans, abandoning an approach we have invested in can feel like admitting we were wrong, which makes it harder to let go. A coding agent, by contrast, can use feedback to try another approach without that personal attachment. Seen from outside, its willingness to revise resembles intellectual humility.

When the Objective Is Clear

Iteration only helps if the agent knows what a better result looks like. This became clearer through my experience using coding agents at work. Early on, the agent could follow the code but lacked the background needed to interpret it in the context of our system. For example, it inferred our file-naming and code-location conventions from older files that didn’t follow them, then repeated those inconsistencies in its own changes. A feature or fix could require infrastructure changes in a separate project, but the agent didn’t know where a service’s infrastructure was defined, so it could overlook the additional changes and merge requests needed to complete the work. It struggled to identify which sources of information to trust and which reasonable-looking approaches had already caused problems. Its investigations consequently stayed shallow, and its implementations were often plausible in isolation but wrong for our system.

Gradually, I started treating recurring failures as gaps in the agent’s environment and documenting the relevant gotchas in AGENTS.md. Much of the context it needed had never been documented; it was domain knowledge that used to live in an engineer’s head or personal notes. One section explained how the corporate VPN limited external network access, which had caused problems for fresh agents making API requests. Another covered styling rules for GitLab merge request descriptions: use !123 to link to a merge request rather than writing out its full URL. I also spelled out when the agent should hand a problem back to me. If my AWS SSO session expires, it should ask me to refresh it instead of trying out some other workaround and masking the issue.

Alongside those instructions, I gave the agent tools for carrying out recurring work. Workflows I used often became skills, like running Argo, Helm, and Kubernetes tests ad hoc in our dev environment. I also added helpers such as a wrapper around the Prefect CLI to inspect flow logs and metrics. To manage how much context accumulated during longer sessions, I set a lower threshold for automatic compaction so they would be compacted well before reaching the model’s context limit.¹

Birgitta Böckeler has a name for this user-built layer: the “outer harness”. Coding agents, she writes, give users “many features to build an outer harness specifically for our use case and system.” Her phrase fits what I have been building. My harness is a way to hand our team’s local knowledge, tools, and feedback to a general-purpose model.

As I improved the harness, the agent investigated more thoroughly and produced more accurate implementations. The model stayed the same for much of that time; the improvements came from the context, tools, and guidance I provided.

That experience with the harness changed how I think about prompting. With the relevant context and skills in place, I can give the agent a problem and ask it to clarify assumptions, discuss tradeoffs with me, and write tests as it works through the implementation. I spend less effort trying to anticipate everything in the initial prompt, because the harness gives the agent ways to gather what it needs and bring unresolved decisions back to me. The exact wording of the prompt matters less when that support is already available.

Software engineering gives agents one more advantage: frequent feedback on whether their changes are working. The compiler rejects invalid code. Tests expose incorrect behavior, and benchmarks show whether performance improved. With such signals, an agent can keep adjusting its work and determine whether it is getting closer to the objective. This feedback loop helps explain why coding agents can appear so capable when the task is well-specified.

My harness has limits, though. It can tell an agent how to run our tests, but those tests don’t settle every design decision. Several implementations might pass the same tests, leaving us to decide where an abstraction belongs, whether a performance improvement justifies its added complexity, or which document or system should serve as the source of truth. Making those choices takes experience and judgment, and some consequences only become clear later—for example, when an abstraction makes the next feature harder to add or an optimization becomes a maintenance burden.

What Is Worth Pursuing

When I think about the questions that shape a life, the tension between feedback and judgment gets harder to ignore. When should I put someone else’s needs before my own? Is it worth leaving a comfortable job for work that might matter more? When, or whether, should I have children?

Although some of these questions concern morality and others meaning, each asks us to consider what is worth pursuing before deciding how to pursue it. Emotions such as satisfaction and regret, along with the reactions of people we care about, can help us reflect on our choices. The difficulty is knowing how to interpret that feedback. A compiler error gives an agent a specific problem to investigate, while a test suite provides criteria for checking its work. Regret can reveal a mistake, though understanding why we regret a choice—and what we should learn from it—still takes judgment. More broadly, we don’t have an equivalent test suite for deciding whether we are becoming the person we hoped to be. We still have to discern what these signals mean and how much weight to give them.

We can gather information and consult people we trust to understand a choice more fully. For someone considering a career change, this might mean examining the effects on their finances, family commitments, and day-to-day work. Uncertainty comes from not knowing how those consequences will unfold, and better predictions could help us make a more informed decision.

To explore how far better prediction could take us, consider Yann LeCun’s work on world models, which aims to help AI systems predict the consequences of actions and plan toward a goal. As a thought experiment, imagine extending that capability to predict how a career change would affect someone’s life. Even with accurate predictions, two people might weigh the same consequences differently. Understanding what each future could involve would still leave them to consider what makes one worth pursuing.

For the moral questions, Hume’s is–ought problem helps explain why predicting the consequences of a choice differs from judging whether we ought to make it. Knowing what would happen does not, by itself, establish what ought to happen; the argument needs some further premise about what matters.² Questions about meaning aren’t identical to moral judgments, but the career example raises a related concern: a prediction alone doesn’t specify what would make one future worth pursuing over another.

In machine learning (ML), we address the practical question of what to optimize by supplying a criterion ourselves. In a coding harness, test results help the agent decide how to revise its code. During training, an objective such as predicting the next token or the next robot action instead guides adjustments to the model’s parameters, changing how it responds to future inputs. In both cases, people specify what improvement means. Even proposals for more autonomous learning rely on people to define success and design the conditions in which the system learns.³

Yet a training objective doesn’t fully specify the capabilities that can develop while it is being optimized. Researchers in mechanistic interpretability investigate the internal structures models learn and how those structures contribute to their behavior.⁴ Anthropic’s work on Claude’s J-space, for example, describes an internal workspace that appears to support deliberate reasoning and emerged during training rather than being explicitly programmed. Still, this finding doesn’t demonstrate that a model understands meaning. It does, however, make me reluctant to assume that models can learn only what we explicitly train them to do. This possibility matters for artificial general intelligence (AGI), if we eventually build it.⁵ Google DeepMind’s framework describes general capability through the breadth and depth of tasks a system can perform. An increasingly capable system might learn to help us examine our values as well as achieve goals we have already chosen.

Suppose that the broader capability did help us understand our values. A system might reveal that two commitments conflict, explain why the conflict matters, and suggest ways to reconcile them. Understanding those competing reasons more clearly could lead us to reconsider our priorities. Even with that help, we would still need to assess whether its recommendation about which commitment to prioritize is worth following. Solving all of humanity’s problems would mean addressing conflicts about what matters as well as finding ways to achieve agreed goals. Can AGI deliver on that broader ambition? I do not know, and answers that make the problem sound simple leave me skeptical.

I have reached a point in my life where judgment interests me more than certainty. Some of my favorite lessons leave room for interpretation. I find meaning in the quirks of human thought, and in the matters of taste and opinion that each of us has to weigh for ourselves. The people I admire are willing to wrestle with ambiguity and accept that some disagreements will stay unresolved.

I expect AI to become an extraordinary scientific collaborator and help solve many problems we care about. Like livestreaming and smartphones, its achievements may eventually become ordinary parts of life. We may grow accustomed to those gains while continuing to benefit from them. We would still have to decide what to do with those gains and how to spend the finite years we are given. Making those choices means considering what matters enough to devote our time to. In ML, a chosen criterion lets us evaluate progress toward an objective. In life, deciding what should count as progress—and revisiting that judgment as our priorities and responsibilities shift—is itself part of what we are trying to figure out. We aren’t given a specification that dictates what is worth pursuing. In that sense, there is no reward function for meaning.