Two things stand out about AI right now. One is where the attention is going. The other is where the real work actually is.
Fully autonomous agents, systems that take a goal and run end to end with no one checking in, get most of the attention. Evaluation and observability get almost none. I think that's backwards.
What "Fully Autonomous" Actually Means in Practice
Every multi-step agentic system I've built that holds up in production has deliberate checkpoints. None of them run full autonomy from the first step to the last.
Take a system that parses a prescription or extracts a lab value. When confidence is low, the output still goes to a person before anything is treated as final. Take a system that chooses between a fast default model and a slower reasoning model. That choice is written as explicit routing logic. The agent does not get to decide, unsupervised, how much of its own output to trust.
That is not a failure to achieve "real" autonomy. That is the pattern that survives contact with production.
An agent that runs ten unsupervised steps multiplies its error rate at every step. A small mistake early in the chain does not stay small. It becomes a large mistake by the end, often before anyone notices. So the systems that work are not the ones with the most autonomy. They are the ones with the clearest boundaries, i.e. here the agent can act alone, and here a check is required.
The "fully autonomous agent" pitch sells demos well. A demo is a clean, curated path. Nothing goes wrong because the path was built so nothing can go wrong.
It does not sell in real-world production scenarios. Production is messy. It is adversarial. It is full of edge cases. Autonomy without checkpoints fails exactly there.
What You Actually Need to Measure
Most agent demos get graded on one thing: did the final answer look right. That is not enough.
An agent can land a correct looking answer after a broken tool call. Or after a skipped policy check. Or after a retrieval miss that did not affect the final answer this time. If you only score the last message, you will expand autonomy on luck.
For multi-step systems I care about at least four layers:
- Final task success: did the user visible outcome match a labeled expectation
- Step / trajectory quality: did the agent call the right tools in a sensible order, or did it flail through bad steps and only reach a correct looking answer by accident
- Constraint violations: taboo topics, PII leaks, tenant bleed, schema breaks, unsafe actions
- Cost and latency budgets: a correct answer that burns 40 tool calls is still a failure in production
Those are different failure modes. One metric will not catch them.
If your harness only does pass/fail on the final string, you are not evaluating an agent. You are evaluating a chatbot that sometimes uses tools.
Why Evaluation Is the Part Nobody Wants to Fund
Evaluation is underrated for a boring reason: it doesn't demo well.
A good harness doesn't produce an impressive live moment. It produces a dashboard. It produces a regression suite. It produces labeled cases that catch when a change quietly makes things worse. None of that is exciting. None of it is optional if you want to trust a system enough to expand what it can do on its own.
In the evaluation work I've done, what mattered wasn't a bigger model. It was a ground truth set. And it was a habit: running the comparison before every meaningful change, not just when something visibly broke.
That's slow and unglamorous work. It's also exactly what gets skipped when a team is under pressure to ship the demo instead of the system.
Offline First. Online Second. Not the Other Way Around
Offline eval is the regression gate. Fixed cases. Known labels. You run it on every prompt change, every model swap, every tool schema change, and every retrieval tweak. If that gate is red, nothing ships.
Online eval is what you learn after traffic hits. Thumbs. Edits. Escalations. Abandon rates. Tool error rates. Retry loops.
If you only have online signals, you are reacting after the fact with no stable baseline. If you only have offline cases, those cases go stale and stop matching what users actually do.
The teams that get stuck usually invert this. They ship because the demo looked good. Then they try to invent metrics after users complain. By then the agent already has enough autonomy that a failure is hard to replay and hard to pin down.
Frameworks Are Useful. None of Them Replace Your Labels
There are decent tools now. Use them, but do not confuse them with a strategy. You still need a real strategy to get the most out of them.
If your system is retrieval heavy, something like RAGAS helps. It checks whether the answer stays faithful to the retrieved chunks, whether it is relevant, and whether the context you pulled was actually useful. That is a retrieval and generation slice. It will not tell you the agent called the wrong tool. It will not tell you the agent broke a business rule that never appeared in the chunks. Do not treat it as a full agent grade.
If you want assertions that look like unit tests around LLM outputs, DeepEval is in that lane. Custom metrics, regression suites, CI hooks. It helps once you already know what "good" means for a step. It will not invent that definition for you.
Promptfoo is good when you need to compare many prompts and models on the same cases and see what broke. That is how you catch a change that fixed the demo and quietly failed the edge cases. It gets weaker when the agent carries long state and lots of tool calls, unless you build fixtures that can replay those paths.
LangSmith and Langfuse style tracing get closer to how agent systems actually fail, because you can inspect the full chain instead of only the last string. Dataset runs against logged traces help. But traces without labeled outcomes are still just nicer logs. You can watch the failure. You still have not graded it.
Google ADK eval is aimed at agents built on ADK. It checks trajectory and final response together: tool use, path quality, trajectory matching, rubric based tool use, final response match, and rubric based response quality even when you do not have a reference answer. You run it with adk eval against an eval set and thresholds, and you can add custom metrics for domain rules the builtins will never know. That is the right shape for agents. The limit is also clear. It is ADK native. If you are on LangGraph, a custom orchestrator, or a mixed stack, you still need a harness that can replay those paths. And without labeled cases and promotion gates, ADK eval is just another way to watch demos pass.
Even with all of that, most production agent systems still need a custom harness for the checks that matter most. The reason is simple. Your domain rules are specific to your product: did the prescription fields parse correctly, are the lab units valid, did tenant A ever see tenant B's data, is the output schema valid, did the draft hit a taboo topic. A generic framework does not know those rules. Frameworks help you run and score tests. Your labeled ground truth decides what pass and fail mean.
So pick tools for the job. Use RAGAS where retrieval matters. Use ADK eval or trajectory assertions where tool paths matter. Use domain validators where the business cares. Put the gate in CI so a small prompt tweak cannot silently regress cases you already solved.
LLM-as-Judge Is a Tool, Not a Ground Truth
Using a model to grade another model is fine for some jobs. It works for ranking rough quality, catching tone drift, and scoring open ended answers where exact string match is meaningless.
Do not use it as your only check when safety is on the line, when a wrong factual claim can hurt someone, or when you would not defend the answer to a customer or an auditor.
There is also a structural problem. If the judge is cheaper and weaker than the agent, it will miss failures the agent is clever enough to produce. If the judge is from the same model family as the agent, they can share the same blind spots.
So calibrate any judge against human labels on a holdout set. Track how often the judge agrees with people. Where the stakes are high, keep humans in the loop for the labels that decide whether you are allowed to give the agent more autonomy.
What a Minimum Serious Harness Looks Like
- A versioned golden set: inputs, expected outputs, and the tool traces you expect when tools matter
- Cases grouped by how they fail: retrieval miss, tool error, policy violation, format break, wrong specialty or domain assumption
- Run deterministic checks first (schema, regex, policy lists, exact field compares), then use model judges for what those checks cannot cover
- Block merges in CI when anything meaningful changes: prompt, model id, tool schema, retriever, temperature
- Log full trajectories in production so you can replay a failure offline later
- Only expand autonomy when the gate stays green on the new scope
That last rule is the point of the whole post in practice. Evaluation is what lets you give an agent more freedom safely. Without it, you are shipping more autonomy and hoping nothing breaks later.
These Are the Same Mistake Wearing Two Faces
Skipping evaluation and giving an agent too much freedom look like opposite problems. They are not. In both cases you are trusting the output without a way to prove it is good.
If you ship broad autonomy without evaluation, small errors stack up and eventually reach real users. If you build evaluation but never use it to widen what the agent is allowed to do, you stay stuck in a cautious pilot. You have measurement, but no path to more useful autonomy.
The teams that get this right will not be the ones racing for maximum autonomy first. They will be the ones who can show, with evidence, that the agent earned the next level of freedom.
What I Expect to Happen Next
I expect the "fully autonomous agent" pitch to get quietly walked back over the next couple of years. Not because models stop improving. It's because enough systems will fail publicly, and the industry will correct toward bounded autonomy with real evaluation behind it.
That's not pessimism about the technology. It's the same pattern every previous wave of automation followed. Measurement ends up mattering more than the flashy demo. It just takes a few expensive failures for that to become obvious.