Agent ops

Testing ends at deployment, watching does not. Agent ops is observability and monitoring after deployment, over three concerns: performance and quality, safety and governance, and cost effectiveness. Anthropic describes production monitoring as the thing that reveals real user behavior at scale and catches what synthetic evals miss.

Cognitive health

Uptime and error rates are not enough for an agent. It can be up and still be wrong. The thing to monitor is its cognitive health, which comes down to five questions.

• Is the answer correct? Online scoring of live traffic, with the same judge used in testing (see LLM as judgeLLM as judgeA second model scores the result against written criteria.Opens in a new tab).
• Is it confused, or stuck in a loop? Count runs that hit a step limit.
• Is it hallucinating? User reports and a judge on sampled runs (see hallucination preventionHallucination preventionIt cannot be switched off, so it is stacked: prompt, context and flow together.Opens in a new tab).
• Is it following the rules? Guardrail violations, blocked and logged (see guardrailsLLM guardrailsChecks around the model, so a bad input or a bad output is caught by something that does not rely on the model behaving.Opens in a new tab).
• What was the chain of thought? Only a full trace answers it (see traceabilityTraceabilityLog the whole reasoning trace, so any answer can be explained afterwards.Opens in a new tab).

Catching a loop

LangGraph raises GraphRecursionError when a run exceeds its recursion limit. The default is 1000 steps, far more than an agent should need, so set it lower for your graph and treat hitting it as an event worth recording.

from langgraph.errors import GraphRecursionError def run_guarded(graph, inputs, log):    try:        return graph.invoke(inputs, config={"recursion_limit": 25})    except GraphRecursionError:        log("agent_loop_stopped", steps=25)  # count these, do not just swallow them        return {"messages": [("ai", "Sorry, I could not finish that. A person will follow up.")]}

The catch

Anthropic lists a maximum number of iterations as a common stopping condition, and it keeps a stuck agent from running forever. But a cap on its own hides the problem. If every stopped run returns a polite apology and nothing is counted, the loop looks like success. Log it, and let the count feed the numbers in agent metricsAgent metricsSix numbers that say whether a deployed agent is healthy.Opens in a new tab.