Traceability

2 min read
On trails:AGOPSstep 5/7

To judge an agent’s cognitive healthAgent opsAfter deployment, watch performance, safety and cost, and whether the agent is thinking straight.Opens in a new tab you need to see how it thought, and a final answer does not show that. Traceability means logging the entire reasoning of the agent, so any answer can be traced back through the steps that produced it. LangChain puts the stakes plainly: if you cannot see the trajectory, you cannot reliably debug the behavior or turn the failures into future evals.

What goes in a trace

  • Context: what the model was given on that step (see the context assemblerThe context assemblerThe step that builds the prompt from scratch on every loop.Opens in a new tab).
  • Reasoning: the LLM’s thoughts and plan.
  • Tool calls and parameters: what it chose to call, and with what.
  • Tool results: what came back.
  • Outputs: what it finally said or did.

A trace is nested

In LangSmith a trace is a series of steps, each one a run. The shape is a tree: one root for the whole agent run, and a child for every model call and tool call inside it.

One agent run is a root span. Inside it, in order: an LLM call that plans, a tool call, an LLM call that reads the result, another tool call, and an LLM call that writes the answer. Each span records its inputs, outputs and timing. TRACE: ONE AGENT RUN (invoke_agent) LLM call plan, decide tool call lookup_order LLM call read result tool call issue_refund LLM call final answer prompt, thought, tokens, time parameters, result, time prompt, thought, tokens, time parameters, result, time prompt, answer, tokens, time root: user input, final output, total cost, user and session id every span: what went in, what came out, how long it took
One run, many spans. Each span keeps what went in, what came out and how long it took.

OpenTelemetry’s GenAI conventions describe the same tree with agent, workflow, plan and tool-execution spans. They are still marked as in development, so names may change.

In LangGraph

For LangChain and LangGraph, tracing is switched on through environment variables, and a tracing_context can scope it to one block of code. Tags and metadata are what make a trace findable later: by user, session or version.

trace_run.pyPython
import langsmith as ls # LANGSMITH_TRACING=true and LANGSMITH_API_KEY are set in the environment.# `graph` is a compiled LangGraph graph.with ls.tracing_context(    project_name="refund-agent",    enabled=True,    tags=["production", "v1.0"],    metadata={"user_id": "user_123", "session_id": "session_456"},):    graph.invoke({"messages": [{"role": "user", "content": "Where is order 42?"}]})

The catch

A full trace holds everything the agent saw, including user data and tool results. The log is as sensitive as the conversation. Mask personal information before it is stored, the same way an output guardrail does (see guardrailsLLM guardrailsChecks around the model, so a bad input or a bad output is caught by something that does not rely on the model behaving.Opens in a new tab). What the trace shows then feeds the numbers in agent metricsAgent metricsSix numbers that say whether a deployed agent is healthy.Opens in a new tab.

Sources