Test datasets and mock tools

The first way to test an agent: build a dataset of real-world scenarios and test the agent against it. While developing, tracing shows the flow of each run (see traceabilityTraceabilityLog the whole reasoning trace, so any answer can be explained afterwards.Opens in a new tab). Once a scenario matters, a pytest case with mock tools keeps checking it.

The dataset

Each example is an input, a reference (an expected outcome or tool sequence) and optional metadata. Where the scenarios come from matters more than how many there are. Start from real failures. Anthropic’s advice is to begin with the manual checks already run during development, then turn user-reported failures into test cases, so the suite reflects actual usage. LangSmith suggests the same for production traces: collect runs with negative feedback, long latency or errors.

Mock tools

A mock tool replaces a real one, so a test is fast, free, repeatable, and cannot touch a real system. Anthropic recommends extensive testing in sandboxed environments, and a fake tool is the smallest sandbox. LangGraph’s own advice is to build the graph in a factory and compile it with a fresh checkpointer in each test, so no state leaks between cases.

import pytestfrom langchain_core.messages import AIMessagefrom langchain_core.tools import toolfrom langgraph.checkpoint.memory import MemorySaverfrom langgraph.graph import MessagesState, StateGraph, STARTfrom langgraph.prebuilt import ToolNode, tools_condition SCENARIOS = [  # real questions, with the tool calls each should cause    ("Where is order 42?", ["lookup_order"]),    ("Hi there", []),] def build_graph(script, calls):    @tool    def lookup_order(order_id: str) -> str:        """Look up an order."""        calls.append("lookup_order")  # mock tool: records the call, touches nothing real        return "order 42: shipped"     def model(state: MessagesState):        return {"messages": [next(script)]}  # scripted stand-in for the LLM     g = StateGraph(MessagesState)    g.add_node("model", model)    g.add_node("tools", ToolNode([lookup_order]))    g.add_edge(START, "model")    g.add_conditional_edges("model", tools_condition)    g.add_edge("tools", "model")    return g.compile(checkpointer=MemorySaver())  # fresh state for every test @pytest.mark.parametrize("question, expected_tools", SCENARIOS)def test_trajectory(question, expected_tools):    ask = AIMessage("", tool_calls=[{"name": "lookup_order", "args": {"order_id": "42"}, "id": "c1"}])    turns = [ask, AIMessage("Order 42 has shipped.")] if expected_tools else [AIMessage("Hello!")]    calls = []    graph = build_graph(iter(turns), calls)    graph.invoke({"messages": [("user", question)]}, {"configurable": {"thread_id": "t1"}})    assert calls == expected_tools  # the flow, not just the final text

The scenarios are the dataset. The mock tool records that it was called and does nothing else. The assertion is on the trajectory (which tools ran), not only on the final text.

The catch

The model here is scripted, so this proves the wiring, not the model’s judgment. With a real model the same scenarios run as evals, and results vary between runs. LangChain’s agentevals package compares a real run with a reference trajectory in four modes: strict, unordered, subset (no extra tools) and superset (at least these tools). Which one fits depends on how much the order of steps matters (see agent evaluationAgent evaluationTest the flow an agent took and what it left behind, against the goal and the guardrails.Opens in a new tab).