Notes on agent engineering

I’ve been spending time with agent observability tooling lately - framework quickstarts, lab demos, production traces. If you’ve graduated past a single API call and want to understand what breaks next, these notes are the map I wish I’d had.

Beyond the weather bot

Every ecosystem has its hello world. For LLMs, it’s a chat completion - send “Hello”, print the reply, done. For agents, it’s the weather example: define a get_weather() tool, ask about San Francisco, watch the model emit a tool call.

Here’s the thing though. That weather demo only proves your harness is wired up correctly. It doesn’t prove anything about agency. A real agent controls its own loop across multiple steps - it calls a tool, reads what came back, and decides whether to keep going or stop. That middle decision, the continue-or-stop, is where everything interesting happens.

Real systems layer planning, memory, and genuine judgment on top of that loop. Consider a task like “what should I pack for my trip?” That requires actual agency - the agent has to read your calendar to figure out where you’re going, then call the weather API for that destination, then synthesize an answer. You literally cannot specify step two until step one returns. That’s a different category of problem from “call a tool and return the result.”

The reliability math nobody shows you

Frontier models are getting better at long runs - dozens of tool calls, hours of execution. They’re being RL-trained to handle more tool calls, which means better context management. But production has a ceiling, and it’s mathematical.

End-to-end success equals per-step reliability raised to the power of the number of steps. That’s it. That’s the formula. And it’s brutal.

At 95% per-step reliability across twenty steps, your full run succeeds about 36% of the time. Basically a coin flip. At 99% per-step, twenty steps still only gives you 82%. This is why polished demos break under real traffic - individual steps look fine, but the sequence drifts into failure.

Once you internalize that curve, certain infrastructure becomes non-negotiable: checkpointing, partial recovery, resume-after-failure. Bumping to a larger model might improve per-step reliability, but the exponent stays. You’re building a state machine with a stochastic transition function in the middle, and no amount of wishful thinking changes that.

Two different things called “stochastic”

The harness itself is deterministic code. Tools run the same way every time. Context assembly is a function. Transitions are edges you drew. But then execution hits the model, and what comes back is text that was impossible to derive in advance.

People confuse two separate problems under the “stochastic” label, and it wastes a lot of debugging time.

The narrow problem is sampling variance - temperature, top-p, and friends. You can set those to zero and still lose bit-identical reproducibility, because batching order, floating-point arithmetic, mixture-of-experts routing, and silent model revisions all sit underneath the exact same model name.

The broader problem is underspecification. There’s no closed-form spec for “the next customer message” the way there is for parseInt("42"). Fixing your sampling parameters leaves this layer completely untouched.

Don’t try to debug the oracle. Constrain what reaches it. Log what comes back. Measure the output distributions. That’s the game.

Testing something that doesn’t have a right answer

Classical tests assert output === expected. Agent runs break this pattern because inputs are open-world and outputs have tons of valid forms. There are a hundred perfectly fine replies to “where is my order?” and they’re all different strings.

Instead, you need scorers that check properties: is the response grounded in actual data? Is the tone appropriate? Did the model use the right tool? Did it invent a return policy that doesn’t exist?

Think of your prompt as versioned application code. Think of your eval dataset as the live input distribution. A good dataset favors variety over depth - twenty-five different edge cases beat twenty-five paraphrases of the same happy path. Weight it toward the tail, because that’s where production systems go to die.

LLM systems add a useful feedback edge that classical systems don’t have: a failed production trace gets annotated and becomes a new row in your eval dataset. Production failures feed the lab. LLMOps platforms are trying hard to make that loop cheap and fast.

The two eval modes serve different purposes. Offline eval scores your curated examples before deployment - it’s a gate. Online eval samples real production traffic after deployment - it’s surveillance. Online catches the new failure modes. Offline pins them down so they can never sneak back in.

Reading what actually happened

When an agent screws up in production, you need the full trajectory. Which tools fired? In what order? With what prompts? How many tokens? Standard HTTP access logs tell you a request completed in 1.3 seconds with a 200 status. That’s the envelope. You need to read the letter inside.

OpenTelemetry with GenAI semantic conventions is how you read the letter.

Vendor quickstarts are basically positioning documents once you know what to look for. LangGraph’s hello world is a calculator agent - a graph with a loop-back edge, designed to show off orchestration. An eval product’s hello world is a scorer over a tiny dataset, designed to demonstrate measurement. An observability product’s hello world is a trivial run whose span tree renders beautifully in a UI.

These layers stack, and the platforms overlap heavily. LangSmith is the default if you’re already on LangChain or LangGraph - it instruments that runtime well. Langfuse is the move for teams wanting open-source self-hosting to control costs; ClickHouse acquired them in early 2026. Braintrust fits when you want deploy gates driven by eval scores right in your CI suite.

When to build and when to buy

All this tooling is relevant for teams building agents in-house. But let’s be honest - that’s the minority path in markets like customer support. Your ticket volume usually decides for you.

Under two thousand resolutions per month? Consumption products like Intercom Fin or Ada are obvious - setup takes hours, costs about a dollar per resolution. In the SMB range, Lindy pencils out well if total support spend stays under half a million a year. Enterprise deployments land on Sierra or Decagon with six-figure contracts and $2-5 per resolution. Already on Zendesk or Salesforce? You’ll probably just adopt Agentforce or Zendesk AI because the integration path is short.

The build path shows up around thirty thousand resolutions per month. At that volume, building your own agent can cost twelve to fifteen percent of what buying one would. But you inherit total operational ownership of the whole stack. That’s where LangSmith-class tooling appears as one small piece of a much larger infrastructure.

The one-liner

Agent engineering, as far as I can tell, is this: you engineer context and control flow as a state machine around a stochastic core, and then you measure it against an input distribution you can’t fully enumerate. Most teams learn that by breaking both halves first.