Been spending a lot of time lately poking around agent observability tools, framework quickstarts, and prod traces. Putting my notes down in one place for anyone moving past basic single-call LLM wrappers.
1. What makes something an actual agent?
Every tech stack has its “hello world”. For plain LLMs, it’s a single chat completion. For agents, every tutorial uses the weather bot: define get_weather(), ask about San Francisco, model emits the tool call, done.
Except that weather demo only tests that tool-calling syntax works. A 1-step tool call is just a router.
An actual agent runs in a loop where step N+1 depends on what came back from step N. Suppose you ask “what should I pack for my trip?” The model has to check your calendar first to even know what city you’re flying to, then call the weather tool for that city, then check what’s on your packing list. You cannot hardcode the sequence upfront coz step 2 depends on the output of step 1.
exercise · classification
Tool call or agent loop?
select a scenario
Criterion: explicit continue/stop between steps.
2. Why 20-step agent runs fall apart in prod
Frontier models are way better at long tool-call chains now than a year ago, mostly coz labs are RL-training them directly on multi-step tool use.
Still, simple probability wrecks long runs:
End-to-end success = (per-step reliability) ^ (number of steps)
Play with the numbers below and see how fast it drops:
- 95% reliability per step across 20 steps = 35.8% full-run success
- 99% reliability per step across 20 steps = 81.8% full-run success
exercise · compounding reliability
End-to-end success = (per-step reliability)n
- end-to-end
That’s why a demo looks magical in a 3-step tweet video and then fails 60% of the time on real prod tickets. Once your workflow is 15-20 steps long, you have to build in checkpoints, step-level retries, and state recovery so a failure at step 17 doesn’t force you to restart from step 1.
3. Two different things people lump under “non-deterministic”
All your harness code is normal deterministic software, and then in the middle you call the model. When people complain that LLMs are “stochastic”, they usually mix up two separate things:
- Sampling / runtime variance: Even if you set
temperature = 0, you still won’t get bit-identical outputs every time because of GPU batching order, floating-point non-associativity, MoE router tie-breaking, and silent provider weight updates. - Prompt underspecification: For open-ended tasks (like drafting a support reply or refactoring a module), there isn’t one single valid string like
parseInt("42") === 42. There are 50 valid ways to answer, and tiny changes in context will shift which path the model takes.
exercise · resampling
Fixed prompt: “Where’s my order?”
Resample to observe variance.
Setting temperature = 0 only narrows #1, it does nothing for #2.
4. Evals when you can’t use assert(output == expected)
Since you can’t do exact string matching on agent outputs, you end up testing properties instead:
- Did it call the right tool with valid args?
- Is every claim in the answer grounded in the retrieved data?
- Did it make up a refund policy that doesn’t exist?
Best way to think about it: your prompt + harness is the code, and your eval dataset is your test suite. Don’t waste time writing 25 slightly reworded happy-path prompts. Grab 25 weird edge cases that actually broke in prod.
That feedback loop is the whole game in LLMOps: whenever an agent fails in prod, take that trace, label what went wrong, and add it as a row in your offline eval suite so you never regress on that bug again.
- Offline evals = CI gate before you deploy a prompt/model change
- Online evals = sampling live prod traces to catch new failure modes
5. Observability: LangSmith vs Langfuse vs Braintrust
Normal APM/HTTP logs tell you POST /agent 200 OK 4.2s. Useless when the agent hallucinated inside step 4 of 7. You need the full span tree: every LLM call, exact prompt messages, tool inputs/outputs, token counts, and latency per step (OpenTelemetry GenAI semantic conventions are becoming the standard for this).
How the main tools compare right now:
- LangSmith - easiest default if your team is already using LangChain or LangGraph.
- Langfuse - best open-source option if you want to self-host and keep observability bills sane (ClickHouse acquired them in early 2026).
- Braintrust - great if your focus is CI eval gates and scoring datasets on every PR.
6. Build vs buy (the customer support math)
Most teams reading about agent harnesses shouldn’t actually be building their own support agent from scratch tbh. Do the math on ticket volume first:
- < 2,000 resolutions/month: Just buy Intercom Fin or Ada. Takes a few hours to set up, costs ~$1/resolution.
- SMB (< $500k/yr support spend): Tools like Lindy usually pencil out.
- Enterprise: Sierra or Decagon (6-figure contracts, ~$2-5/resolution), or Agentforce / Zendesk AI if you’re already locked into Salesforce/Zendesk.
- > 30,000 resolutions/month: This is where building in-house actually makes financial sense - running your own stack costs ~12-15% of vendor per-resolution pricing at that scale, which pays for the engineers maintaining the harness and evals.