three ways to put an LLM inside a Temporal workflow

Companion to running durable agent sessions with OpenAI SDK, Docker, and Temporal.

Whenever you run an LLM in production for backend/infra work, you want a durable workflow (like Temporal) around it for retries, timeouts, audit logs, and human approvals.

The real design question is: what actually goes inside the workflow, and who owns the loop?

Every setup shares the same outer pipeline:

prepare ─► context ─► model ─► validate ─► persist

Depending on whether deterministic code or the model computes the context and decides the next step, you end up with one of three designs.

Design 1: Single LLM call inside a deterministic workflow

prepare ─► context ─► respond ─► validate ─► persist ─► DONE

  context: code-computed · respond: one LLM call, answer out
  no tools · no iteration · no side effects
  loop: none · safety: post-hoc verify, flag for review

Here your workflow code does 90% of the work upfront. Code fetches the logs, metrics, or diffs, stuffs them into a prompt, makes one LLM call for a structured JSON/markdown output, validates the schema, and saves the result. The model has zero tools and never loops.

Where this works well in SRE/infra:

• Alert triage notes: Alert fires -> workflow fetches recent deploys, dashboard links, and runbook sections -> 1 LLM call posts a “likely cause + first 3 things to check” note into the Slack incident channel.
• Postmortem first drafts: Incident closes -> code pulls the PagerDuty timeline, deploy events, and Slack thread -> 1 LLM call generates the markdown draft for the on-call engineer to edit.
• Change-request summaries: PR/ticket opens -> code pulls diff stats + recent outages on that service -> 1 LLM call writes a short risk summary.

When to use Design 1: You already know the exact input data to fetch and the exact output shape you want, and a wrong answer can’t break prod.

Design 2: Bounded specialist loops (workflow owns the loop)

admit ─► context ─► specialists ─► decide ─► publish ─► DONE

  admit: policy gate (allow / deny / approve)
  context: pinned snapshot (immutable input)
  specialists: fixed LLM roles, each looping ≤N rounds
    DRAFTER ⇄ REVIEWER ── revise on findings
    PRODUCER ⇄ CHECKS ── revise on sandbox / CI failures
    AUDITORS ×3 ── parallel, independent
  decide: policy ── proceed / revise / human / stop
  LLM proposes · code disposes

Here the model gets to iterate on its output, but the workflow still controls the loop and runs all commands.

For example:

1. A Drafter prompt proposes a patch or plan, and a Reviewer prompt critiques it (max N rounds).
2. A Producer prompt emits a git diff.
3. Deterministic workflow code applies that diff inside a sandbox and runs terraform plan, unit tests, or OPA policy checks.
4. If the check fails, the workflow feeds the error output back to the Producer for another round.
5. Once checks pass, the workflow opens a draft PR for a human to merge.

Notice the model never gets a raw shell tool and never decides what stage runs next. It only emits text/diffs, and deterministic code runs the sandbox checks between stages.

Where this fits in SRE/infra:

• Policy-violation fix PRs: OPA/conftest flags a public S3 bucket or missing k8s resource limits -> workflow loops Drafter/Producer until terraform plan + policy checks pass in a sandbox -> opens a draft PR.
• CVE / dependency bumps: Scanner flags a vulnerable library -> workflow generates the version bump + code fixes, runs the test suite in CI against that commit, and opens the PR with rollback notes.
• Terraform drift reconciliation: Drift detector spots live AWS state diverging from git -> workflow generates the reconciling diff, gates it on terraform plan, and routes any destructive plan to a human approval gate.

When to use Design 2: You know what artifact needs to come out at the end (e.g. a working PR diff) and you have an automated test/plan command to verify it, so you just need a bounded retry loop until the check passes.

Design 3: Full agent loop in a sandbox (model owns the loop)

prepare ─► context ─► observe⇄DECIDE⇄ACT ─► validate ─► DONE

  observe ─► DECIDE ─► ACT
  tools: shell · http · files · browser · APIs
  risky action? ─► WAIT_FOR_APPROVAL ─► resume
  loop: model-owned, open · context: gathered by the model
  safety: sandbox boundary · approval before side effects

Here you don’t know upfront which logs, metrics, or files need to be inspected. So you put the agent inside an isolated sandbox, give it read-only tools (kubectl get/describe, log search, PromQL, file search), and let the model decide what to run next based on what it finds.

The Temporal workflow acts as the outer cage around the agent:

• provisions and destroys the ephemeral container
• enforces hard caps on max turns, tokens, and runtime
• intercepts any write/risky tool call and pauses the workflow on a durable WAIT_FOR_APPROVAL signal until a human approves it

Where this fits in SRE/infra:

• Incident investigation: Alert fires -> agent runs read-only PromQL queries, checks pod logs, inspects the 14:05 deploy diff, tests hypotheses on its own, and posts a root-cause writeup with links to the exact logs/graphs. Only triggers an approval signal if it wants to roll back or restart pods.
• AWS cost-spike forensics: Billing alert fires -> agent pokes around Cost Explorer, tags, and deploy history across accounts to find which service/job caused the spike.
• Noisy-neighbor / latency debugging: Latency jumps on a k8s node -> agent inspects per-pod CPU throttling, cgroup metrics, and recent scheduling events to find the offending pod.

When to use Design 3: You don’t even know the investigation path upfront, so a hardcoded pipeline won’t work.

Comparing the three side by side

1  function, no loop:
prepare ─► context ─► respond ─► validate ─► persist

2  specialists, bounded workflow loop:
admit ─► context ─► DRAFTER ⇄ REVIEWER ─► decide ─► publish

3  agent, open model loop:
prepare ─► context ─► observe⇄DECIDE⇄ACT ─► validate ─► DONE

The nice thing about building all three on top of the same Temporal worker setup is that you don’t need three different platforms. Start with Design 1 whenever you can, use Design 2 when you need code/diff generation verified by CI, and save Design 3 for open-ended debugging where you genuinely need the model to explore.

For the concrete implementation details of Design 3, check out the companion post: running durable agent sessions with OpenAI SDK, Docker, and Temporal.