Putting an LLM Inside a Workflow: Three Runtime Designs

The same durable spine - prepare → context → model → validate → persist - with one variable in the middle: who owns the loop.

The question I keep getting isn’t should I wrap my LLM in a workflow? The answer to that is always yes. Every model in production needs trusted input, scoped context, a validation step that catches hallucinations, and a durable record of what happened. A workflow gives you retries, timeouts, signals, and an audit trail - no matter how much freedom you grant the model inside it.

The real question is: what goes inside the workflow?

And there’s really only one axis that matters - who computes the context, and who decides what happens next? Deterministic code, or the model? Three designs sit along that axis. They share the same spine:

prepare ─► context ─► model ─► validate ─► persist

Any trigger - webhook, API, schedule, chat message - enters stage one. Any durable outcome - a message, document, commit, PR - exits stage five. What changes is what happens in the middle.

Design 1: LLM as a Function

Code is the brain. The model is a lookup table that sometimes reasons.

prepare ─► context ─► respond ─► validate ─► persist ─► DONE

  context: code-computed · respond: one LLM call, answer out
  no tools · no iteration · no side effects
  loop: none · safety: post-hoc verify, flag for review

Here’s the thing about this design: the workflow decides the entire trajectory before the model ever gets called. Code pulls the data, filters it, assembles context, hands it to the model. The model produces exactly one structured response. Validation happens afterward - an independent check, a confidence score, a human review flag - but nothing blocks.

Want more capability? Write more workflow code. One narrow job, executed reliably, at high volume and low cost.

Where this fits in SRE:

Alert triage notes. An alert fires. Code pulls recent deploys, related dashboards, runbook excerpts, past similar incidents. One LLM call produces a “likely cause + first three checks” note that lands in the incident channel. If it’s wrong, the on-call still acts - a bad note is harmless.

Postmortem first drafts. Incident closes. Code assembles the timeline from alerts, deploys, Slack threads, Jira, metrics snapshots. One call produces a structured draft. Humans edit and own the final.

Change-request risk summaries. A change ticket gets filed. Code pulls diff stats, affected services, recent incident history. One call produces a risk paragraph and suggested reviewers.

Use this when: you know both the input shape and the output shape, context is retrievable deterministically, and a wrong answer has zero blast radius.

Design 2: LLM as Bounded Specialists

The model proposes. Code disposes. The workflow owns every loop.

admit ─► context ─► specialists ─► decide ─► publish ─► DONE

  admit: policy gate (allow / deny / approve)
  context: pinned snapshot (immutable input)
  specialists: fixed LLM roles, each looping ≤N rounds
    DRAFTER ⇄ REVIEWER ── revise on findings
    PRODUCER ⇄ CHECKS ── revise on sandbox / CI failures
    AUDITORS ×3 ── parallel, independent
  decide: policy ── proceed / revise / human / stop
  LLM proposes · code disposes

The pipeline has a fixed shape. Each LLM has one role and one typed contract. A drafter emits a design. An independent reviewer critiques it with specific findings. A producer emits a change set - say, a diff. Auditors review the exact commit in parallel.

Between each pair of stages sits a deterministic gate. That might be a sandbox materializing a workspace and running allow-listed commands. A real git commit. A CI result pinned to that specific commit. A policy decision.

The model iterates on content. It sees reviewer findings and revises. But it never chooses what happens next. Never runs a command. Never touches the source of truth. Code applies every artifact and decides whether it lands.

Agency over content. Zero agency over execution. I think that’s the right tradeoff for most production work.

Where this fits in SRE:

Policy-violation fix PRs. OPA or conftest flags a public S3 bucket or missing resource limits. Drafter, reviewer, and producer emit a Terraform or Kubernetes diff. terraform plan and policy checks gate it inside a sandbox. A draft PR opens. A human merges - the pipeline never does.

CVE and dependency bumps. A scanner finds a vulnerable base image or library. Specialists produce the version bump and any required code changes. CI plus policy checks gate the exact commit. PR lands with rollback notes attached.

Config-drift remediation. A drift detector reports live state diverging from git. Specialists decide whether git or live is correct and produce the reconciling change. Dry-run gates it; destructive diffs force a human-review decision by policy.

Use this when: the output artifact shape is known and has objective verification - a diff, a plan, a test suite - but the correct content needs iteration and independent review before it can land.

Design 3: LLM as an Agent

The model is the intelligence. The workflow is the cage.

prepare ─► context ─► observe⇄DECIDE⇄ACT ─► validate ─► DONE

  observe ─► DECIDE ─► ACT
  tools: shell · http · files · browser · APIs
  risky action? ─► WAIT_FOR_APPROVAL ─► resume
  loop: model-owned, open · context: gathered by the model
  safety: sandbox boundary · approval before side effects

The agent receives tools, not answers. It observes its environment, decides what to do, acts - queries a metric, reads a log, lists pods, fetches a trace - then observes the result. No fixed sequence. The workflow’s job is to contain that open loop: provide an isolated environment, intercept risky actions with a durable WAIT_FOR_APPROVAL signal that survives restarts, and validate whatever comes out.

Adding a new capability requires no new workflow code. Just new tools and a new system prompt. The harness is general-purpose.

Where this fits in SRE:

Incident investigation. An alert gets acknowledged. The agent receives read-only observability tools - metrics, logs, traces, kubectl get/describe - inside a sandbox. It forms and tests hypotheses on its own. CPU spike after the 14:05 deploy? Which pods? What changed in that deploy? Error rate by endpoint? It produces a root-cause narrative with evidence links. An approval gate fires only if it wants a write action - restart, rollback, scale.

Cost-anomaly forensics. A billing spike lands. The agent explores the cost explorer, tags, usage patterns, deploy history across accounts. Identifies the driver, proposes a rightsizing plan.

Capacity and noisy-neighbor hunts. Latency regresses on a shared node. The agent queries per-pod metrics, CPU throttling data, recent scheduling changes. Finds the offender, quantifies the blast radius, proposes limit or quota changes as a gated action.

Use this when: even the path to the goal is unknown. No fixed pipeline can predict which queries to run. You need the model to choose the next step, and you need to contain its exploration rather than prescribe it.

Side by side

1  function, no loop:
prepare ─► context ─► respond ─► validate ─► persist

2  specialists, bounded workflow loop:
admit ─► context ─► DRAFTER ⇄ REVIEWER ─► decide ─► publish

3  agent, open model loop:
prepare ─► context ─► observe⇄DECIDE⇄ACT ─► validate ─► DONE

Notice how the safety design inverts as capability grows. Design 1 relies on post-hoc verification. Design 2 uses gated iteration with pinned artifacts. Design 3 requires pre-hoc approval before side effects. More freedom demands more containment.

The decision framework

Ask one question: how well can you specify the task before the model starts?

Fully specified input and output? Design 1. The question shape is known, the answer shape is JSON, retrieval is deterministic.

Specified output artifact but unspecified correct content? Design 2. You know a diff or PR must come out and you have objective checks - plan, CI, policy - that can gate it. The loop converges.

Unspecified path to the goal? Design 3. No pipeline can predict which queries or actions will be needed. Contain the exploration instead of prescribing it.

Why they all share the same durable spine

All three run inside the same durable workflow substrate. That’s not incidental - it’s the reason each design is actually production-viable rather than just a clever idea.

History and audit. Every handoff is recorded. In Design 2, a run can be replayed as a conversation between specialists from history alone. In Design 3, the approval signal survives crashes.

Bounded cost. Token budgets, revision limits, concurrent-work caps - these are workflow-level counters, not per-call heuristics. The workflow sees the whole run, not just the current step.

Human in the loop without extra infrastructure. WAIT_FOR_APPROVAL is a workflow signal, not a queue or a polling job. It survives crashes with exactly the same guarantees as every other step.

Gradual adoption. The same worker infrastructure hosts all three. A team starts with Design 1 for a narrow answering task, graduates to Design 2 for remediation PRs, and reserves Design 3 for open-ended investigations. No second runtime required.

The spine stays constant. Only the middle - the agency you grant the model - moves.

Three designs, one axis: who owns the loop - no one, the workflow, or the model.

Companion piece: Durable agent sessions with the OpenAI SDK, Docker, and Temporal - how one concrete build landed on the third runtime.