Autonomous LLM agents: from prototype to production

2024 was the year of spectacular demos and production disappointments. AutoGPT and other open-loop agents fascinated Twitter, then failed as soon as they were given real tasks with a budget and an SLA. The verdict is clear: an agent that arms itself with vague objectives, loops without guardrails, and burns 200 K tokens to answer an email is not a product. It's a lab experiment.

Three patterns nonetheless survived contact with reality. First the ReAct pattern (reason, act, observe), which structures the model's thinking into explicit steps and remains the foundation of most serious agents. Then structured tool use, with native function calling at OpenAI and tool use at Anthropic — a model's ability to reliably invoke a typed function transformed perceived quality. Finally the agent + RAG coupling, where retrieval becomes one tool among others rather than a frozen preprocessing stage.

On the other hand, large-scale multi-agent coordination (three to ten agents talking to each other) fell flat in production. Compounded latency, exploding costs, endless confirmation loops: on our client cases, a three-agent scheme cost 4 to 6 times more than a single well-orchestrated agent, for lower quality. The lesson: one well-tooled agent beats ten agents holding a meeting.

The tipping point came in late 2024 with Claude 3.5 Sonnet and then GPT-4o, which made tool use reliable enough (> 95% success on strict JSON schemas) to dare run production. By mid-2025, Claude Sonnet 4.5 and Opus 4.5 pushed that threshold so far that the agent is no longer the weak link — the orchestration around it is.

The ecosystem consolidated around four building blocks. LangGraph established itself as the standard for orchestration graphs: typed nodes, explicit transitions, persisted state, checkpoints. You can finally design an agent the way you design a state machine — test it, replay it, inspect it. It's verbose, but it's what holds up in production. In parallel, vendor SDKs (Claude Agent SDK, OpenAI Assistants v2) offer a shorter integration when you accept the lock-in.

Anthropic added an important layer with Computer Use: a model that sees the screen and clicks. Promising for automating legacy systems without APIs, but to be reserved for tightly controlled contexts — an agent moving the mouse in production remains a risk. For classic cases (query a database, call an API, write a file), a classic tooled agent is faster, more traceable, and cheaper.

On the durable-execution side, Temporal and Restate have become the natural companions of agents. The promise: survive a process crash, a network timeout, an orchestrator upgrade. An agent that runs for 12 minutes, calls 8 tools, and dies on the 9th call because of a redeploy must be able to resume — not start over. Temporal does this with enterprise maturity; Restate, lighter, wins on deployment simplicity.

The target stack we deploy for clients in 2025: LangGraph for the graph, Temporal or Restate for durability, a semantic cache in front (GPTCache or Helicone caching), and a dedicated observability layer (Langfuse or Arize). It's verbose, but every block answers a real production failure we have witnessed at least once.

Tool use has become the primitive that separates demos from systems. Two concrete advances flipped it: structured outputs via JSON schema (OpenAI structured outputs, Anthropic tool use with strict JSON schema) and parallel tool calls. The model no longer merely proposes a call — it proposes five calls in parallel when the task allows it, which divides p95 latency by three on research workflows.

Concretely, between a Claude Sonnet 4.5 agent and a GPT-4o agent on the same internal benchmark (client-file analysis, 7 available tools, 50 cases): Sonnet 4.5 completes 88% of cases in under 12 steps, versus 74% for GPT-4o, with an invalid-tool rate of 1.2% versus 3.8%. Opus 4.5 pushes to 94% success but triples the token cost. The right trade-off on our French B2B cases is currently Sonnet 4.5 by default, escalating to Opus for long, high-stakes tasks.

Three rules have served us well. First, type every tool with a strict JSON schema — no vague optional fields, no short descriptions. The model reads the description as a contract. Next, expose small composable tools rather than one polymorphic mega-tool: a `searchUsers`, a `getUserById`, an `updateUserStatus` beat a `manageUsers(action, payload)`. Finally, return structured errors (`{ error_code, message, retryable }`) instead of a string — the model makes far fewer mistakes on a typed code.

The classic trap remains over-tooling. The more tools an agent has, the worse it chooses. Beyond 12 exposed tools, the correct-tool selection rate drops by 15 to 20 points. The countermeasure: precede the agent with a lightweight router (a small LLM or a classifier) that selects a relevant subset of tools based on intent, then lets the main agent work in a reduced space.

A production agent without dedicated observability is an agent you will debug over the phone with an unhappy client. Classic APM tools (Sentry, Datadog) aren't enough: they see the final error, not the path. You need an agent-aware layer that traces every step, every tool call, every token.

Langfuse has become our open-source reference: tree-structured traces, cost per run, latency per node, prompt comparison. Arize and Helicone play in the same league with different angles (automatic evaluation at Arize, caching proxy at Helicone). The bare minimum: trace every model call (prompt, completion, model, tokens, latency, cost), every tool call (input, output, status), and every graph transition.

The metrics that matter in agent production are not the demo ones. The global success rate is nearly useless — it masks drift. What to track: the average number of steps per task (an agent going from 8 to 14 steps over two weeks is drifting), token cost per successful task (not per attempted task), invalid-tool rate per model, and above all the rate of tasks interrupted by the user (a sign the agent is going off the rails).

Concretely, on an internal support agent deployed at a client, we watched cost per ticket climb from €0.18 to €0.47 over three weeks without anyone complaining. The answers were still correct, but the agent had become chatty, multiplied unnecessary verifications, and called the same tool twice for safety. Without dedicated observability, the drift would have been caught at the next budget review — too late. The rule we apply: any alert on abnormal cost or steps must trigger a human review within 48 h.

The absolute, non-negotiable rule must be carved into the code: an agent never executes a destructive action without explicit human approval. Deleting a record, emailing a client, charging a card, altering a database schema — every irreversible action goes through a confirmation gate. This is not a configuration option, it is a system invariant.

Concretely, every tool is annotated with a risk level (`read`, `write_reversible`, `irreversible`). `irreversible` tools are wrapped in a gate that suspends execution, notifies a human (Slack, email, dedicated interface), and waits for a signed confirmation. Temporal handles this suspension elegantly: the workflow sleeps, the state persists, the human answers whenever they want. If nobody answers within 24 h, the task is cancelled — not executed by default.

The second pillar is sandboxed tool execution. An agent that can execute code (because it must, say, transform a CSV file) runs in a sandbox: ephemeral Docker container, gVisor for kernel isolation, network cut off by default except an explicit allowlist, read-only filesystem except a working directory. The sandbox's output is schema-validated before being fed back into the agent. The marginal cost is real (~300 ms of container start) but security wins.

On the audit side, every agent execution is logged: initiating user, objective, steps, tools invoked, parameters (anonymised if PII), outputs, final decision. The trace is retained for 12 months minimum and can be replayed. On a real incident (an agent that almost sent the wrong follow-up template to 200 clients), that trace is what let us understand in 20 minutes what had happened and harden the guardrail for the future.

Anthropic's Model Context Protocol (MCP), announced in late 2024, is starting to establish itself as the missing interoperability layer. An MCP server exposes tools, resources, and prompts under a standardised protocol; any MCP-compatible client (Claude Desktop, Cursor, IDEs, agent runtimes) can consume them. For the first time, writing a tool for an agent becomes as runtime-independent as writing an HTTP route. Our bet for 2026: most B2B SaaS vendors will expose a native MCP server the way they expose a REST API today.

Standardised agent-to-agent communication is the second workstream. Today, making two agents from different vendors collaborate takes ad hoc glue. The work on the A2A Protocol (Google) and MCP extensions aims to define a common vocabulary: how an agent declares its capabilities, how another one requests work, how results are typed. We see this as the future of event-driven architectures: an orchestrator agent, specialist agents reachable on demand, typed contracts between them. But maturity won't arrive before mid-2026 at best.

The convergence with traditional BPM and orchestration tools (Camunda, Temporal, enterprise n8n) is accelerating. Business workflows historically coded in BPMN will be able to integrate agentic steps: an LLM node that negotiates with a supplier, drafts a commercial proposal, classifies a complaint. What looked like two worlds (deterministic orchestration on one side, probabilistic agents on the other) is converging into a hybrid discipline where, for each step, you choose between an explicit rule and delegation to a model.

Above all, agent ops is becoming the new MLOps. Just as MLOps absorbed the discipline of shipping models (evaluation, drift, A/B, monitoring), agent ops must absorb the specifics of agentic systems: multi-step tracing, quality control by simulation, action guardrails, auditability, human escalation. Teams that invested in serious MLOps in 2023-2024 are a year ahead — and those that skipped it will have to catch up on both blocks at once. The 2026 plan we recommend to clients: tool up agent ops before scaling the number of agents. One well-observed agent beats ten opaque ones.