"Agentic AI" is easy to sell and hard to scope. Every second demo shows an agent browsing the web, writing code, and coordinating ten sub-agents — and then the slide deck ends before anyone discusses what happens when one of those agents writes the wrong row into a production database. This article covers a real deployment: a data-ops automation system for a client with large volumes of repetitive reconciliation work. We automated a meaningful portion of it, rejected three agent architectures that would have created more problems than they solved, and built the oversight structure that made the parts we did automate auditable and recoverable.
What we actually built — and what we did not
The client team was spending roughly 60% of engineering time on structured reconciliation work: matching transaction IDs across three export formats, filling standard report templates from verified source data, and routing completed files to the correct downstream folder based on rules that were documented but tedious to apply consistently. The patterns were clear, the success criteria were measurable, and errors were detectable by a downstream check.
We built a multi-step agent pipeline using tool-calling: one agent for format normalisation and ID matching, one for template population, one for routing with a confidence check. Each agent had a defined input schema, a defined output schema, and a defined set of tools it was allowed to call. Nothing acted on data it could not validate against the schema. Anything below a confidence threshold was routed to a human review queue rather than being auto-resolved.
We explicitly did not build agents for three tasks the client initially requested: exception handling for edge-case transactions, contract interpretation for ambiguous billing rules, and a "general assistant" that could be asked arbitrary data questions. All three had failure modes that were silent and consequential — the kind where a wrong answer looks right until it is audited three months later.
Where agents performed well
Structured back-office tasks with clear, measurable success criteria were the strongest fit. When the definition of "correct" can be stated as a rule — "field X in output matches field Y in source document, modulo these three known normalisation rules" — agents with tool access handle it reliably and produce verifiable audit trails.
Volume tasks where human review was a bottleneck rather than a quality gate benefited most from automation. The client was applying a 100% human review to template population that was correct 97% of the time. Moving to 100% automated with human review only on low-confidence outputs reduced review load by roughly 85% while maintaining the same error detection rate.
Routing decisions based on explicit rule sets were handled well and consistently. Humans applying the same routing rules showed 3-5% inconsistency on edge cases (different interpretations of the same rule). The agent applied the rules identically every time — and when the rules were wrong, the error was systematic and immediately identifiable rather than random and hard to trace.
The audit trail produced by the agent pipeline was better than the manual process it replaced. Every decision was logged with the input, the tool calls made, the confidence score, and the output. The manual process had no comparable record. Regulators reviewing a six-month sample found the automated output easier to audit than the manual records preceding it.
Where we said no — and why
Judgment calls with legal or financial exposure stayed human-led. When the correct answer depends on context that is not in any document the agent can retrieve — relationship history, informal agreements, regulatory interpretation specific to a jurisdiction — the agent should not be making the call. It can surface the relevant information and draft options; a person with authority should decide.
One-off exceptions were explicitly out of scope. Edge cases that appear rarely, require escalation to a domain expert, and have never appeared in training data are exactly the cases agents handle worst. Automating exception handling often means automating incorrect resolutions for the cases that matter most.
Anything where the failure mode is silent wrong data was rejected. If an agent makes an error that propagates through three downstream systems before being detected, the recovery cost exceeds any efficiency the automation provided. The automation bar for silent failure modes is higher: near-perfect accuracy with independent verification, not the same bar as human error rates.
Tasks requiring tacit organisational knowledge — understanding which exception to file versus which to escalate based on the counterparty involved, reading the subtext of a client relationship — remained human-led with agent assistance. Agents proposed drafts and surfaced relevant history; people signed off with authority and full context.
Guardrails, audit trails, and oversight structure
Every tool call was logged with the full request, response, and agent state. Not as a debugging convenience — as a compliance requirement. Regulators and future engineers need to understand exactly what the system did and why. "The agent decided" is not an acceptable audit response. "The agent called this tool with these inputs and received this output, which triggered this routing decision" is.
Rollback paths were designed before deployment. Every automated action that modified state had an inverse operation and a time window in which it could be reversed without manual intervention. The client team reviewed and approved the rollback design before the first production run. Knowing how to undo an agent action is as important as knowing how to trigger it.
Human review queues were sized for the expected exception rate plus a buffer. A confidence threshold that routes 5% of cases for human review requires a human queue that can clear 5% of daily volume without becoming a bottleneck. If the queue fills, it defeats the efficiency purpose of automation and creates a pressure to lower the confidence threshold — exactly the wrong direction for quality control.
The agent scope was contractually fixed for the first six months. Feature requests to expand agent authority — "can it also handle X?" — went through the same evaluation process as the original design: failure mode analysis, audit trail requirements, rollback design, and measurable success criteria. Scope creep in agentic systems without that process is how consequential errors happen outside the original guardrails.
What production-ready actually means for agentic systems
An agent is production-ready when the team can answer five questions without hesitation: What does it do? What is it not allowed to do? How do you know when it is wrong? How do you reverse an error? Who is accountable for the output? If any of those answers require reading code or asking the team that built it, the system is not ready.
Monitoring for agentic systems requires tracking more than standard service metrics. Tool call frequency, retry rates, confidence score distributions, and human review queue depth are the signals that indicate whether the agent is operating within expected parameters — or drifting toward a mode the original design did not anticipate.
The efficiency gains from agentic automation compound when the system remains trustworthy. Gains evaporate quickly if the team loses confidence in the output and re-introduces manual review across the board. Maintaining that trust requires consistent monitoring, fast response to quality signals, and a clear process for scope changes. The engineering discipline is ongoing — not a one-time build.
