Case study
Rebuilding a Support Agent That Failed at Scale
A support agent that passed every rehearsal and failed on real tickets, rebuilt around an evaluation set drawn from the client's own transcripts.
- Financial services
- United States
- 10 weeks
- A Series B consumer lending platform
- Financial services
- 58%
What happened?
A US lending platform had built an AI support agent internally that resolved tickets correctly in every demo and failed on roughly half of real ones. The cause was not the model — it was that the agent had never been tested against the client's actual ticket distribution, and it had no access to live loan and payment state. We rebuilt it around an evaluation set drawn from 1,400 historical transcripts, added live account retrieval, and reached containment that held as volume scaled.
The situation
What they were dealing with
The client had built an AI support agent in-house over about three months. In demonstrations it handled every ticket the team put in front of it. On live traffic it resolved correctly around half the time, and the support team had begun routing around it — checking and redoing its work rather than trusting it, which made the automation a net cost. Leadership was two weeks from switching it off. Nobody could say what the actual accuracy was, which ticket types it failed on, or whether the previous week's prompt changes had made it better or worse, because no measurement existed beyond anecdote and a containment counter that treated every closed ticket as a success.
What we did
The approach
We started with measurement rather than with the agent. Over one week we pulled 1,400 historical tickets, sampled to match the real category distribution including the awkward tail, and adjudicated the correct outcome for each with the client's support lead. Scored against that set, the existing agent came in at 51% correct — and the breakdown showed the failures concentrated in tickets requiring account state, which the agent had no access to. We then rebuilt in three layers: hybrid retrieval over help content with citations, live integration into the loan servicing and payment systems so the agent could read the specific customer's situation, and an action layer with value thresholds and human approval on anything touching money. Two ticket categories were deliberately excluded after the adjudication exercise showed the client's own agents resolved them inconsistently, meaning there was no correct answer to optimise toward.
Stack
- Claude
- LangGraph
- pgvector
- Zendesk API
- Temporal
- LangFuse
- Python
Outcome
Results
- Correct resolution, measured on the same evaluation set
- 51% → 58%
- Customer satisfaction on contained tickets
- Flat
- Reduction in median first-response time
- 74%
- Deliberately excluded as unsuitable for automation
- 2 categories
- Scoping to full production traffic
- 10 weeks
“REPLACE with an approved quote. The thing that changed was having a number. We had been arguing about whether the agent was working for two months, and after that first week we could just look.”
The number that ended the argument
The most useful week of this engagement produced no agent code at all.
Before it, the team had spent two months in a recurring disagreement: engineering believed the agent was close, support believed it was unusable, and neither position was falsifiable. Prompt changes shipped weekly with no way to tell whether they helped.
The evaluation set ended that. Scored against 1,400 adjudicated real tickets, the agent was 51% correct, and the failures were concentrated in a way that pointed directly at the cause. From that point the disagreement was about priorities rather than about facts.
The access problem nobody had named
The category breakdown showed something the team had not articulated: about two-thirds of their ticket volume was about one customer’s specific situation — this payment, this loan, this account — and the agent could only read help-centre articles.
It had been given a knowledge base and asked to answer questions that required a database. No amount of prompt work closes that gap. Adding live retrieval from the loan servicing and payment systems was the single largest quality improvement in the rebuild.
Excluding two categories was a result, not a failure
During adjudication, two ticket types turned out to have no consistent correct answer — the client’s own experienced agents resolved them differently from each other, based on judgement that had never been written down.
We recommended leaving both with humans. There is nothing for a system to optimise toward when the target is undefined, and an agent trained to imitate inconsistent decisions produces inconsistent decisions faster. The client’s containment ceiling dropped as a result, and the quality of what was contained went up.
Why containment was never reported alone
We instrumented containment, satisfaction on contained tickets, reopen rate at seven days, and escalation accuracy from the first day of the ramp.
That mattered in week eight, when a retrieval change lifted containment by three points while quietly pushing the reopen rate up. On containment alone it looked like an improvement. On the four numbers together it was obviously a regression, and it was reverted the same day.
Frequently asked questions
Why did the original agent fail?
How long did the rebuild take?
What was the single most valuable step?
Related services and sectors
- AI agent developmentAutonomous agents that execute a business process end to end, with the guardrails and evaluation infrastructure that keep them trustworthy at volume.
- Evaluation & observabilityEvaluation suites, regression gates and production tracing that turn AI quality from an opinion into a number you can defend.
- Customer support agentResolves routine support tickets end to end using your own help content and order data, and escalates cleanly when it should not act.
- Financial servicesDocument intelligence, KYC review and customer operations built to the explainability and model risk standards financial regulators expect.
Next Step
Dealing with something similar?
A 30-minute technical call with an engineer who has shipped this before — not a sales qualification round. You leave with a feasibility read, a rough shape for the build, and an honest answer about whether it is worth doing at all.