---
title: "Rebuilding a Support Agent That Failed at Scale"
section: "Case studies"
canonical_url: "https://leverge.ai/case-studies/support-agent-fintech-containment"
topic: "AI support agent case study"
published: "2026-05-20"
updated: "2026-07-26"
publisher: "Ailoitte Technologies Private Limited"
---

# Rebuilding a Support Agent That Failed at Scale

A US lending platform had built an AI support agent internally that resolved tickets correctly in every demo and failed on roughly half of real ones. The cause was not the model — it was that the agent had never been tested against the client's actual ticket distribution, and it had no access to live loan and payment state. We rebuilt it around an evaluation set drawn from 1,400 historical transcripts, added live account retrieval, and reached containment that held as volume scaled.

## Key takeaways

- The agent's failure was a measurement failure — nobody had tested it against the real ticket distribution before launch.
- Roughly two-thirds of the client's ticket volume required live account state, which the original agent could not read at all.
- Building the evaluation set from 1,400 real transcripts took one week and reshaped every subsequent engineering decision.
- Containment was deliberately reported alongside satisfaction and reopen rate, which caught one early regression that containment alone hid.
- Escalation quality, not containment, is what turned the support team from sceptics into advocates.

## The number that ended the argument

The most useful week of this engagement produced no agent code at all.

Before it, the team had spent two months in a recurring disagreement: engineering
believed the agent was close, support believed it was unusable, and neither position was
falsifiable. Prompt changes shipped weekly with no way to tell whether they helped.

The evaluation set ended that. Scored against 1,400 adjudicated real tickets, the agent
was 51% correct, and the failures were concentrated in a way that pointed directly at the
cause. From that point the disagreement was about priorities rather than about facts.

## The access problem nobody had named

The category breakdown showed something the team had not articulated: about two-thirds of
their ticket volume was about one customer's specific situation — this payment, this
loan, this account — and the agent could only read help-centre articles.

It had been given a knowledge base and asked to answer questions that required a
database. No amount of prompt work closes that gap. Adding live retrieval from the loan
servicing and payment systems was the single largest quality improvement in the rebuild.

## Excluding two categories was a result, not a failure

During adjudication, two ticket types turned out to have no consistent correct answer —
the client's own experienced agents resolved them differently from each other, based on
judgement that had never been written down.

We recommended leaving both with humans. There is nothing for a system to optimise toward
when the target is undefined, and an agent trained to imitate inconsistent decisions
produces inconsistent decisions faster. The client's containment ceiling dropped as a
result, and the quality of what was contained went up.

## Why containment was never reported alone

We instrumented containment, satisfaction on contained tickets, reopen rate at seven days,
and escalation accuracy from the first day of the ramp.

That mattered in week eight, when a retrieval change lifted containment by three points
while quietly pushing the reopen rate up. On containment alone it looked like an
improvement. On the four numbers together it was obviously a regression, and it was
reverted the same day.

## Frequently asked questions

### Why did the original agent fail?

Two reasons that compounded. It had been validated against a small set of hand-picked example tickets that did not reflect the real distribution, so the long tail was entirely untested. And it only had access to help-centre content, while about two-thirds of the actual ticket volume was about a specific loan, payment or account state it could not read. It was structurally unable to resolve most of what arrived.

### How long did the rebuild take?

Ten weeks from the start of scoping to full production traffic, including one week building the evaluation set and three weeks on the retrieval and integration layer. The rollout itself was staged over the final three weeks, ramping from 10% of eligible tickets upward as the numbers held.

### What was the single most valuable step?

Building the evaluation set. It took a week of pulling transcripts and adjudicating correct outcomes with the client's support lead, and it changed almost every engineering decision that followed — including revealing that two ticket categories the client wanted automated had no consistent correct answer and should stay with humans.

---

Source: https://leverge.ai/case-studies/support-agent-fintech-containment — Leverge