Containment is the wrong metric on its own
Containment — the share of tickets closed without a human — is the number every
support automation vendor leads with, and on its own it is dangerously easy to
game. An agent that confidently answers everything, correct or not, posts
excellent containment. The cost shows up later as reopened tickets, chargebacks and
churn that nobody attributes back to the automation.
So we report containment against three companions from day one: customer
satisfaction on contained tickets, reopen rate within seven days, and escalation
accuracy — how often the agent correctly recognised that it should hand over. An
agent at 55% containment with stable satisfaction is a better system than one at
75% with a rising reopen rate, and only the four numbers together show that.
Most tickets are about a specific customer, not a general policy
The common failure of first-generation support bots is that they only know your
help centre. But the majority of real tickets are not “what is your return
policy” — they are “where is my order”, “why was I charged twice”, “I need to
change the address on order 88213”.
Answering those requires reading live state: the order, the shipment, the
subscription, the billing history, the previous tickets. An agent without that
access can only ever paraphrase documentation, which is why customers experience it
as an obstacle between them and a person.
Design escalation for the human, not for the metric
Whether your support team accepts the system is decided almost entirely by what an
escalation looks like when it lands.
A raw transcript with a note saying the bot could not help is worse than no
automation, because the human now reads a conversation before starting work. A
prepared case — the issue in two lines, the retrieved policy, the account state,
what was attempted, what is recommended and why it stopped — means the human
resolves it faster than if they had picked it up cold.
The second version takes real engineering effort and is the reason support teams
end up advocating for the agent rather than working around it.