Case study
Getting a Clinical RAG System Approved Internally
An accurate clinical retrieval tool that clinicians refused to adopt, and the two design changes that got it approved.
- Healthcare
- United States
- 6 weeks
- A clinical documentation company
- Healthcare
- 6 weeks
What happened?
A healthcare client had a retrieval system that scored well on internal accuracy tests and that clinicians would not use. The blocker was not accuracy — it was that answers arrived as unsourced prose, and a clinician carrying liability for acting on an answer will not accept one they cannot verify. We rebuilt the retrieval layer for passage-level citation and added calibrated refusal, and the tool went from stalled to approved by the clinical review board in six weeks.
The situation
What they were dealing with
The client had built a retrieval system over clinical guidelines and internal protocols to help their documentation specialists answer coding and compliance questions. On their own test set it answered correctly 89% of the time. Adoption after three months was close to zero. The clinical review board had declined to approve it for wider rollout, and the stated reason was that they could not audit it: answers arrived as fluent prose with a document name attached, and there was no way to confirm which part of a forty-page guideline the answer came from. Separately, the system always produced an answer, which the board correctly identified as a liability — it meant a question the corpus did not cover would still receive a confident response.
What we did
The approach
We rebuilt the retrieval layer around two requirements the original design could not satisfy. First, passage-level citation: we re-chunked the corpus on document structure — sections, headings, table rows — rather than fixed token counts, retained parent context with each chunk, and threaded passage identifiers through generation so every sentence in an answer links to the specific paragraph supporting it. Second, calibrated refusal: we built a labelled evaluation set that included questions deliberately not covered by the corpus, scored refusal as a correct outcome for those, and enforced a grounding check that rejects any claim without a supporting retrieved passage. On the compliance side we moved inference into the client's own cloud tenancy with zero retention, enforced access at retrieval time against their identity provider, and added record-level access logging so the board could see who retrieved what.
Stack
- Claude via AWS Bedrock
- pgvector
- Elasticsearch
- Cohere Rerank
- Python
- LangFuse
Outcome
Results
- To clinical review board approval
- 6 weeks
- Answer sentences linked to a source passage
- 100%
- Refusal accuracy on questions outside the corpus
- 91%
- Access logging granularity for PHI
- Record level
- Answer accuracy, as a side effect of better chunking
- 89% → 92%
“REPLACE with an approved quote. Our clinicians would not accept an answer without a citation they could open. Once every claim linked to the exact paragraph, the conversation about approval took one meeting instead of three months.”
Accuracy was not the problem
This engagement is the clearest example we have of a system failing for a reason that does not appear in its metrics. The tool was 89% accurate. Adoption was near zero.
The gap is explained entirely by who carries the risk. A clinician who acts on an answer owns the consequence. An unsourced statement asks them to accept that on trust, which means the rational response is to verify it independently — and verifying it independently is slower than not using the tool at all.
So the system’s accuracy was, from the user’s point of view, irrelevant. What mattered was the cost of confirming an answer.
Document citations are not citations
The original system attached a document name to each answer. That sounds like citation and functionally is not: confirming a claim still meant opening a forty-page guideline and searching it.
Passage-level citation — the specific paragraph or table row, linked so it opens at that location — makes verification a single click. That is the entire difference, and it is what moved daily usage from a handful of people to the whole team.
Making it possible required re-chunking. Fixed-token chunks cut across sections and tables, so there was no stable passage to point at. Chunking on document structure, with parent context retained, gave us addressable units that were also meaningful on their own.
Refusal was the review board’s real concern
The board’s second objection was sharper than the first: a system that always answers will answer questions the corpus does not cover.
We built refusal into the evaluation set as a scored behaviour, including questions deliberately outside the corpus, and enforced a grounding check that rejects unsupported claims before they reach the user. Refusal accuracy reached 91%.
That number, more than the accuracy figure, is what the board approved on. A tool that admits its gaps is auditable. One that is fluent everywhere is not.
The accuracy gain was incidental
Answer accuracy moved from 89% to 92% during the rebuild, and we would not present that as the achievement. It was a side effect of better chunking — passages that arrive with their heading intact are simply easier to reason over.
The result that mattered was approval, and approval came from verifiability rather than from three points of accuracy.
Frequently asked questions
If the system was 89% accurate, why would clinicians not use it?
What exactly changed about the citations?
How was HIPAA handled?
Related services and sectors
- RAG developmentRetrieval systems that answer over your own data with a citation for every claim, and a retrieval score you can actually measure.
- Evaluation & observabilityEvaluation suites, regression gates and production tracing that turn AI quality from an opinion into a number you can defend.
- HealthcareClinical documentation, prior authorisation and knowledge retrieval built to the evidence and audit standards healthcare review boards require.
Next Step
Dealing with something similar?
A 30-minute technical call with an engineer who has shipped this before — not a sales qualification round. You leave with a feasibility read, a rough shape for the build, and an honest answer about whether it is worth doing at all.