Cutting LLM Token Cost 74% with Hybrid Retrieval

The easiest way to make an LLM feature "work" is to give the model everything and let it figure out what matters. Dump the whole patient chart into the context window, dump every guideline document, dump the last fifty messages of chat history. It works, in the sense that the model can usually find the answer somewhere in there. It also works out to be extremely expensive, noticeably slower, and — counter-intuitively — less accurate, because burying the relevant sentence inside four thousand irrelevant tokens makes it easier for the model to skim past it.

On the AgeCare platform, clinical entity extraction from free-text care notes was exactly this problem. Care workers write notes the way people write notes: unstructured, inconsistent, full of shorthand. Extracting structured clinical signals from that text needed retrieval, not context-stuffing — and the retrieval step is where the 74% token cost reduction actually came from.

Why single-method retrieval wasn't enough

Two retrieval approaches solve different problems, and neither is sufficient alone:

Vector search (pgvector, cosine similarity over text-embedding-3-small embeddings) is excellent at semantic matches — it'll surface a passage about "wandering at night" when the query is about "nocturnal exit-seeking behaviour," even though the exact words don't overlap. What it's bad at is precision on specific terms: drug names, dosages, ICD-style codes, exact phrases a clinician actually typed.

Lexical search (PostgreSQL's built-in tsvector/tsquery, i.e. BM25-style ranking) is the mirror image. It nails exact-term matches and rare vocabulary, but it has no concept of meaning — it won't connect "wandering" and "exit-seeking" unless both literally appear.

Clinical text needs both. A query like "recent falls with medication changes" has to match on the semantic concept of a fall and the literal names of specific medications. Pick one retrieval method and you systematically miss one half of the relevant evidence.

Fusing the two with Reciprocal Rank Fusion

Rather than trying to build one embedding space that's good at both jobs, the system runs both searches independently and fuses the rankings:

score(doc) = Σ 1 / (k + rank_i(doc))

for each retrieval method i the document appears in, with k a small constant (60 is the usual default, and what's used here) that discounts the tail of each ranked list. RRF has a property that matters a lot in practice: it doesn't require the two scoring scales to be comparable. Cosine similarity and BM25 scores live in completely different numeric ranges, and trying to weight-and-sum them directly means constant recalibration as the corpus changes. RRF sidesteps that by only caring about rank position, not raw score — a document ranked #1 by vector search and #1 by BM25 gets a high fused score regardless of what the underlying similarity numbers were.

The retrieval pipeline for a clinical query looks like this:

async def hybrid_retrieve(query: str, k: int = 60, top_n: int = 8):
    vector_hits = await vector_search(query, limit=25)   # pgvector, cosine
    bm25_hits = await bm25_search(query, limit=25)        # tsvector, ts_rank
 
    fused: dict[str, float] = {}
    for rank, doc in enumerate(vector_hits, start=1):
        fused[doc.id] = fused.get(doc.id, 0) + 1 / (k + rank)
    for rank, doc in enumerate(bm25_hits, start=1):
        fused[doc.id] = fused.get(doc.id, 0) + 1 / (k + rank)
 
    ranked_ids = sorted(fused, key=fused.get, reverse=True)[:top_n]
    return await fetch_documents(ranked_ids)

Both searches run over the same PostgreSQL instance — the vector index lives alongside a generated tsvector column on the same table, so there's no second datastore to keep in sync, no separate infrastructure to operate, and no consistency lag between the two indexes.

Chunking is half the retrieval problem

Fusion only helps if the underlying chunks are the right size and shape to begin with. The ingestion pipeline for guideline PDFs uses semantic chunking rather than fixed-size windows — splitting on topic and structural boundaries (headings, clinical sections) instead of "every 500 tokens," so a chunk is far more likely to contain one complete, coherent clinical idea instead of half of two unrelated ones. That matters twice: it makes the embeddings more semantically coherent (a vector for "half a paragraph about falls plus half a paragraph about medication timing" is a vector for neither), and it means a retrieved chunk is more likely to be exactly what the downstream extraction step needs, with less padding.

Where the 74% actually comes from

The cost reduction isn't from a cheaper model or a shorter prompt template — it's from precision. Before hybrid retrieval, the extraction step over-fetched broad context windows to be safe, on the theory that missing information was worse than including too much. After: the top 6–8 fused, semantically-chunked results reliably contain the relevant clinical signal, so the LLM call's input shrinks dramatically without losing recall. Fewer input tokens per call, multiplied across every clinical-note extraction running in production, is where the number comes from.

It's worth being honest about the trade-off: hybrid retrieval is more infrastructure than either method alone — two indexes, a fusion step, tuning k and top_n against real query traffic. For a low-volume feature that wouldn't be worth it. At production scale, on a cost-sensitive healthcare workload, it was the highest-leverage change in the pipeline.

What I'd check before reaching for this again

• Is retrieval actually the bottleneck, or is the prompt template just verbose? Profile token spend before assuming retrieval is the fix.
• Do queries genuinely need both lexical and semantic matching? If the domain vocabulary is small and consistent, BM25 alone might already be enough.
• Is the chunking strategy doing real work, or is it fixed-size windows with a semantic-sounding name? The quality ceiling of any retrieval system is set by what got indexed, not by how cleverly it's ranked at query time.