Guardrails for a 25-Tool Clinical Agent

A 25-tool LangGraph agent answering natural-language questions over Electronic Health Record data is, on paper, a straightforward agentic system: route the query, call the right tools, synthesize an answer. In a healthcare setting the "straightforward" part is doing a lot of quiet work, because the cost of a wrong answer isn't a bad Yelp recommendation — it's a wrong claim about a resident's care. The interesting engineering problem was never "can the agent answer the question." It was "can the agent be trusted to either answer correctly or visibly refuse."

Splitting the agent instead of growing one

The first design decision was to not build one agent that does everything. A single agent juggling 25+ tools plus clinical reasoning has a large, ambiguous action space at every step, which is exactly the condition under which agents hallucinate tool calls or skip verification steps under time/token pressure.

Instead, the system is a dual-agent pipeline behind an LLM intent router:

User query
    │
    ▼
Intent Router (gpt-4o-mini, temperature=0)
    │
    ├── data query ──────► Data Agent (25 async SQL tools, 11 EHR domains)
    │
    └── clinical reasoning ► Reasoning Agent (4 extraction tools)

The router's only job is classification, run at temperature=0 so the same query routes the same way every time — no creative reinterpretation of "how many falls did resident X have last month" into something it isn't. Once classified, the query goes to a narrower agent with a much smaller, more coherent tool surface. The data agent's 25 tools are async SQL calls scoped to specific EHR domains (medications, vitals, incidents, care plans, and so on); the reasoning agent's 4 tools are for structured extraction from clinical text. Neither agent needs to reason about the other's tool space, which shrinks the space in which either can go wrong.

Forcing the first move

LLM agents left to their own devices will sometimes answer from parametric knowledge instead of calling a tool — especially for questions that sound like they have an obvious answer. That's the highest-risk failure mode in this system: a plausible-sounding clinical answer that was never actually grounded in the resident's real data.

The fix is structural rather than a prompt request: the agent's first step is forced to be a tool call. It is not permitted to emit a final answer as its first action, full stop. This closes off an entire class of "the model just knew the answer" hallucinations, because there is no code path where the first LLM output is an unverified answer.

Blocking on the way out, not just steering on the way in

Prompt instructions ("don't make unsafe clinical recommendations") are necessary but not sufficient — they're a request, not a guarantee, and the failure mode that matters most is the tail case a prompt didn't anticipate. So the system also runs an output blocklist: a post-generation check on the agent's final response before it reaches the user, scanning for unsafe clinical recommendation patterns and blocking or rewriting the response if it matches. This is a deliberately dumb, deterministic layer sitting after a non-deterministic one — the LLM guardrail steers behaviour probabilistically, and the blocklist is the hard floor underneath it that doesn't care how confident the model was.

Memory that doesn't grow the context forever, or leak between sessions

The agent needs conversational memory — clinicians ask follow-up questions ("and what about the week before that?") that only make sense with prior turns in view. Two constraints shaped how that memory works:

1. Cost and latency don't allow the full transcript to ride along on every turn. Memory is PostgreSQL-persisted with sliding-window LLM summarisation — older turns get compressed into a running summary instead of staying verbatim in context, so a long conversation doesn't linearly grow the prompt (and the token bill) with every exchange.
2. Persistence has to be per-conversation and auditable, not an in-memory cache that disappears on restart or bleeds across sessions. PostgreSQL as the memory store means conversation state survives deploys and is queryable for review.

Streaming and the PHI audit trail

Two more pieces round out the production shape of the system, both less about the model and more about operating it responsibly:

• Real-time SSE streaming so responses appear token-by-token instead of after a multi-second silent wait — this matters more in a clinical workflow than it sounds, because a blank screen for several seconds reads as "did this break," and that erodes trust in the tool independent of answer quality.
• PHI audit logging on every data access. Every one of the 25 SQL tools logs what protected health information it touched and for whom, independent of whether the agent's final answer used that data. The audit trail isn't a security afterthought bolted on top — it's a logging call inside the same tool boundary the agent calls through, so there's no path to data access that skips it.

What actually made this trustworthy

None of the individual pieces here are exotic — intent routing, forced tool calls, output filtering, summarised memory, audit logging. What made the system trustworthy in a clinical setting was refusing to rely on any single one of them. The prompt asks the model to behave well; the forced-first-tool-call rule removes an entire failure mode structurally; the blocklist catches what slips through both; the audit log means every data touch is reviewable after the fact regardless of what the agent said. Layered, boring, deterministic safeguards around a non-deterministic core — that's the part that scales past the first demo.