Everyone has seen the RAG demo: upload a PDF, ask a question, get a plausible answer in under two seconds. Then real traffic arrives, documents change without warning, latency budgets shrink, and "plausible" stops being good enough at 2 a.m. when an ops team is mid-incident. This is the story of one domain-specific operations chatbot that we took from a notebook experiment to roughly 180,000 inferences per day — covering the architecture decisions that held, the retrieval tuning that mattered most, the observability layer we built around it, and the unglamorous choices that kept it running six months after launch without a permanent ML engineer on babysitting duty.

Why the demo lied

The operations team did not need a general-purpose assistant. They needed precise, citable answers about ingestion windows, internal metric names, SLA wording, escalation paths, and contract-specific exceptions — in exactly the language their own runbooks used. When a general LLM answered those questions, it did so confidently: plausible names, plausible values, plausible caveats. Almost none of them matched the actual documents. An ops engineer following a hallucinated escalation path during an incident is not a demo failure — it is a liability.

A vanilla LLM is not lying; it is completing a pattern. It does not know what it does not know, and it has no mechanism to signal uncertainty on specific domain facts versus general knowledge. That is not a prompt engineering problem alone — it is a retrieval problem, a governance problem, and a trust problem. Teams that discover hallucinations after go-live spend the following month manually reviewing outputs, which eliminates most of the efficiency the system was supposed to create.

We stopped treating the model as the product. The product was: accurate answers, cited sources, measurable quality at the query category level, fast recovery when documents changed, and a clear on-call path when quality drifted in ways the system itself could not catch. The model was one component inside that product. An important one — but still just one.

The first three weeks were spent defining what "correct" meant before writing a single line of retrieval code. Which question categories mattered most? Which errors were tolerable versus operationally dangerous? Which answers required a human sign-off before going out? Those answers determined chunk size, confidence thresholds, template policy, and the alert criteria — everything downstream that the team would eventually use to judge whether the system was healthy.

Retrieval architecture we actually shipped

We chose Postgres with pgvector as the vector store. Not because it is state-of-the-art for billions of documents, but because the client already ran Postgres in production, their infrastructure team could operate it, and debugging a bad retrieval result was straightforward: run a query, inspect the ranked chunks, identify the miss. At 180k queries per day on a corpus of a few thousand documents, purpose-built vector databases added operational complexity without meaningfully improving recall on the eval set we actually had.

Chunking strategy produced more quality gain than model swaps across the first month. We landed on 768-token chunks with 128-token overlap, weighted toward sentence boundaries to avoid cutting mid-clause. Smaller chunks improved precision for short factual queries; larger chunks helped for procedural questions that needed surrounding context. We tested both ends of the spectrum against the real query distribution — not a synthetic benchmark — and the 768-token split won consistently.

Embeddings came from a domain-adapted encoder evaluated on a hard-negatives pool: questions where two plausible chunks compete for the top slot. Hit rate is easy to inflate by retrieving many candidates; ranking quality on hard examples predicts production behaviour far more accurately. We measured both, optimised for the latter, and ran the eval before and after any index change.

Re-ranking made a meaningful difference for multi-part procedural questions. A lightweight cross-encoder over the top-10 candidates before passing context to generation added roughly 80ms to p95 latency — comfortably within the 1.5-second budget for this use case. Whether that cost is worth it depends entirely on the query mix. For this client, multi-hop procedural questions were common enough that re-ranking reduced the "partially right" category of answers by a measurable margin in our weekly eval.

 

Retrieval checklist before tuning generation

  • Instrument retrieval hit rate and empty-context responses separately from downstream answer quality.
  • Version the document index like code — documents get updated without anyone telling the ML team.
  • Profile query latency at your target chunk size before committing to an embedding model.
  • Build your eval set from real user questions, not questions you think users will ask.
  • Plan for "I do not know" as a first-class output, not an edge-case fallback.

Answer quality at scale

Query rewriting was the single highest-leverage improvement in the first three months. Users phrase questions the way they think, not the way documents are written. A raw query like "what is the SLA for critical tickets" misses half the relevant chunks because the document uses "Priority 1 resolution commitment" and similar variants. Rewriting the query before embedding lookup — using the same LLM in a short chain — cut retrieval misses by roughly a third on the eval set. If you tune one component after chunk size, make it query rewriting.

We kept human review on answer templates for regulated phrases. The LLM handled retrieval and draft generation; any response touching SLA commitments, escalation authority, or contractual language used approved template fragments rather than free generation. This is not distrust of the model — it is recognition that some sentences carry legal and operational weight that no retrieval confidence score can independently validate.

"I do not know" was designed as a first-class answer, not a last resort. When retrieval returned below a confidence threshold, the system said so clearly and offered to surface the closest document section as context. Users trusted the system more when it admitted uncertainty than when it hedged with a paragraph of caveats. That trust is what made 180k daily inferences operationally feasible — the team learned when to rely on output and when to verify directly, rather than treating every answer as suspect.

We ran a weekly eval pipeline against a fixed question set with known answers. Not a perfect scientific benchmark — but it caught the day a policy document update silently broke three query categories before any user noticed. Silent regressions are the most damaging kind because users stop trusting the system before anyone on the team knows something is wrong. Catching them early is worth more than achieving an impressive launch benchmark.

Citation links were surfaced in every answer, not just on request. Showing which document and which section supported the answer reduced follow-up verification time significantly — and it gave the team a direct path from a bad answer to the retrieval miss that caused it.

Keeping it alive in production

Production observability meant Prometheus for latency and error budgets, Grafana for dashboards the ops team actually opens without being reminded, and ELK for tracing bad sessions end to end. We did not build all three on day one. We built what the team could act on: inference latency, error rate by model version, and queue depth. Dashboards nobody opens are vanity; dashboards someone checks before stand-up are production infrastructure.

We added an isolation forest on CPU utilisation and queue depth after a near-miss six weeks after launch. Traffic doubled for a week following an internal product announcement, inference instances throttled, and the chatbot degraded to a crawl. The team found out from a Slack message, not an alert. The isolation forest — trained on two weeks of normal baseline — would have paged the on-call engineer 14 minutes earlier based on the exact pattern that developed. Two weeks of baseline was enough to catch that shape.

Alert lag went from "check the dashboard after stand-up" to under a minute for the signals that mattered. That result is not engineering magic. It is consistent metric labels, a single on-call channel, runbooks linked directly from Grafana panel titles, and a firm rule: no new alert without a corresponding runbook. Noisy alerts train the team to mute the channel. Muted channels mean incidents last 18 minutes. The silence is the problem, not the alert system.

Document index rotation was the operational task we most underestimated at the start. The policy team updated documents three times in the first month post-launch, each time without notifying the ML team. We built a lightweight webhook that triggered re-indexing and ran the eval suite on the diff before promoting the new index. That automation took one sprint to build and prevented the exact class of incident — an authoritative-sounding answer referencing an outdated policy — that damages user trust more than any latency problem.

Model version gating mattered more than we expected. We ran A/B splits on generation parameters against the eval set before promoting any change, including prompt updates. What felt like a minor wording improvement in the system prompt changed answer structure in ways that broke downstream parsing in two integrations. Treating prompt changes as deployments — with eval gates and rollback paths — saved two incidents that would otherwise have hit production.

What we would do differently

Instrument retrieval hit rate in week one, not week six. We had inference metrics from day one and retrieval metrics from the first incident. That gap cost two regression cycles that a single Prometheus counter per query would have caught automatically.

Build the document versioning webhook before launch, not after the first silent regression. The engineering cost was one sprint. The business cost of the alternative — one bad-answer incident referencing an outdated policy — was a two-week trust recovery effort with the operations team that used it.

Start with a narrower scope and a tighter eval loop. We launched with broad coverage across fifteen document categories and weak evals on most of them. Focusing on the five highest-traffic query types with strong evals would have shipped a more reliable product faster, with the remaining categories following in sprint-sized increments.

Define the "I do not know" threshold before the first user test, not after feedback arrives that the system is overconfident. Users calibrate their trust based on early interactions. A system that hedges appropriately from day one earns more long-term trust than one that sounds confident and turns out to be wrong.

None of these lessons are about model choice or embedding dimensions. They are about applying engineering discipline to ML systems consistently — the same habits that keep any production service healthy. The tooling exists. The discipline of applying it before the first incident is what separates a system running 180k queries a day from a notebook that worked great in demo.