AI Systems & Infrastructure
Grounded AI Assistant — RAG on Private Infrastructure
Most website chatbots are a system prompt and an API key, and they confabulate under pressure — inventing offices, headcounts, rates, and client names whenever a visitor asks something the prompt doesn't cover. We rebuilt ours as a retrieval system: a knowledge base on our own server, semantic search over it, and a hard rule that the model may only state facts it was handed. When the knowledge base can't answer, the assistant says so.
Stack
Concepts
How it works
The whole flow, traced from your first tap.
- 01
Vercel · Next.js API
Ask a question
The latest message becomes the search query. Follow-ups like "what about for Walmart?" retrieve poorly alone, so the previous question is folded into the query used for search — while the model still sees the whole conversation.
App RouterConversation context - 02
Cloudflare Tunnel
Reach the knowledge base
The site calls a private service over a tunnel that dials out from our hardware, so no inbound ports are ever opened. A two-second timeout and a circuit breaker mean a server reboot costs a visitor nothing — after three failures the site stops calling and degrades quietly.
Outbound-only tunnelBearer auth2s timeout - 03
Private server
Search the knowledge base
The question is embedded locally — no per-query API cost, and nothing leaves the machine. Two searches run in parallel, semantic and keyword, and are combined by reciprocal rank fusion, which ranks by position rather than raw score because cosine similarity and text rank aren't on the same scale.
Ollamapgvector HNSWPostgres full-textReciprocal rank fusion - 04
Private server
Decide whether to answer at all
The refusal decision uses raw semantic similarity, not the fused rank — fusion happily crowns the least-bad of twenty irrelevant passages and reports a healthy-looking number. Below the measured floor, the service returns no passages and records the question as a gap worth writing about.
Measured relevance floorKnowledge-gap logging - 05
Vercel · Next.js API
Answer, or admit the gap
Passages returned means a closed-book prompt containing zero company facts, so every claim must come from the retrieved text. Nothing returned means a prompt that instructs an honest "I don't have that." Either way, citations are checked against what was actually retrieved and invented ones are stripped before the reply goes out.
Closed-book promptCitation validation - 06
Private server
Keep the knowledge current
A background job pulls the content repository, skips any document whose hash is unchanged, and re-embeds only what moved. A database advisory lock keeps a scheduled run and a manual rebuild from overwriting each other, and content is indexed only once explicitly marked confirmed.
Git-backed contentContent hashingAdvisory locks
The problem
We benchmarked nine inexpensive language models against our own site prompt. Asked whether we had an office in Austin and how many employees we had — details the prompt never mentioned — every single model invented an answer. One went further and fabricated an hourly rate along with a Fortune 500 client we have never worked with. Cheap models don't hallucinate randomly; they fill silence with whatever is plausible. For a business, that means a chatbot confidently misrepresenting you to a prospect.
What we built
A retrieval service running on our own hardware. Company knowledge lives as version-controlled markdown; a background job chunks it, generates embeddings locally, and indexes it into Postgres. When a visitor asks something, the site retrieves the most relevant passages and hands the model those passages plus a closed-book instruction: answer from this, or say you don't know. Answers carry citations back to the source document.
Making silence detectable
The hardest part is knowing when the knowledge base has nothing useful. We set a relevance threshold by measurement rather than by guesswork — scoring deliberately absurd questions against real content and reading the actual distribution. That mattered: the value that seemed sensible in advance turned out to refuse nothing at all, because the embedding model didn't span the range we assumed. Below the measured floor, the service returns an explicit refusal and no passages, and logs the question as a knowledge gap. Those gaps become a prioritized list of documentation worth writing.
Built to fail safely
Retrieval runs locally; answer generation stays with a hosted model. If the local server reboots, the assistant degrades to its previous behavior instead of breaking, guarded by a two-second timeout and a circuit breaker. Content reaches the assistant only when explicitly marked confirmed, so an unreviewed draft can't be quoted to a visitor. And any citation the model invents is stripped server-side before the answer is sent, because a fabricated source is worse than none — it makes an unsupported claim look verified.
Outcome
The same questions that produced invented offices and fictional clients now return either a cited fact or an honest "I don't have that, let me connect you with the team." The knowledge base is markdown in a repository, so updating what the assistant knows is a pull request, not a redeploy — and the whole index rebuilds from scratch on any machine with a clone and one command.
Interested in something similar?
Tell us what you need and we'll figure out how to ship it.