DocuMind — evaluation-first RAG

A retrieval-augmented generation platform where every quality claim is measured — and a CI gate blocks any pull request that makes retrieval worse.

FastAPI · Gemini embeddings (768d) · PostgreSQL/pgvector · FlashRank reranker · golden-set eval harness · GitHub Actions CI gate · Docker

The premise

Every RAG demo answers questions about documents. Almost none of them can tell you how well. DocuMind is built around that gap: it grounds every answer in retrieved passages, cites its sources, refuses when the answer isn't in the corpus — and proves all of it with a hand-validated golden dataset, a three-metric evaluation harness, and a GitHub Actions gate that fails any pull request whose retrieval drops below a committed baseline. The question-answering is table stakes. The measurement is the project.

The system

The backend is typed Python on FastAPI, with deliberately synchronous endpoints running in FastAPI's threadpool — the I/O is blocking, and blocking calls inside async def would freeze the event loop. Retrieval is two-stage: dense cosine search in PostgreSQL with pgvector, accessed through psycopg 3 and a connection pool, pulls the top-12 candidates; a FlashRank cross-encoder — a lightweight local ONNX model, no torch, in keeping with the small-footprint constraint — rescores them; and Reciprocal Rank Fusion merges the reranked and dense orders into the final top-k. Documents are embedded with Google's gemini-embedding-001 at 768 dimensions — Matryoshka-truncated from 3072 and re-normalized to unit length — using task-type asymmetry: documents as RETRIEVAL_DOCUMENT, queries as RETRIEVAL_QUERY. Generation is Gemini 2.5 Flash behind a strict grounding prompt; the evaluation side (answer generation and the LLM judge) runs on DeepSeek. The whole system is dockerized — app plus pgvector, schema auto-created on first boot, healthcheck-gated — and runs end-to-end from a clean clone with one command.

It didn't start here. The first version was a naive ChromaDB baseline with a local embedding model; I re-architected it so persistence and search live in one durable Postgres system and dropped the torch/CUDA stack entirely, leaving an app light enough for a small cloud instance.

The answer key

You can't measure retrieval without ground truth. The evaluation corpus is the FastAPI framework's tutorial documentation — 40 Markdown files, chunked into 667 passages (size 500, overlap 50) — and over it I hand-built a 100-question golden dataset: 71 single-hop, 14 multi-hop, and 15 deliberately unanswerable questions that test whether the system fabricates or declines.

Every gold answer is anchored to verified source text — verified programmatically, by a validator I wrote that checks each answer actually appears in the file it claims to come from. Its first run flagged 88 of 100 questions as failures. That number is the lesson: when a check fails on nearly everything, the check is broken, not the world. The real culprit was a Markdown-normalization mismatch between clean snippets and backtick-laden source; fixing it left 6 genuinely wrong entries — paraphrases that had slipped in — which I corrected by hand to a final 0. The same normalization logic is shared by the validator and every metric, so "found" means the same thing everywhere.

The numbers

measured on the validated golden dataset; no LLM in the loop — pure text-matching arithmetic

The multi-hop row is low, and it's reported anyway — because diagnosing it produced the best engineering in the project. Multi-hop recall@3 came back at an alarming 0.14; rather than assume the retriever was broken, I built an inspector that traced every multi-hop miss back to the database and classified it: in every single case the correct passage was retrieved but ranked below the cutoff. A ranking problem, not a data or retrieval-quality failure — which turned a scary number into an evidence-backed case for a reranker, with a baseline waiting to be beaten.

The fix

The first attempt confirmed a classic tradeoff: feeding the cross-encoder's order straight through helped multi-hop but hurt single-hop — a reranker can disrupt a first stage that is already strong. Rather than trade one question type for the other, I fused the two signals with Reciprocal Rank Fusion, so the dense ranking and the reranker each keep their say and neither can unilaterally sink a result. The candidate count was tuned the same way: 12 candidates beat 20, because a wider net hands the cross-encoder more distractors than signal. Measure the problem, diagnose the cause, apply the standard fix, tune it with evidence — the same loop the rest of the project runs.

same golden dataset, same arithmetic — only the retriever changed

Multi-hop recall@10 went 0.36 → 0.57, a 58% relative improvement on exactly the diagnosed weakness, while single-hop held — flat at 0.97 at k=10, within three points at the tightest cutoff.

On the generation side, the hallucination guard held on all 15 unanswerable cases — zero fabricated answers, including on topically-adjacent retrievals — and answer faithfulness scored a 4.82/5 mean, with 95% (74/78) of answered questions at 4 or above. Faithfulness is a hand-rolled, single-call LLM-as-judge: chosen over the RAGAS framework because multi-call judging was impractical under API rate limits, and hand-rolling kept the metric explainable and calibratable. The judge was spot-checked, not yet formally calibrated against human labels, so it's used directionally — and it stayed the same model across every run, because a metric is only comparable to itself if the instrument stays constant.

The gate

The centerpiece: on every pull request, GitHub Actions spins up a pgvector database, loads pre-computed corpus embeddings from a committed SQL fixture (db/fixtures/chunks_seed.sql) instead of re-embedding anything, runs the recall suite, and fails the build if retrieval regresses below the committed baseline. Faithfulness runs alongside as a non-blocking report. A change that quietly makes retrieval worse cannot be merged.

The gate earned its keep immediately — by going red on a false alarm. Its first real pull request failed on recall@3; investigation showed embedding non-determinism at the tightest cutoff (recall@10 held steady, so retrieval hadn't actually regressed) plus a baseline that had been measured outside the CI environment. I recalibrated instead of deleting: gate on the stable cutoffs (recall@5 and @10) and re-measure the baseline where it runs. Telling "bad metric" apart from "bad code" — without blindly trusting or dismissing the gate — is the judgment the whole project practices.

That recalibration was triage; the durable fix came after. I froze the corpus embeddings into the committed fixture CI now loads — every run starts from identical vectors instead of re-embedding, which makes CI faster and cheaper and makes recall deterministic run-to-run, removing the jitter that caused the false alarm at its source. With the noise gone, recall@3 came back: the gate now enforces all three cutoffs — recall@3, @5, and @10 — against a baseline measured in the CI environment.

When the reranker landed, the gate moved with it: CI now runs the two-stage path, and the baseline was re-measured on it — the floors the gate defends are the improved numbers. One honest boundary: single-hop and overall recall are hard-gated; multi-hop is reported on every run but doesn't block, because with 14 questions a single flip swings the score seven points — too noisy for a pass/fail. A gate you can't trust is worse than no gate.

The experiment that didn't ship

With ranking fixed, the obvious next move was hybrid search. Technical documentation is dense with exact identifiers — response_model, Depends, @app.get — and the folk wisdom says embedding models fumble exact-match terms, so adding BM25 keyword search, fused with dense retrieval via RRF, should lift recall. I implemented it and measured it against the same golden dataset: recall moved by ≤0.01 at every k. No meaningful change.

The per-question failure analysis explains why: the correct passages were already in the candidate set but ranked too low — a ranking problem, which the reranker had already addressed — not missing from it, the coverage problem BM25 solves. Dense retrieval over this corpus wasn't failing on exact-match terms; the embedding model handles identifier-heavy text better than the folk wisdom suggests.

So the complexity didn't earn its place, and the branch didn't merge — the negative result is documented in the repo instead of deleted. This is the eval harness cutting the other way: the same measurement that made the evidence-backed case for the reranker killed the hybrid stage. Without it, "added hybrid search" would have shipped as a résumé bullet — plus a tokenizer, an index, and a fusion step of pure maintenance burden for nothing. Multi-hop retrieval remains the honest weak point at 0.57 recall@10, and its remaining failures are ranking- and decomposition-shaped, not coverage-shaped — which points the next round of work at query decomposition, not more retrieval signals.

The deployment

The system runs live at documind.amr-mohammed.com, on an AWS EC2 instance (t3.small, Ubuntu 24.04) behind an Elastic IP for a stable address. It's the same Docker Compose stack as development — the FastAPI app, PostgreSQL/pgvector, and a Caddy reverse proxy. Caddy over nginx for one concrete reason: it obtains and auto-renews a free Let's Encrypt certificate on its own, so HTTPS costs zero manual certificate management. The corpus deploys from the same committed embedding fixture CI uses, so standing up a fresh instance makes zero embedding API calls. Only ports 22, 80, and 443 are open; Postgres is never exposed to the internet.

Uploads are disabled in the public demo (a DEMO_MODE flag): a public, unauthenticated ingest endpoint would burn API quota and mix strangers' documents into one shared corpus — session-scoped uploads are the planned fix. The demo offers three suggested questions, and one is a trap on purpose: "How do I connect FastAPI to MongoDB?" isn't answered anywhere in the corpus, so the system refuses it. Click the MongoDB question and watch it refuse — that refusal is the proof that answers come from retrieved documents, not the model's own knowledge.