DocuMind — evaluation-first RAG

A retrieval-augmented generation platform where every quality claim is measured — and a CI gate blocks any pull request that makes retrieval worse.

FastAPI · Gemini embeddings (768d) · PostgreSQL/pgvector · FlashRank reranker · golden-set eval harness · GitHub Actions CI gate · Docker

The premise

Every RAG demo answers questions about documents. Almost none of them can tell you how well. DocuMind is built around that gap: it grounds every answer in retrieved passages, cites its sources, refuses when the answer isn't in the corpus — and proves all of it with a hand-validated golden dataset, a three-metric evaluation harness, and a GitHub Actions gate that fails any pull request whose retrieval drops below a committed baseline. The question-answering is table stakes. The measurement is the project.

The system

The backend is typed Python on FastAPI, with deliberately synchronous endpoints running in FastAPI's threadpool — the I/O is blocking, and blocking calls inside async def would freeze the event loop. Retrieval is two-stage: dense cosine search in PostgreSQL with pgvector, accessed through psycopg 3 and a connection pool, pulls the top-12 candidates; a FlashRank cross-encoder — a lightweight local ONNX model, no torch, in keeping with the small-footprint constraint — rescores them; and Reciprocal Rank Fusion merges the reranked and dense orders into the final top-k. Documents are embedded with Google's gemini-embedding-001 at 768 dimensions — Matryoshka-truncated from 3072 and re-normalized to unit length — using task-type asymmetry: documents as RETRIEVAL_DOCUMENT, queries as RETRIEVAL_QUERY. Generation is Gemini 2.5 Flash behind a strict grounding prompt; the evaluation side (answer generation and the LLM judge) runs on DeepSeek. The whole system is dockerized — app plus pgvector, schema auto-created on first boot, healthcheck-gated — and runs end-to-end from a clean clone with one command.

It didn't start here. The first version was a naive ChromaDB baseline with a local embedding model; I re-architected it so persistence and search live in one durable Postgres system and dropped the torch/CUDA stack entirely, leaving an app light enough for a small cloud instance.

The answer key

You can't measure retrieval without ground truth. The evaluation corpus is the FastAPI framework's tutorial documentation — 40 Markdown files, chunked into 667 passages (size 500, overlap 50) — and over it I hand-built a 100-question golden dataset: 71 single-hop, 14 multi-hop, and 15 deliberately unanswerable questions that test whether the system fabricates or declines.

Every gold answer is anchored to verified source text — verified programmatically, by a validator I wrote that checks each answer actually appears in the file it claims to come from. Its first run flagged 88 of 100 questions as failures. That number is the lesson: when a check fails on nearly everything, the check is broken, not the world. The real culprit was a Markdown-normalization mismatch between clean snippets and backtick-laden source; fixing it left 6 genuinely wrong entries — paraphrases that had slipped in — which I corrected by hand to a final 0. The same normalization logic is shared by the validator and every metric, so "found" means the same thing everywhere.

The numbers

measured on the validated golden dataset; no LLM in the loop — pure text-matching arithmetic

The multi-hop row is low, and it's reported anyway — because diagnosing it produced the best engineering in the project. Multi-hop recall@3 came back at an alarming 0.14; rather than assume the retriever was broken, I built an inspector that traced every multi-hop miss back to the database and classified it: in every single case the correct passage was retrieved but ranked below the cutoff. A ranking problem, not a data or retrieval-quality failure — which turned a scary number into an evidence-backed case for a reranker, with a baseline waiting to be beaten.

The fix

The first attempt confirmed a classic tradeoff: feeding the cross-encoder's order straight through helped multi-hop but hurt single-hop — a reranker can disrupt a first stage that is already strong. Rather than trade one question type for the other, I fused the two signals with Reciprocal Rank Fusion, so the dense ranking and the reranker each keep their say and neither can unilaterally sink a result. The candidate count was tuned the same way: 12 candidates beat 20, because a wider net hands the cross-encoder more distractors than signal. Measure the problem, diagnose the cause, apply the standard fix, tune it with evidence — the same loop the rest of the project runs.

same golden dataset, same arithmetic — only the retriever changed

Multi-hop recall@10 went 0.36 → 0.57, a 58% relative improvement on exactly the diagnosed weakness, while single-hop held — flat at 0.97 at k=10, within three points at the tightest cutoff.

On the generation side, the hallucination guard held on all 15 unanswerable cases — zero fabricated answers, including on topically-adjacent retrievals — and answer faithfulness scored a 4.82/5 mean, with 95% (74/78) of answered questions at 4 or above. Faithfulness is a hand-rolled, single-call LLM-as-judge: chosen over the RAGAS framework because multi-call judging was impractical under API rate limits, and hand-rolling kept the metric explainable and calibratable. The judge was spot-checked, not yet formally calibrated against human labels, so it's used directionally — and it stayed the same model across every run, because a metric is only comparable to itself if the instrument stays constant.

The gate

The centerpiece: on every pull request, GitHub Actions spins up a pgvector database, loads pre-computed corpus embeddings from a committed SQL fixture (db/fixtures/chunks_seed.sql) instead of re-embedding anything, runs the recall suite, and fails the build if retrieval regresses below the committed baseline. Faithfulness runs alongside as a non-blocking report. A change that quietly makes retrieval worse cannot be merged.

The gate earned its keep immediately — by going red on a false alarm. Its first real pull request failed on recall@3; investigation showed embedding non-determinism at the tightest cutoff (recall@10 held steady, so retrieval hadn't actually regressed) plus a baseline that had been measured outside the CI environment. I recalibrated instead of deleting: gate on the stable cutoffs (recall@5 and @10) and re-measure the baseline where it runs. Telling "bad metric" apart from "bad code" — without blindly trusting or dismissing the gate — is the judgment the whole project practices.

That recalibration was triage; the durable fix came after. I froze the corpus embeddings into the committed fixture CI now loads — every run starts from identical vectors instead of re-embedding, which makes CI faster and cheaper and makes recall deterministic run-to-run, removing the jitter that caused the false alarm at its source. With the noise gone, recall@3 came back: the gate now enforces all three cutoffs — recall@3, @5, and @10 — against a baseline measured in the CI environment.

When the reranker landed, the gate moved with it: CI now runs the two-stage path, and the committed baseline was ratcheted up to the improved numbers. The improvement is enforced, not just reported — 0.57 is the new floor a pull request has to clear.