# DocuMind — evaluation-first RAG

> A retrieval-augmented generation platform where **every quality claim is measured** — and a CI gate blocks any pull request that makes retrieval worse.

**Stack:** FastAPI · Gemini embeddings (768d) · PostgreSQL/pgvector · FlashRank reranker · golden-set eval harness · GitHub Actions CI gate · Docker

**Links:** [Live demo](https://documind.amr-mohammed.com) · [Source code](https://github.com/AmrMohamed17/Documind) · [HTML version](https://www.amr-mohammed.com/documind) · [Portfolio](https://www.amr-mohammed.com/)

**Headline metrics:**

- **0.97** single-hop recall@10 — held through reranking
- **0.57** multi-hop recall@10 — up from 0.36, reranked
- **15/15** unanswerable questions refused
- **4.82/5** mean answer faithfulness

## The premise

Every RAG demo answers questions about documents. Almost none of them can tell you how well. DocuMind is built around that gap: it grounds every answer in retrieved passages, cites its sources, refuses when the answer isn't in the corpus — and proves all of it with a hand-validated golden dataset, a three-metric evaluation harness, and a GitHub Actions gate that fails any pull request whose retrieval drops below a committed baseline. The question-answering is table stakes. The measurement is the project.

## The system

The backend is typed Python on FastAPI, with deliberately synchronous endpoints running in FastAPI's threadpool — the I/O is blocking, and blocking calls inside `async def` would freeze the event loop. Retrieval is **two-stage**: dense cosine search in **PostgreSQL with pgvector**, accessed through psycopg 3 and a connection pool, pulls the top-12 candidates; a **FlashRank cross-encoder** — a lightweight local ONNX model, no torch, in keeping with the small-footprint constraint — rescores them; and **Reciprocal Rank Fusion** merges the reranked and dense orders into the final top-k. Documents are embedded with Google's `gemini-embedding-001` at **768 dimensions** — Matryoshka-truncated from 3072 and re-normalized to unit length — using task-type asymmetry: documents as `RETRIEVAL_DOCUMENT`, queries as `RETRIEVAL_QUERY`. Generation is Gemini 2.5 Flash behind a strict grounding prompt; the evaluation side (answer generation and the LLM judge) runs on DeepSeek. The whole system is dockerized — app plus pgvector, schema auto-created on first boot, healthcheck-gated — and runs end-to-end from a clean clone with one command.

It didn't start here. The first version was a naive ChromaDB baseline with a local embedding model; I re-architected it so persistence and search live in one durable Postgres system and dropped the `torch`/CUDA stack entirely, leaving an app light enough for a small cloud instance.

## The answer key

You can't measure retrieval without ground truth. The evaluation corpus is the FastAPI framework's tutorial documentation — 40 Markdown files, chunked into **667 passages** (size 500, overlap 50) — and over it I hand-built a **100-question golden dataset**: 71 single-hop, 14 multi-hop, and 15 deliberately unanswerable questions that test whether the system fabricates or declines.

Every gold answer is anchored to verified source text — verified *programmatically*, by a validator I wrote that checks each answer actually appears in the file it claims to come from. Its first run flagged **88 of 100** questions as failures. That number is the lesson: when a check fails on nearly everything, the check is broken, not the world. The real culprit was a Markdown-normalization mismatch between clean snippets and backtick-laden source; fixing it left **6 genuinely wrong entries** — paraphrases that had slipped in — which I corrected by hand to a final **0**. The same normalization logic is shared by the validator and every metric, so "found" means the same thing everywhere.

## The numbers

Retrieval recall@k — dense-only first stage (the baseline):

| question type   | k=3      | k=5  | k=10     |
| --------------- | -------- | ---- | -------- |
| single-hop (71) | 0.93     | 0.93 | **0.97** |
| multi-hop (14)  | **0.14** | 0.29 | 0.36     |

*measured on the validated golden dataset; no LLM in the loop — pure text-matching arithmetic*

The multi-hop row is low, and it's reported anyway — because diagnosing it produced the best engineering in the project. Multi-hop recall@3 came back at an alarming 0.14; rather than assume the retriever was broken, I built an inspector that traced every multi-hop miss back to the database and classified it: **in every single case the correct passage was retrieved but ranked below the cutoff**. A ranking problem, not a data or retrieval-quality failure — which turned a scary number into an evidence-backed case for a reranker, with a baseline waiting to be beaten.

## The fix

The first attempt confirmed a classic tradeoff: feeding the cross-encoder's order straight through helped multi-hop but **hurt single-hop** — a reranker can disrupt a first stage that is already strong. Rather than trade one question type for the other, I fused the two signals with **Reciprocal Rank Fusion**, so the dense ranking and the reranker each keep their say and neither can unilaterally sink a result. The candidate count was tuned the same way: **12 candidates beat 20**, because a wider net hands the cross-encoder more distractors than signal. Measure the problem, diagnose the cause, apply the standard fix, tune it with evidence — the same loop the rest of the project runs.

Recall@k — dense baseline → two-stage (rerank + RRF):

| question type   | k=3         | k=5         | k=10            |
| --------------- | ----------- | ----------- | --------------- |
| single-hop (71) | 0.93 → 0.90 | 0.93 → 0.92 | 0.97 → 0.97     |
| multi-hop (14)  | 0.14 → 0.21 | 0.29 → 0.36 | **0.36 → 0.57** |

*same golden dataset, same arithmetic — only the retriever changed*

Multi-hop recall@10 went **0.36 → 0.57**, a 58% relative improvement on exactly the diagnosed weakness, while single-hop held — flat at 0.97 at k=10, within three points at the tightest cutoff.

On the generation side, the hallucination guard held on **all 15 unanswerable cases** — zero fabricated answers, including on topically-adjacent retrievals — and answer faithfulness scored a **4.82/5 mean**, with 95% (74/78) of answered questions at 4 or above. Faithfulness is a hand-rolled, single-call LLM-as-judge: chosen over the RAGAS framework because multi-call judging was impractical under API rate limits, and hand-rolling kept the metric explainable and calibratable. The judge was spot-checked, not yet formally calibrated against human labels, so it's used directionally — and it stayed the *same model across every run*, because a metric is only comparable to itself if the instrument stays constant.

## The gate

The centerpiece: on every pull request, GitHub Actions spins up a pgvector database, loads pre-computed corpus embeddings from a committed SQL fixture (`db/fixtures/chunks_seed.sql`) instead of re-embedding anything, runs the recall suite, and **fails the build if retrieval regresses** below the committed baseline. Faithfulness runs alongside as a non-blocking report. A change that quietly makes retrieval worse cannot be merged.

The gate earned its keep immediately — by going red on a false alarm. Its first real pull request failed on recall@3; investigation showed embedding non-determinism at the tightest cutoff (recall@10 held steady, so retrieval hadn't actually regressed) plus a baseline that had been measured outside the CI environment. I recalibrated instead of deleting: gate on the stable cutoffs (recall@5 and @10) and re-measure the baseline where it runs. Telling "bad metric" apart from "bad code" — without blindly trusting or dismissing the gate — is the judgment the whole project practices.

That recalibration was triage; the durable fix came after. I froze the corpus embeddings into the committed fixture CI now loads — every run starts from identical vectors instead of re-embedding, which makes CI faster and cheaper and makes recall **deterministic run-to-run**, removing the jitter that caused the false alarm at its source. With the noise gone, recall@3 came back: the gate now enforces **all three cutoffs** — recall@3, @5, and @10 — against a baseline measured in the CI environment.

When the reranker landed, the gate moved with it: CI now runs the two-stage path, and the baseline was re-measured on it — the floors the gate defends are the improved numbers. One honest boundary: single-hop and overall recall are hard-gated; multi-hop is reported on every run but doesn't block, because with 14 questions a single flip swings the score seven points — too noisy for a pass/fail. A gate you can't trust is worse than no gate.

## The experiment that didn't ship

With ranking fixed, the obvious next move was **hybrid search**. Technical documentation is dense with exact identifiers — `response_model`, `Depends`, `@app.get` — and the folk wisdom says embedding models fumble exact-match terms, so adding BM25 keyword search, fused with dense retrieval via RRF, should lift recall. I implemented it and measured it against the same golden dataset: **recall moved by ≤0.01 at every k**. No meaningful change.

The per-question failure analysis explains why: the correct passages were already *in* the candidate set but ranked too low — a **ranking** problem, which the reranker had already addressed — not *missing* from it, the **coverage** problem BM25 solves. Dense retrieval over this corpus wasn't failing on exact-match terms; the embedding model handles identifier-heavy text better than the folk wisdom suggests.

So the complexity didn't earn its place, and the branch didn't merge — the negative result is documented in the repo instead of deleted. This is the eval harness cutting the other way: the same measurement that made the evidence-backed case for the reranker killed the hybrid stage. Without it, "added hybrid search" would have shipped as a résumé bullet — plus a tokenizer, an index, and a fusion step of pure maintenance burden for nothing. Multi-hop retrieval remains the honest weak point at **0.57** recall@10, and its remaining failures are ranking- and decomposition-shaped, not coverage-shaped — which points the next round of work at **query decomposition**, not more retrieval signals.

## The deployment

The system runs live at [documind.amr-mohammed.com](https://documind.amr-mohammed.com), on an AWS EC2 instance (t3.small, Ubuntu 24.04) behind an Elastic IP for a stable address. It's the same Docker Compose stack as development — the FastAPI app, PostgreSQL/pgvector, and a Caddy reverse proxy. Caddy over nginx for one concrete reason: it obtains and auto-renews a free Let's Encrypt certificate on its own, so HTTPS costs zero manual certificate management. The corpus deploys from the same committed embedding fixture CI uses, so standing up a fresh instance makes **zero embedding API calls**. Only ports 22, 80, and 443 are open; Postgres is never exposed to the internet.

Uploads are disabled in the public demo (a `DEMO_MODE` flag): a public, unauthenticated ingest endpoint would burn API quota and mix strangers' documents into one shared corpus — session-scoped uploads are the planned fix. The demo offers three suggested questions, and one is a trap on purpose: *"How do I connect FastAPI to MongoDB?"* isn't answered anywhere in the corpus, so the system refuses it. **Click the MongoDB question and watch it refuse** — that refusal is the proof that answers come from retrieved documents, not the model's own knowledge.

## Status — active development

**Built and measured:** the pgvector/Gemini architecture, the 100-question validated golden dataset, the three-metric evaluation harness, the two-stage FlashRank + RRF reranker (multi-hop recall@10 0.36 → 0.57), a hybrid BM25 + dense stage — implemented, measured (recall moved ≤0.01), and deliberately *not* shipped — the CI quality gate enforcing the improved baseline on every pull request, and a [live public deployment](https://documind.amr-mohammed.com) on AWS (EC2 · Docker Compose · Caddy auto-HTTPS).

**Next:** query decomposition — splitting multi-hop questions into sub-queries — to lift multi-hop further, and session-scoped uploads for the public demo.

---

© 2026 Amr Mohammed · [amr-mohammed.com](https://www.amr-mohammed.com/) · amrm88289@gmail.com