How to evaluate a RAG system: recall, faithfulness and the questions that matter

Pranjul Rathour
Pranjul Rathour

Originally published at https://pranjulrathour.scult.in/blog/rag-evaluation-metrics-faithfulness-recall. That copy is the canonical version and gets updates first.

You cannot improve a RAG system you are not measuring, and "I asked it five questions and it seemed fine" is not measuring. Every serious change I made to RAG.NextUpgrad — hybrid search, reranking, the confidence gate — was justified by a number moving on a fixed evaluation set. Here is the setup, sized for a student or a small team.

On the mic
On the mic

Build the question set first

Collect 50 to 100 real questions. For each, record the passage (and page) that answers it, and the expected answer in one sentence. Include 10 to 20 questions the documents cannot answer — those test whether the system refuses correctly. This set is your most valuable asset; version it alongside the code.

Four metrics, in order of importance

  1. Retrieval recall@k — is the correct passage among the top k retrieved? If this is low, nothing downstream can save you. Fix chunking and retrieval before anything else.
  2. Faithfulness — does every claim in the answer appear in the retrieved context? Grade it with a rubric, by hand for the first hundred answers and with an LLM judge afterwards, spot-checking the judge.
  3. Refusal accuracy — on unanswerable questions, does the system say it does not know? On answerable ones, does it avoid refusing? A confidence gate is tuned entirely on this metric.
  4. Answer correctness — does the answer match the expected one? Useful, but it hides whether a correct answer came from the documents or from the model's memory.

A minimal evaluation loop

Run the set on every change — a script that indexes, queries, and writes a CSV is enough. Track recall@5, faithfulness rate, false-refusal rate and false-answer rate over time. When a change improves one metric and hurts another, you have a real decision to make instead of an argument.

Presenting to a room
Presenting to a room

Traps I fell into

  • Writing questions by reading the documents, which produces questions phrased exactly like the text. Real users paraphrase; add paraphrased versions.
  • Grading with the same model that generated the answer, without checking the judge. Judges are lenient on fluent prose.
  • Evaluating only happy-path questions. The refusal set is where a production system earns trust.

A confidence gate without an evaluation set is a guess with a threshold. With one, it becomes the most defensible feature in the product — I explain the gate itself in why my RAG platform says "I don't know".

From my carousels
7 Levels of RAG Apps
7 Levels of RAG Apps, slide 17 Levels of RAG Apps, slide 2
7 Levels of RAG Apps, slide 37 Levels of RAG Apps, slide 4
Full carousel on Instagram and LinkedIn.
Pranjul Rathour
Pranjul Rathour
GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor
I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave.
Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges.
On stage, at hackathons and on campus
Pranjul Rathour
Pranjul Rathour
Pranjul Rathour, GenAI engineer, Kanpur
Pranjul Rathour, GenAI engineer, Kanpur
In a packed college auditorium
In a packed college auditorium

Comments

Post a Comment

Popular posts from this blog

Forming a hackathon team: roles, skills and the mistake most teams make

Image
At an Integral Startup Foundation hackathon Originally published at https://pranjulrathour.scult.in/blog/hackathon-team-roles-and-formation . That copy is the canonical version and gets updates first. Team TechVerse — Raman Shukla, Saksham Gupta and me — won three first prizes in 2025 not because we were the strongest coders in the room, but because we had stopped overlapping. When I mentor teams before a hackathon, formation is the first thing I fix. In a packed college auditorium The four roles Builder — owns the core feature and the demo path. Usually backend or the AI piece. Interface — owns what the judge sees: the front end, the flow, the seeded data. Storyteller — owns the problem statement, the pitch, the slides and the Q&A plan; also does user research in hour one. Integrator — keeps the pieces working together, manages the repo, runs the demo rehearsals. In a team of three, the builder or interface person takes this. The mistake Five friends who all write R...
Read more

Hello from Kanpur: what I build, and what I'll write about here

Image
On the mic Originally published at https://dev.to/pranjulrathour/hello-from-kanpur-what-i-build-and-what-ill-write-about-here-f2j . That copy is the canonical version and gets updates first. I'm Pranjul Rathour, a GenAI engineer from Kanpur, India. This is the first post on this blog, so here is who is writing, what I have actually shipped, and what you can expect if you follow along. What I build Between February and July 2026, at Next Upgrad Web Solutions, I designed, built and shipped five production AI applications end to end, alone, and deployed all of them with Docker, CI and observability: RAG.NextUpgrad : a retrieval-augmented generation service with hybrid dense plus BM25 retrieval, cross-encoder reranking, streaming cited answers, and a confidence gate that refuses to call the model when the evidence is weak. It runs in about 220 MB of memory, inside a free 512 MB instance. FineTune Studio : fine-tune open models with QLoRA in a browser. Upload a dataset, validate ...
Read more

I built 15 free tools, 1,211 prompts and a 50,000-skill library — here's what's inside tools.scult.in

Image
Carousel · 12 slides I built every part of tools.scult.in myself — 15 browser tools, 1,211 version-stamped prompts, a 50,000+ agent-skills library, a public MCP server, and 40 retro photo prompts where I'm the face in every example. All of it free. No signup, no ads, no upload step. Swipe through for what's inside and why I built it this way. Link in the last slide. Slide 1 of 12 Slide 2 of 12 Slide 3 of 12 Slide 4 of 12 Slide 5 of 12 Slide 6 of 12 Slide 7 of 12 Slide 8 of 12 Slide 9 of 12 Slide 10 of 12 Slide 11 of 12 Slide 12 of 12 Pranjul Rathour GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave. Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email: pranjulrathour41@gmail.com Portfolio & ...
Read more