Reranking in RAG: why a cross-encoder second pass fixes most bad answers

In a packed college auditorium
In a packed college auditorium

Originally published at https://pranjulrathour.scult.in/blog/reranking-cross-encoders-rag. That copy is the canonical version and gets updates first.

Vector search compares a query embedding to passage embeddings that were computed separately, so it never actually reads the query and the passage together. That is why it can rank a chunk about "refund policy for cancelled orders" above the one about "refund timelines" when a user asks how long refunds take. A reranker fixes exactly this gap.

What a cross-encoder does differently

A cross-encoder takes the query and one candidate passage as a single input and outputs a relevance score. Because it attends across both texts at once, it catches negations, specific entities and conditions that a bi-encoder blurs. The cost is that it must run once per candidate, so you never rerank the whole index — only the top 20 to 50 results from the first-stage retriever.

Pranjul Rathour
Pranjul Rathour

The two-stage pipeline

  1. Retrieve broadly: hybrid dense + BM25 search returns the top 30 candidates (see hybrid retrieval explained).
  2. Rerank precisely: score each candidate with the cross-encoder, keep the top 5.
  3. Gate: if the best reranked score is below a threshold, refuse instead of answering (see what a confidence gate is).
  4. Generate with citations from the surviving chunks only.

What it changed in RAG.NextUpgrad

Adding a reranker was the step that made the confidence gate trustworthy. Before it, first-stage scores were noisy enough that a threshold either blocked good answers or let weak ones through. Reranked scores separate relevant from irrelevant far more sharply, so the gate has something honest to measure. The platform still streams the answer token by token; the reranking pass adds a short pause before the first token, which users accept in exchange for fewer wrong answers.

Cost and latency, honestly

A small cross-encoder on CPU scores 30 passages in a few hundred milliseconds. On a free-tier server that is noticeable but fine; on GPU it is negligible. The memory footprint matters more than speed on small hosts — I covered that trade-off in the free-tier memory-budget playbook.

On the mic
On the mic

When to skip it

  • Your corpus is tiny and questions are simple — first-stage retrieval already puts the right chunk on top.
  • You have hard latency limits under 300 ms and no GPU.
  • Your evaluation set shows recall@5 above 95% without reranking. Measure before adding machinery.

If your RAG app gives confident wrong answers, add a reranker before you touch the prompt. In my experience it is the highest-leverage component after chunking.

From my carousels
7 Levels of RAG Apps
7 Levels of RAG Apps, slide 17 Levels of RAG Apps, slide 2
7 Levels of RAG Apps, slide 37 Levels of RAG Apps, slide 4
Full carousel on Instagram and LinkedIn.
Pranjul Rathour
Pranjul Rathour
GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor
I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave.
Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges.
On stage, at hackathons and on campus
Presenting to a room
Presenting to a room
Pranjul Rathour
Pranjul Rathour
Pranjul Rathour, GenAI engineer, Kanpur
Pranjul Rathour, GenAI engineer, Kanpur

Comments

Post a Comment

Popular posts from this blog

Forming a hackathon team: roles, skills and the mistake most teams make

Image
At an Integral Startup Foundation hackathon Originally published at https://pranjulrathour.scult.in/blog/hackathon-team-roles-and-formation . That copy is the canonical version and gets updates first. Team TechVerse — Raman Shukla, Saksham Gupta and me — won three first prizes in 2025 not because we were the strongest coders in the room, but because we had stopped overlapping. When I mentor teams before a hackathon, formation is the first thing I fix. In a packed college auditorium The four roles Builder — owns the core feature and the demo path. Usually backend or the AI piece. Interface — owns what the judge sees: the front end, the flow, the seeded data. Storyteller — owns the problem statement, the pitch, the slides and the Q&A plan; also does user research in hour one. Integrator — keeps the pieces working together, manages the repo, runs the demo rehearsals. In a team of three, the builder or interface person takes this. The mistake Five friends who all write R...
Read more

Hello from Kanpur: what I build, and what I'll write about here

Image
On the mic Originally published at https://dev.to/pranjulrathour/hello-from-kanpur-what-i-build-and-what-ill-write-about-here-f2j . That copy is the canonical version and gets updates first. I'm Pranjul Rathour, a GenAI engineer from Kanpur, India. This is the first post on this blog, so here is who is writing, what I have actually shipped, and what you can expect if you follow along. What I build Between February and July 2026, at Next Upgrad Web Solutions, I designed, built and shipped five production AI applications end to end, alone, and deployed all of them with Docker, CI and observability: RAG.NextUpgrad : a retrieval-augmented generation service with hybrid dense plus BM25 retrieval, cross-encoder reranking, streaming cited answers, and a confidence gate that refuses to call the model when the evidence is weak. It runs in about 220 MB of memory, inside a free 512 MB instance. FineTune Studio : fine-tune open models with QLoRA in a browser. Upload a dataset, validate ...
Read more

I built 15 free tools, 1,211 prompts and a 50,000-skill library — here's what's inside tools.scult.in

Image
Carousel · 12 slides I built every part of tools.scult.in myself — 15 browser tools, 1,211 version-stamped prompts, a 50,000+ agent-skills library, a public MCP server, and 40 retro photo prompts where I'm the face in every example. All of it free. No signup, no ads, no upload step. Swipe through for what's inside and why I built it this way. Link in the last slide. Slide 1 of 12 Slide 2 of 12 Slide 3 of 12 Slide 4 of 12 Slide 5 of 12 Slide 6 of 12 Slide 7 of 12 Slide 8 of 12 Slide 9 of 12 Slide 10 of 12 Slide 11 of 12 Slide 12 of 12 Pranjul Rathour GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave. Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email: pranjulrathour41@gmail.com Portfolio & ...
Read more