Chunking strategies for RAG: sizes, overlap and structure-aware splits

On the mic
On the mic

Originally published at https://pranjulrathour.scult.in/blog/rag-chunking-strategies-guide. That copy is the canonical version and gets updates first.

Most RAG quality problems I have debugged were chunking problems wearing a disguise. The model looked like it was hallucinating; the retriever was actually handing it half a table, or a paragraph whose subject lived in the previous chunk. Before you tune prompts or switch models, look at what your chunks actually contain.

Three rules that survive every project

  1. Chunk for the question, not the document. A chunk should be able to answer a realistic user question on its own. For policy documents that is often 300–500 tokens; for API references it can be a single function signature plus its description.
  2. Overlap is insurance, not a strategy. 10–15% overlap catches sentences that straddle a boundary. Larger overlaps mostly duplicate what the reranker sees and inflate your index.
  3. Never split inside a structure. Tables, code blocks, numbered procedures and headings with their first paragraph belong together. Splitting them produces chunks that are individually meaningless.

Structure-aware splitting in practice

In RAG.NextUpgrad the ingest step first detects structure — headings, list runs, table rows — and only then applies a size budget. A heading is prepended to every chunk beneath it, so a chunk that says "the limit is 5 requests per second" also carries "Section 4.2: Rate limits". That single trick did more for answer accuracy than any embedding model change.

Presenting to a room
Presenting to a room

For scanned PDFs, run OCR page by page and keep page numbers as metadata. In the OCR & Speech Workspace every chunk records its page, which is what makes citations like "page 14" possible in the answer.

How to pick a size without guessing

Build a small evaluation set first: 30–50 real questions with the passage that answers each. Index the corpus at three chunk sizes (say 256, 512 and 1024 tokens), run the questions, and measure how often the correct passage appears in the top 5. The size that wins on recall is your starting point; then check answer quality, because a chunk that retrieves well can still be too fragmentary to answer from.

Pranjul Rathour
Pranjul Rathour

Signs your chunking is wrong

  • Answers that are correct but cite the wrong section — the heading context is missing.
  • Numbers quoted without their unit or condition — a table was split across chunks.
  • The model says "the document does not specify" for facts you know are there — the fact straddles a boundary and neither half ranks.
  • Retrieval scores are uniformly high — chunks are so large that everything looks relevant.

Chunking is unglamorous, and it is the first thing I check when a student shows me a RAG demo that "sometimes works". Fix the chunks and the rest of the pipeline suddenly looks smarter than it is.

From my carousels
7 Levels of RAG Apps
7 Levels of RAG Apps, slide 17 Levels of RAG Apps, slide 2
7 Levels of RAG Apps, slide 37 Levels of RAG Apps, slide 4
Full carousel on Instagram and LinkedIn.
Pranjul Rathour
Pranjul Rathour
GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor
I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave.
Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges.
On stage, at hackathons and on campus
Pranjul Rathour, GenAI engineer, Kanpur
Pranjul Rathour, GenAI engineer, Kanpur
In a packed college auditorium
In a packed college auditorium
Pranjul Rathour
Pranjul Rathour

Comments

Post a Comment

Popular posts from this blog

Forming a hackathon team: roles, skills and the mistake most teams make

Image
At an Integral Startup Foundation hackathon Originally published at https://pranjulrathour.scult.in/blog/hackathon-team-roles-and-formation . That copy is the canonical version and gets updates first. Team TechVerse — Raman Shukla, Saksham Gupta and me — won three first prizes in 2025 not because we were the strongest coders in the room, but because we had stopped overlapping. When I mentor teams before a hackathon, formation is the first thing I fix. In a packed college auditorium The four roles Builder — owns the core feature and the demo path. Usually backend or the AI piece. Interface — owns what the judge sees: the front end, the flow, the seeded data. Storyteller — owns the problem statement, the pitch, the slides and the Q&A plan; also does user research in hour one. Integrator — keeps the pieces working together, manages the repo, runs the demo rehearsals. In a team of three, the builder or interface person takes this. The mistake Five friends who all write R...
Read more

Hello from Kanpur: what I build, and what I'll write about here

Image
On the mic Originally published at https://dev.to/pranjulrathour/hello-from-kanpur-what-i-build-and-what-ill-write-about-here-f2j . That copy is the canonical version and gets updates first. I'm Pranjul Rathour, a GenAI engineer from Kanpur, India. This is the first post on this blog, so here is who is writing, what I have actually shipped, and what you can expect if you follow along. What I build Between February and July 2026, at Next Upgrad Web Solutions, I designed, built and shipped five production AI applications end to end, alone, and deployed all of them with Docker, CI and observability: RAG.NextUpgrad : a retrieval-augmented generation service with hybrid dense plus BM25 retrieval, cross-encoder reranking, streaming cited answers, and a confidence gate that refuses to call the model when the evidence is weak. It runs in about 220 MB of memory, inside a free 512 MB instance. FineTune Studio : fine-tune open models with QLoRA in a browser. Upload a dataset, validate ...
Read more

I built 15 free tools, 1,211 prompts and a 50,000-skill library — here's what's inside tools.scult.in

Image
Carousel · 12 slides I built every part of tools.scult.in myself — 15 browser tools, 1,211 version-stamped prompts, a 50,000+ agent-skills library, a public MCP server, and 40 retro photo prompts where I'm the face in every example. All of it free. No signup, no ads, no upload step. Swipe through for what's inside and why I built it this way. Link in the last slide. Slide 1 of 12 Slide 2 of 12 Slide 3 of 12 Slide 4 of 12 Slide 5 of 12 Slide 6 of 12 Slide 7 of 12 Slide 8 of 12 Slide 9 of 12 Slide 10 of 12 Slide 11 of 12 Slide 12 of 12 Pranjul Rathour GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave. Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email: pranjulrathour41@gmail.com Portfolio & ...
Read more