Streaming LLM answers with FastAPI and Server-Sent Events

Presenting to a room
Presenting to a room

Originally published at https://pranjulrathour.scult.in/blog/streaming-llm-responses-fastapi-sse. That copy is the canonical version and gets updates first.

A ten-second wait followed by a wall of text feels broken. The same answer streamed word by word feels fast, even when total time is identical. Every AI product I ship streams, and Server-Sent Events over FastAPI is the simplest way to do it.

Why SSE over WebSockets

Token streaming is one-directional: the server talks, the client listens. SSE is plain HTTP, works through most proxies, reconnects automatically in the browser, and needs no upgrade handshake. WebSockets earn their complexity when the client also streams — live speech-to-text is the case where I use them instead.

Pranjul Rathour
Pranjul Rathour

The endpoint shape

  1. Accept the question and any document scope as a POST, validate, and run retrieval before opening the stream — retrieval errors should return a normal HTTP error, not a broken stream.
  2. Return a StreamingResponse with media type text/event-stream and an async generator that yields events.
  3. Send a first event carrying metadata: the citation map, the retrieval confidence, and a request id.
  4. Yield token events as the provider streams them; end with a done event that includes usage and timing.

Handling failure mid-stream

Once headers are sent you cannot change the status code, so errors must travel inside the stream as a typed event. RAG.NextUpgrad's multi-provider fallback works at this layer: if the primary provider fails before the first token, the generator switches providers and the client never notices; if it fails mid-answer, the client receives an error event and a retry hint rather than a truncated paragraph.

Pitfalls

  • Reverse proxies buffering the response — disable buffering for the stream route and send a comment line every few seconds as a heartbeat.
  • Idle timeouts on free hosts — heartbeats also keep the connection alive during a slow reranking pass.
  • Forgetting to flush — yield small strings; do not accumulate.
  • Client-side, use fetch with a reader rather than EventSource if you need POST bodies; EventSource only supports GET.

Streaming is a small amount of code with an outsized effect on how a product feels. It is also the first thing I demo to students who think an AI app is just an API call and a text box.

Pranjul Rathour, GenAI engineer, Kanpur
Pranjul Rathour, GenAI engineer, Kanpur

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

From my carousels
7 Levels of RAG Apps
7 Levels of RAG Apps, slide 17 Levels of RAG Apps, slide 2
7 Levels of RAG Apps, slide 37 Levels of RAG Apps, slide 4
Full carousel on Instagram and LinkedIn.
Pranjul Rathour
Pranjul Rathour
GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor
I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave.
Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges.
On stage, at hackathons and on campus
In a packed college auditorium
In a packed college auditorium
Pranjul Rathour
Pranjul Rathour
On the mic
On the mic

Comments

Post a Comment

Popular posts from this blog

Forming a hackathon team: roles, skills and the mistake most teams make

Image
At an Integral Startup Foundation hackathon Originally published at https://pranjulrathour.scult.in/blog/hackathon-team-roles-and-formation . That copy is the canonical version and gets updates first. Team TechVerse — Raman Shukla, Saksham Gupta and me — won three first prizes in 2025 not because we were the strongest coders in the room, but because we had stopped overlapping. When I mentor teams before a hackathon, formation is the first thing I fix. In a packed college auditorium The four roles Builder — owns the core feature and the demo path. Usually backend or the AI piece. Interface — owns what the judge sees: the front end, the flow, the seeded data. Storyteller — owns the problem statement, the pitch, the slides and the Q&A plan; also does user research in hour one. Integrator — keeps the pieces working together, manages the repo, runs the demo rehearsals. In a team of three, the builder or interface person takes this. The mistake Five friends who all write R...
Read more

Hello from Kanpur: what I build, and what I'll write about here

Image
On the mic Originally published at https://dev.to/pranjulrathour/hello-from-kanpur-what-i-build-and-what-ill-write-about-here-f2j . That copy is the canonical version and gets updates first. I'm Pranjul Rathour, a GenAI engineer from Kanpur, India. This is the first post on this blog, so here is who is writing, what I have actually shipped, and what you can expect if you follow along. What I build Between February and July 2026, at Next Upgrad Web Solutions, I designed, built and shipped five production AI applications end to end, alone, and deployed all of them with Docker, CI and observability: RAG.NextUpgrad : a retrieval-augmented generation service with hybrid dense plus BM25 retrieval, cross-encoder reranking, streaming cited answers, and a confidence gate that refuses to call the model when the evidence is weak. It runs in about 220 MB of memory, inside a free 512 MB instance. FineTune Studio : fine-tune open models with QLoRA in a browser. Upload a dataset, validate ...
Read more

I built 15 free tools, 1,211 prompts and a 50,000-skill library — here's what's inside tools.scult.in

Image
Carousel · 12 slides I built every part of tools.scult.in myself — 15 browser tools, 1,211 version-stamped prompts, a 50,000+ agent-skills library, a public MCP server, and 40 retro photo prompts where I'm the face in every example. All of it free. No signup, no ads, no upload step. Swipe through for what's inside and why I built it this way. Link in the last slide. Slide 1 of 12 Slide 2 of 12 Slide 3 of 12 Slide 4 of 12 Slide 5 of 12 Slide 6 of 12 Slide 7 of 12 Slide 8 of 12 Slide 9 of 12 Slide 10 of 12 Slide 11 of 12 Slide 12 of 12 Pranjul Rathour GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave. Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email: pranjulrathour41@gmail.com Portfolio & ...
Read more