How to read a loss curve during fine-tuning (and when to stop)
Originally published at https://pranjulrathour.scult.in/blog/reading-a-loss-curve-fine-tuning. That copy is the canonical version and gets updates first.
FineTune Studio streams loss, learning rate and throughput live to the browser because the loss curve is the one instrument you have while a run is in progress. Learning to read it saves GPU hours and, more importantly, saves you from shipping a model that memorised your dataset.
Training loss versus evaluation loss
Training loss measures fit to examples the model is learning from; it almost always goes down. Evaluation loss is measured on held-out examples and is the honest number. Hold out 5–10% of your data before training and evaluate every few hundred steps. Without an eval split you are flying on one gauge.
Curve shapes and what they mean
- Both falling, eval flattening — healthy. Stop when eval stops improving.
- Training falling, eval rising — overfitting. Stop, reduce epochs, add data or lower rank.
- A sharp drop then a spike — learning rate too high or a bad batch. Lower the rate; check the data around that step.
- Flat from the start — the rate is too low, adapters are not attached to the right modules, or the loss mask is wrong and the model is learning nothing new.
- Suspiciously near zero quickly — duplicated examples or leakage between train and eval splits.
When to stop
Stop at the checkpoint with the best evaluation loss, not at the end of the scheduled epochs. Save checkpoints frequently enough that "best" exists. The final epoch often looks better on training loss and worse on everything you care about.
Why live beats logs
A run that is clearly overfitting at step 400 should be killed at step 401, not discovered in a log file two hours later. Streaming telemetry over the same connection the frontend already holds meant a spike was visible the moment it happened. On rented GPUs that is money; on a borrowed laptop it is your evening.
Loss is not quality
A lower eval loss usually means a better model, but the only real test is side-by-side outputs from the base and tuned model on prompts you care about. That comparison is the whole point of evaluating a fine-tuned model honestly.
 | Pranjul Rathour GenAI engineer, Kanpur · 3x first-prize hackathon winner · campus mentor I ship production RAG pipelines, fine-tune LLMs and build agentic AI products end to end. I lead engineering at SCULT INDIA for a 14-member team and have mentored 200+ students through TechVerse Enclave. Open to: GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email: pranjulrathour41@gmail.com |
 Presenting to a room |  Pranjul Rathour |  Presenting BrandHive |
Comments
Post a Comment