AI Engineer Roadmap All lessons Interactive Quiz GitHub

Evaluating Fine-Tuned Models

Advanced · 12 min read · Updated · By Saurabh Pathak

In short: You only know a fine-tune helped if it beats the base model on held-out examples it never saw, with a margin of error you can defend, no regressions on things that used to work, and an honest view of the limits of any automatic judge; preference tuning (RLHF, DPO) is the optional next step that teaches a model what people prefer.

Play the interactive lesson: Evaluating a tuned modelRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Advanced · Reading time: 12 min · Phase: 8 · Fine-tuning

Before you start: Training Pipelines and Hyperparameters and Dataset Preparation. For evaluation ideas in retrieval, see Phase 2 · Advanced RAG. For RLHF and DPO in one page, see How LLMs Are Trained.

Words to know:

Explain it like I'm new

A student says they studied hard. You do not ask them; you give an exam they have never seen, and you give the same exam to their previous self. If the new score is only a bit higher on a ten-question quiz, you cannot tell skill from luck, so you want more questions and a sense of how much the score would wobble. You also check that learning the new material did not make them forget the old (regression).

Then there is the grader. If the exam is marked by another student (an LLM judge), that grader has habits: it likes long answers, it likes answers that look like its own, and sometimes it prefers whichever answer comes first. Useful, but you must check it against human marking.

Watch it

Evaluating a tuned model: Same held-out questions, base against tuned, with a margin of error.
Evaluating a tuned model: Same held-out questions, base against tuned, with a margin of error.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

1. A held-out set, and a baseline

Build the evaluation set at the same time as the training set, from the same distribution, split by group (lesson 3) so nothing leaks. It should include the awkward cases you care about. Then score the base model first, with its best prompt. If you never measure the baseline, you cannot say what the tuning added. Measure the tuned model on the same questions with the same settings (temperature, prompt wrapper, stop rules).

2. Metrics that fit the task

3. A margin of error

A score on 12 examples is a noisy estimate. A bootstrap resamples the examples with replacement many times, recomputes the mean each time, and reports the middle 95% of those means. For a comparison use a paired bootstrap: resample the same examples for both models and bootstrap the per-example differences, so you ask "does the gain survive?" and not just "do the two ranges overlap?".

4. Regression tests

Averages hide damage. Keep a list of prompts the base model handled well (and general capabilities you care about: following system rules, refusing correctly, not breaking JSON) and check that the tuned model still does them. Fine-tuning can cause forgetting of abilities outside your data, especially with a high learning rate or many epochs.

Worked example

From ft_eval.py: a 12-question held-out set, with the base and the tuned model given as small dictionaries of answers.

  1. Exact match: base 0.417 (5 of 12), tuned 0.917 (11 of 12).
  2. Token F1: base 0.496, tuned 0.988.
  3. Win rate (tuned against base, by token F1): wins 7, ties 4, losses 1, so (7 + 0.5 x 4) / 12 = 0.750.
  4. Bootstrap: the tuned exact-match score has a 95% interval of [0.750, 1.000]. The paired difference (tuned minus base) has a 95% interval of [0.083, 0.833]: it stays above zero, so the gain is believable, but the interval is wide because 12 examples is few.
  5. Too small a sample: using only the first 4 questions, the paired interval is (0.0, 1.0): it includes zero, so you could not claim a gain.
  6. Regression: the one loss is "Accepted cards?". The tuned model said "Visa Mastercard and Amex", adding a card that is not accepted, so its F1 fell below the base. The average rose a lot and still hid a real error.

All data here is toy data written for the demo, not results from a real model.

5. LLM-as-judge: useful, with caveats

For open-ended outputs a strong model can grade answers against a rubric. It scales, but:

Calibrate by having humans grade a sample (50 to 100 examples) and check the judge agrees with them. Prefer objective checks wherever the task allows.

6. Preference tuning: RLHF and DPO (conceptually)

SFT shows the model good answers. It cannot easily say "this answer is better than that one". Preference tuning uses pairs: for a prompt, a chosen answer and a rejected answer.

The DPO loss, as given in the paper (with y_w the chosen answer, y_l the rejected one, pi_theta the model being trained, pi_ref the frozen reference, beta a strength setting and sigma the sigmoid):

L_DPO = - log sigmoid( beta * [ (log pi_theta(y_w|x) - log pi_ref(y_w|x))
                              - (log pi_theta(y_l|x) - log pi_ref(y_l|x)) ] )

In words: measure how much the model has moved from the reference on the chosen answer and on the rejected answer. Reward moving up on the chosen one and down on the rejected one. Beta controls how strongly: it acts like how far you let the model stray from the reference.

Worked numbers (beta = 0.1, summed log-probabilities of whole answers, all from the demo):

  1. Nothing learned yet: policy equals reference, so both differences are 0, the margin is 0, and the loss is -log(0.5) = 0.6931. That is ln 2, the loss of a coin flip.
  2. Policy improved: chosen log-prob rises from -10 to -8 (up 2), rejected falls from -12 to -15 (down 3). Margin = 0.1 x (2 - (-3)) = +0.50, loss 0.4741.
  3. Policy went the wrong way: chosen falls from -10 to -13, rejected rises from -12 to -9. Margin = 0.1 x (-3 - 3) = -0.60, loss 1.0375.

Preference tuning is a refinement: do it after SFT works, with good pairs, and evaluate it just like everything above (including a regression check, since it can change style, verbosity and refusals).

flowchart TD
    H[Held-out set, split by group] --> B[Score base model]
    H --> T[Score tuned model, same settings]
    B --> C[Compare: EM, F1, task checks, win rate]
    T --> C
    C --> I[Paired bootstrap interval]
    C --> R[Regression list]
    I --> D{Gain real, no harmful regressions?}
    R --> D
    D -->|yes| S[Ship, keep monitoring]
    D -->|no| F[Fix data or settings, retrain]

Code example

From ft_eval.py (run python 10-fine-tuning/code/ft_eval.py):

import ft_eval as e

b = e.score_model(e.BASE, e.HELDOUT, e.exact_match)
t = e.score_model(e.TUNED, e.HELDOUT, e.exact_match)
print(round(e.mean(b), 3), round(e.mean(t), 3))        # -> 0.417 0.917

lo, hi = e.paired_bootstrap_diff(t, b, seed=1)
print(round(lo, 3), round(hi, 3))                      # -> 0.083 0.833

loss, margin = e.dpo_loss(-8, -15, -10, -12, beta=0.1)
print(round(margin, 2), round(loss, 4))                # -> 0.5 0.4741

The tests in test_ft_eval.py check token F1 by hand (0.8), the win-rate counts, that the bootstrap is reproducible for a seed, that the paired interval excludes zero on 12 examples and not on 4, that the regression finder returns exactly "Accepted cards?", and that the DPO loss at the start is ln 2 and equals the hand-computed values.

In real projects use your task's real metrics and a larger held-out set (hundreds of examples if you can), keep a growing regression suite, and for preference tuning use a trainer library such as TRL, whose API changes, so check the current docs.

When to use it / when not to

SituationDo this
Any fine-tune you intend to shipHeld-out set, baseline, regression suite
Short, checkable answersExact match, F1, task checks
Open-ended answersPairwise win rate, human spot checks, a judge with a rubric
Small eval setReport the interval, not just the mean; grow the set
You have rankings of good vs bad answersConsider DPO after SFT works
You only have ideal answers, no preference dataSFT is enough; skip preference tuning

Common mistakes

Recap

If you remember only three things:

Interview questions

1. How do you prove a fine-tune is better than the base model? <details><summary>Answer</summary> Score both on the same held-out set, split without leakage, with the same prompts and settings and the base model's best prompt. Use metrics that fit the task, report a paired bootstrap interval for the difference, and check a regression suite for abilities that got worse. </details>

2. What are the main weaknesses of LLM-as-judge? <details><summary>Answer</summary> Position bias (prefers the first answer), length bias, self-preference for its own style, noise on small sets, and sensitivity to the prompt and judge version. Mitigate by randomising order, using a specific rubric, and checking agreement with human grades on a sample. </details>

3. Explain the DPO loss in plain words and what beta does. <details><summary>Answer</summary> For a chosen and a rejected answer, measure how much the trained model's log-probability has moved from a frozen reference on each. The loss is -log sigmoid(beta times (chosen movement minus rejected movement)), so it is smallest when the model raises the chosen answer and lowers the rejected one relative to the reference. Beta scales how strongly that margin counts, and so how far the model is allowed to move from the reference. </details>

Go deeper

Prev: Training Pipelines and Hyperparameters · Next: Phase 9 · Multimodal AI · Back to phase: Phase 8