AI Engineer Roadmap All lessons Interactive Quiz GitHub

MLflow

Intermediate · 8 min read · Updated · By Saurabh Pathak

In short: MLflow is an open-source tool that records your experiments (settings, scores, files) so you can compare runs and reproduce the best one.

Play the interactive lesson: MLflowRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 8 min · Phase: 7 — Production AI

Before you start: LangSmith and Chunking. You should know that RAG splits documents into chunks, and that a score is a number you measure.

Explain it like I'm new

Imagine a cooking notebook. Every time you try a recipe you write the ingredients, the oven temperature and how it tasted. Later you can say "the version with less salt at 180°C was best."

Without notes, AI work becomes chaos: "Which chunk size gave that good score last Tuesday?" MLflow is the notebook.

Words to know:

MLflow started with classic machine learning and also has features for LLM apps (tracing, evaluation). Check the MLflow docs for current GenAI features.

Watch it

MLflow: Record experiments so the best run can be compared and reproduced.
MLflow: Record experiments so the best run can be compared and reproduced.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

  1. Start a run.
  2. Log parameters (what you changed).
  3. Log metrics (what you measured).
  4. Log artifacts (files).
  5. Open the UI and compare runs in a table or chart.
flowchart LR
  E[Experiment: rag-tuning] --> R1[Run 1: chunk 300]
  E --> R2[Run 2: chunk 500]
  E --> R3[Run 3: chunk 800]
  R1 --> M[Compare metrics in UI]
  R2 --> M
  R3 --> M

Worked example

You want to pick a chunk size for a RAG app. You try three sizes on the same 100 test questions. Each try is one run, and you log one parameter and one metric. These are the demo's made-up scores:

RunParameter chunk_sizeMetric hit_rate
13000.71
25000.82
38000.76

The highest hit rate is 0.82, so run 2 wins. Running the mini-tracker below prints the real result:

best: {'chunk_size': 500} {'hit_rate': 0.82}

Because the parameter was logged next to the metric, you know which setting produced 0.82 and can use it again.

Code example

A stdlib mini-tracker with the same shape as MLflow's API:

runs = []

class Run:
    def __init__(self): self.params, self.metrics = {}, {}
    def log_param(self, k, v): self.params[k] = v
    def log_metric(self, k, v): self.metrics[k] = v

def fake_eval(chunk_size):
    return {300: 0.71, 500: 0.82, 800: 0.76}[chunk_size]   # made-up demo scores

for size in (300, 500, 800):
    run = Run()
    run.log_param("chunk_size", size)
    run.log_metric("hit_rate", fake_eval(size))
    runs.append(run)

best = max(runs, key=lambda r: r.metrics["hit_rate"])
print("best:", best.params, best.metrics)

In real projects

pip install mlflow, then:

import mlflow

mlflow.set_experiment("rag-tuning")

for chunk_size in (300, 500, 800):
    with mlflow.start_run():
        mlflow.log_param("chunk_size", chunk_size)
        score = evaluate_pipeline(chunk_size)        # your function
        mlflow.log_metric("hit_rate", score)

Run mlflow ui and open the address it prints to compare runs. By default results are stored in a local mlruns folder; teams usually point to a shared tracking server. See the MLflow docs.

When to use it / when not to

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Parameter vs metric? <details><summary>Answer</summary>A parameter is an input you choose. A metric is a measured result.</details>

2. Why track experiments? <details><summary>Answer</summary>To compare options fairly and reproduce the best result later.</details>

3. How would you use MLflow for RAG? <details><summary>Answer</summary>Log chunk size, embedding model, top-k and prompt version as parameters, and retrieval hit rate and answer quality scores as metrics, on a fixed eval set.</details>

Go deeper

Prev: 08-langsmith.md · Next: 10-prompt-versioning.md · Back to phase: Phase 7