AI Engineer Roadmap All lessons Interactive Quiz GitHub

AI engineering glossary

62 AI engineering terms in plain English: token, embedding, RAG, agent, MCP, LoRA, attention and more.

LLM (large language model)
A model that reads text and predicts the next piece of text, over and over, to write answers.
Token
The small piece of text a model reads and writes: a word, part of a word, or punctuation. Limits and prices are counted in tokens.
Context window
The most text (in tokens) the model can look at in one request, including your prompt and its answer.
Prompt
The instructions and text you send to the model.
Hallucination
When a model states something false in a confident way.
RAG (retrieval-augmented generation)
First find the most relevant pieces of your documents, then give them to the model so it answers from them and can cite them.
Embedding
A list of numbers that represents the meaning of a piece of text, so similar meanings end up close together.
Chunking
Cutting long documents into smaller pieces so each one can be embedded, searched and shown to the model on its own.
Vector database
A store that keeps embeddings and quickly finds the ones closest to a query.
Cosine similarity
A score for how closely two embeddings point in the same direction.
BM25
A classic keyword-matching score used for search.
Running keyword search and embedding search together and merging the results.
Reranking
A second, more careful pass that re-orders the first search results so the best ones come first.
Evals (evaluation)
Tests that measure how good your system is, using a fixed set of questions with known good answers.
Agent
A program where the model decides what to do next, usually by choosing tools, until the goal is met or a limit is hit.
Tool call (function call)
The model asks your code to run a named function with arguments; your code runs it and sends back the result. The model never runs anything itself.
Agent loop
The repeating cycle of think, act (call a tool), observe the result, and decide whether to stop.
MCP (Model Context Protocol)
An open standard for connecting AI apps to tools, data and prompts, so one server works with many clients.
Guardrails
Checks around the model, on its inputs and outputs, that block or fix unsafe or invalid content.
Prompt injection
Hostile instructions hidden in text the system reads (a web page, a document) that the model may follow.
Observability
Recording what your system did (requests, steps, tokens, time, cost) so you can find out why it behaved a certain way.
Temperature
A setting that controls how random the model's word choices are. Low values give steadier answers, high values give more varied ones.
Transformer
The neural network design behind modern LLMs. It uses attention to weigh how much each word matters to every other word.
Self-attention
The step where each token looks at the other tokens in the text and decides which ones to focus on.
KV cache
Saved intermediate results from earlier tokens, so the model does not recompute them for every new token it writes.
Query expansion
Rewriting or adding to a search question so retrieval finds more of the relevant passages.
HyDE (hypothetical document embeddings)
Have the model write a made-up answer first, then search with that answer's embedding instead of the short question.
Cost optimization
Cutting token use and model spend, for example with shorter prompts, caching and smaller models, without hurting quality.
Fine-tuning
Training a model a little more on your own examples so it learns a style or a narrow task. Try prompting and RAG first.
hit@k
Out of your test questions, the share where the right chunk appeared in the top k results.
MRR (mean reciprocal rank)
Like hit@k, but it rewards finding the right chunk higher up: rank 1 scores 1, rank 2 scores 1/2, rank 3 scores 1/3.
RRF (reciprocal rank fusion)
A simple way to merge several ranked lists: each item scores the sum of 1 / (k + rank) over the lists, with k usually 60.
ANN (approximate nearest neighbour)
A search that may miss the very closest vector but is much faster. HNSW and IVF are two kinds.
Stride
How far a sliding window moves each step. Overlap equals window minus stride.
Rate limit
A cap on how many requests are allowed in a time window, to stop overload and abuse.
Backoff
Waiting longer after each failed try (1 s, then 2 s, then 4 s) so you do not hammer a service that is already struggling.
Structured output
A model answer in a fixed shape (such as JSON with named fields) that your code can read without guessing.
Quantization
Storing a model's numbers with fewer bits (for example 4 instead of 16) so it uses less memory, at the cost of a small rounding error.
LoRA (low-rank adaptation)
Fine-tuning by freezing the model and training a small add-on, so it needs far less memory than changing every weight.
Continuous batching
A serving trick: when one request finishes, a waiting request takes its place on the GPU straight away instead of waiting for the whole batch.
LLM-as-a-judge
Using a language model to grade answers when there is no exact right answer to compare with. It has biases you must test for.
Cohen's kappa
A score for how much two graders agree, after removing the agreement you would get by pure chance. 1 is perfect, 0 is chance.
Query, Key, Value (Q, K, V)
Three lists of numbers made from every token. The query is what the token is looking for, the key is how it advertises itself, and the value is the content it hands over when matched.
Embedding vector
The list of numbers that stands for one token. Tokens with similar meaning get similar numbers.
Logit
A raw score the model gives each possible next token before it is turned into a probability.
Parameter (weight)
One learned number inside the model. Training adjusts billions of them.
Residual connection
A shortcut that adds a layer's input back to its output, so information and learning signals can skip layers.
LayerNorm
Rescales each token's numbers to a steady range so deep networks train stably.
Feed-forward network (FFN)
A small network applied to each token on its own after attention. It holds much of what the model has memorised.
Cross-entropy loss
The training score for next-token prediction: minus the log of the probability given to the right token. Lower is better.
Gradient descent
Repeatedly nudging every parameter a little in the direction that lowers the loss.
RLHF / DPO
Ways to tune a model toward answers people prefer, using ranked answer pairs.
GQA / MQA
Attention designs where many query heads share a few key/value heads, which shrinks the KV cache.
FlashAttention
An exact attention algorithm that works in small tiles so the huge score table never has to be stored in slow memory.
RoPE (rotary position embedding)
Encodes word position by rotating query and key vectors, so attention depends on how far apart two tokens are.
Mixture of Experts (MoE)
A model with many expert sub-networks where a router sends each token to only a few of them, so only some parameters are active per token.
Scaling law
The observed pattern that model loss falls smoothly as parameters, data and compute grow.
Untrusted
Text you did not write yourself (such as user input, web pages or retrieved files), so your system must not obey it as instructions.
Least privilege
Giving each tool, service or user only the minimum permissions needed to do its job.
Escaping
Converting special characters (such as < to <) so model output is displayed as plain text instead of executed as code.
Defence in depth
Stacking several independent protections (input checks, least privilege, output escaping, human approval) so one missed check does not cause a breach.
PII (personally identifiable information)
Private data that can identify a person, such as names, email addresses, phone numbers or payment card numbers.