AI engineering glossary
62 AI engineering terms in plain English: token, embedding, RAG, agent, MCP, LoRA, attention and more.
- LLM (large language model)
- A model that reads text and predicts the next piece of text, over and over, to write answers.
- Token
- The small piece of text a model reads and writes: a word, part of a word, or punctuation. Limits and prices are counted in tokens.
- Context window
- The most text (in tokens) the model can look at in one request, including your prompt and its answer.
- Prompt
- The instructions and text you send to the model.
- Hallucination
- When a model states something false in a confident way.
- RAG (retrieval-augmented generation)
- First find the most relevant pieces of your documents, then give them to the model so it answers from them and can cite them.
- Embedding
- A list of numbers that represents the meaning of a piece of text, so similar meanings end up close together.
- Chunking
- Cutting long documents into smaller pieces so each one can be embedded, searched and shown to the model on its own.
- Vector database
- A store that keeps embeddings and quickly finds the ones closest to a query.
- Cosine similarity
- A score for how closely two embeddings point in the same direction.
- BM25
- A classic keyword-matching score used for search.
- Hybrid search
- Running keyword search and embedding search together and merging the results.
- Reranking
- A second, more careful pass that re-orders the first search results so the best ones come first.
- Evals (evaluation)
- Tests that measure how good your system is, using a fixed set of questions with known good answers.
- Agent
- A program where the model decides what to do next, usually by choosing tools, until the goal is met or a limit is hit.
- Tool call (function call)
- The model asks your code to run a named function with arguments; your code runs it and sends back the result. The model never runs anything itself.
- Agent loop
- The repeating cycle of think, act (call a tool), observe the result, and decide whether to stop.
- MCP (Model Context Protocol)
- An open standard for connecting AI apps to tools, data and prompts, so one server works with many clients.
- Guardrails
- Checks around the model, on its inputs and outputs, that block or fix unsafe or invalid content.
- Prompt injection
- Hostile instructions hidden in text the system reads (a web page, a document) that the model may follow.
- Observability
- Recording what your system did (requests, steps, tokens, time, cost) so you can find out why it behaved a certain way.
- Temperature
- A setting that controls how random the model's word choices are. Low values give steadier answers, high values give more varied ones.
- Transformer
- The neural network design behind modern LLMs. It uses attention to weigh how much each word matters to every other word.
- Self-attention
- The step where each token looks at the other tokens in the text and decides which ones to focus on.
- KV cache
- Saved intermediate results from earlier tokens, so the model does not recompute them for every new token it writes.
- Query expansion
- Rewriting or adding to a search question so retrieval finds more of the relevant passages.
- HyDE (hypothetical document embeddings)
- Have the model write a made-up answer first, then search with that answer's embedding instead of the short question.
- Cost optimization
- Cutting token use and model spend, for example with shorter prompts, caching and smaller models, without hurting quality.
- Fine-tuning
- Training a model a little more on your own examples so it learns a style or a narrow task. Try prompting and RAG first.
- hit@k
- Out of your test questions, the share where the right chunk appeared in the top k results.
- MRR (mean reciprocal rank)
- Like hit@k, but it rewards finding the right chunk higher up: rank 1 scores 1, rank 2 scores 1/2, rank 3 scores 1/3.
- RRF (reciprocal rank fusion)
- A simple way to merge several ranked lists: each item scores the sum of 1 / (k + rank) over the lists, with k usually 60.
- ANN (approximate nearest neighbour)
- A search that may miss the very closest vector but is much faster. HNSW and IVF are two kinds.
- Stride
- How far a sliding window moves each step. Overlap equals window minus stride.
- Rate limit
- A cap on how many requests are allowed in a time window, to stop overload and abuse.
- Backoff
- Waiting longer after each failed try (1 s, then 2 s, then 4 s) so you do not hammer a service that is already struggling.
- Structured output
- A model answer in a fixed shape (such as JSON with named fields) that your code can read without guessing.
- Quantization
- Storing a model's numbers with fewer bits (for example 4 instead of 16) so it uses less memory, at the cost of a small rounding error.
- LoRA (low-rank adaptation)
- Fine-tuning by freezing the model and training a small add-on, so it needs far less memory than changing every weight.
- Continuous batching
- A serving trick: when one request finishes, a waiting request takes its place on the GPU straight away instead of waiting for the whole batch.
- LLM-as-a-judge
- Using a language model to grade answers when there is no exact right answer to compare with. It has biases you must test for.
- Cohen's kappa
- A score for how much two graders agree, after removing the agreement you would get by pure chance. 1 is perfect, 0 is chance.
- Query, Key, Value (Q, K, V)
- Three lists of numbers made from every token. The query is what the token is looking for, the key is how it advertises itself, and the value is the content it hands over when matched.
- Embedding vector
- The list of numbers that stands for one token. Tokens with similar meaning get similar numbers.
- Logit
- A raw score the model gives each possible next token before it is turned into a probability.
- Parameter (weight)
- One learned number inside the model. Training adjusts billions of them.
- Residual connection
- A shortcut that adds a layer's input back to its output, so information and learning signals can skip layers.
- LayerNorm
- Rescales each token's numbers to a steady range so deep networks train stably.
- Feed-forward network (FFN)
- A small network applied to each token on its own after attention. It holds much of what the model has memorised.
- Cross-entropy loss
- The training score for next-token prediction: minus the log of the probability given to the right token. Lower is better.
- Gradient descent
- Repeatedly nudging every parameter a little in the direction that lowers the loss.
- RLHF / DPO
- Ways to tune a model toward answers people prefer, using ranked answer pairs.
- GQA / MQA
- Attention designs where many query heads share a few key/value heads, which shrinks the KV cache.
- FlashAttention
- An exact attention algorithm that works in small tiles so the huge score table never has to be stored in slow memory.
- RoPE (rotary position embedding)
- Encodes word position by rotating query and key vectors, so attention depends on how far apart two tokens are.
- Mixture of Experts (MoE)
- A model with many expert sub-networks where a router sends each token to only a few of them, so only some parameters are active per token.
- Scaling law
- The observed pattern that model loss falls smoothly as parameters, data and compute grow.
- Untrusted
- Text you did not write yourself (such as user input, web pages or retrieved files), so your system must not obey it as instructions.
- Least privilege
- Giving each tool, service or user only the minimum permissions needed to do its job.
- Escaping
- Converting special characters (such as < to <) so model output is displayed as plain text instead of executed as code.
- Defence in depth
- Stacking several independent protections (input checks, least privilege, output escaping, human approval) so one missed check does not cause a breach.
- PII (personally identifiable information)
- Private data that can identify a person, such as names, email addresses, phone numbers or payment card numbers.