Agent Guardrails
In short: Guardrails are checks in your code that limit what an agent can see and do (allowed tools, valid arguments, step and cost limits, input and output filters), no matter what the model decides.
On this page
Level: Advanced · Reading time: 9 min · Phase: 4 · Agentic AI
Before you start: Human in the Loop and Function Calling. You should know how code validates a tool call and how risky actions wait for approval.
Explain it like I'm new
A new driver can steer, but the road has lane markings, speed limits, and barriers on dangerous bends. The driver mostly stays safe on their own; the barriers stop the worst outcomes when they don't.
An agent is a capable but unreliable driver. It can misunderstand, hallucinate tool arguments, or be tricked by prompt injection (instructions hidden in a web page, email, or file it reads). Guardrails are the barriers:
- Allowlist: only these tools can be called, by this agent.
- Validation: arguments must match the schema and business rules.
- Limits: max steps, max cost, max time.
- Approval: risky actions need a human (Human in the Loop).
- Filters: check inputs (block obvious attacks, strip secrets) and outputs (no leaked personal data, valid format).
The golden rule: guardrails live in code, not in the prompt. "Never delete files" in a system prompt is a request. A delete tool that isn't on the allowlist is a fact.
Words to know:
- Least privilege: give the agent only the access it needs, and nothing more.
- Sandbox: an isolated area (a folder, container, or VM) where mistakes cannot reach the rest of the system.
- Business rule: a check specific to your app, such as a maximum refund amount.
- PII: personally identifiable information, such as names, phone numbers, or ID numbers.
- Audit log: a record of every call and decision, kept for review.
Watch it
It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.
How it works
Layers, from outside in:
- Least privilege. Give the agent only the tools it needs, with the narrowest scope (read-only DB user, one folder, one API scope).
- Allowlist check on every tool call. Unknown or disallowed tool → refuse.
- Argument validation. Schema check, then business rules (amount ≤ limit, path inside the sandbox, recipient in the company domain).
- Budget limits. Max steps, tokens, cost, wall-clock time across the whole run.
- Approval gate for irreversible actions.
- Untrusted content handling. Mark tool outputs and documents as data. Don't let them change permissions. Consider stripping or flagging instruction-like text.
- Output checks. Validate format, scan for secrets or personal data, apply content policy.
- Audit log. Record every call, check, and decision.
flowchart LR
M[model wants tool call] --> A{on allowlist?}
A -- no --> X[refuse + log]
A -- yes --> V{args valid + within rules?}
V -- no --> X
V -- yes --> B{within step/cost budget?}
B -- no --> S[stop run]
B -- yes --> R{risky?}
R -- yes --> H{human approves?}
H -- no --> X
H -- yes --> RUN[run tool]
R -- no --> RUN
RUN --> O[output filter] --> LOG[(audit log)]
Worked example
Take Guardrails(["write_text"], max_steps=2) from the code below. Real output when we call check three times:
| Call | What happens | Result |
|---|---|---|
check("write_text") | step 1, on the allowlist | passes silently |
check("send_email") | step 2, not on the allowlist | tool 'send_email' is not on the allowlist |
check("write_text") | step 3, over the cap of 2 | max_steps=2 exceeded |
Notice that every call counts as a step, even a refused one. That is on purpose: an agent that keeps asking for forbidden tools still hits the cap.
At pipeline level, run_pipeline("agents and email it", ..., allowed_tools=("write_text", "add_example")) returns status blocked and its log ends with guardrail: tool 'send_email' is not on the allowlist.
Code example
From agent_multi.py: a Guardrails object that enforces an allowlist and a step cap, and marks risky tools for approval.
class GuardrailError(Exception):
pass
class Guardrails:
def __init__(self, allowed_tools, max_steps=10, risky_tools=()):
self.allowed, self.risky = set(allowed_tools), set(risky_tools)
self.max_steps, self.steps = max_steps, 0
def check(self, tool):
self.steps += 1
if self.steps > self.max_steps:
raise GuardrailError(f"max_steps={self.max_steps} exceeded")
if tool not in self.allowed:
raise GuardrailError(f"tool '{tool}' is not on the allowlist")
def needs_approval(self, tool):
return tool in self.risky
g = Guardrails(["write_text"], max_steps=2)
g.check("write_text") # ok
try:
g.check("send_email")
except GuardrailError as e:
print(e) # tool 'send_email' is not on the allowlist
An argument-level business rule, added on top of schema validation from agent_react.py:
import os
SANDBOX = os.path.realpath("/tmp/agent-sandbox")
def safe_path(path):
full = os.path.realpath(os.path.join(SANDBOX, path))
if not full.startswith(SANDBOX + os.sep):
raise GuardrailError("path escapes the sandbox")
return full
In real projects, frameworks offer hooks for these checks (for example, tool-level callbacks or graph interrupts in LangGraph). Run code-executing tools in a real sandbox (container or VM with no secrets and restricted network).
When to use it / when not to
Always. The question is how strict.
| Agent can... | Minimum guardrails |
|---|---|
| Only read public data | allowlist, step cap, output format check |
| Read private data | + per-user access filters, output PII/secret scan |
| Write or act externally | + argument rules, approval for irreversible actions, audit log |
| Run code | + sandbox with no secrets, network limits, timeouts |
Common mistakes
- Relying on the system prompt for safety.
- One agent with every tool "just in case".
- Validating schema but not business rules (a valid JSON refund of ₹10,00,000 is still a problem).
- Treating text from tools, web pages, or other agents as trusted instructions.
- No global budget, so a multi-agent system can run up costs even if each agent has a limit.
Recap
If you remember only three things:
- Guardrails are code checks around tool execution: allowlist, argument validation, limits, approvals, and filters.
- Safety rules in a prompt are requests. Rules in code are enforced.
- Use layers and least privilege, because no single check stops every attack.
Interview questions
1. Why can't a system prompt alone keep an agent safe? <details><summary>Answer</summary> The model may misunderstand, ignore, or be manipulated out of prompt instructions, for example by prompt injection in content it reads. Only checks in code around tool execution reliably block disallowed actions. </details>
2. List five guardrails you'd add to a production agent. <details><summary>Answer</summary> Tool allowlist with least privilege, argument schema plus business-rule validation, max steps / token / cost / time limits, human approval for irreversible actions, input and output filtering (injection, secrets, PII), sandboxing for code execution, and audit logging. </details>
3. How do you defend an agent that browses the web against prompt injection? <details><summary>Answer</summary> Treat page content as untrusted data, keep high-risk tools unavailable or approval-gated during browsing, validate every action in code, limit data the agent can send out (no secrets in context, restricted destinations), and monitor/log. No single filter fully solves it, so use layers. </details>
Go deeper
- OWASP Top 10 for Large Language Model Applications
- Not what you've signed up for: Indirect Prompt Injection (Greshake et al., 2023)
- Anthropic docs: Tool use with Claude
Prev: Agent Evaluation · Next: Phase 5 · Frameworks · Back to phase: Phase 4