AI agent governance: a practical framework for securing agents and their permissions

AI agent governance is the set of rules and controls that decide what each AI agent may do, who it acts for, and how you prove afterwards what it did. In practice that is nine things: an inventory, an identity per agent, least privilege, a check before every action, spend and loop limits, a person on anything irreversible, a tamper-evident record, a review cadence and an incident playbook.

Most teams find their gaps the expensive way. Gartner predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance gaps found only after production incidents (Gartner, 26 May 2026). Its advice: match controls to how much each agent can do on its own.

I lead product for GVNR at Instruxi, guardrails that decide allow, deny or ask before an agent acts. This is the framework I use, in plain words.

What is AI agent governance?

Model governance asks whether a model is fair and fit for purpose. Agent governance asks: may this agent do this, right now? An agent holds keys, calls tools and spends money, so the controls sit in front of its actions, outside the model. A line in the prompt saying “never delete anything” is a request. A permission the agent does not hold is a control.

The AI agent governance framework in nine steps

1. Inventory every agent. List each agent, the person who owns it, what it touches and how much it can do on its own. Gartner’s four autonomy levels are a useful label: observe, advise, act with approval, act autonomously. Include the ones people set up on their own laptops.

2. Give each agent its own identity. No shared API keys. Each agent gets its own credential and acts for a named person, so every action traces back to both. This is the answer to OWASP’s ASI03, Identity and Privilege Abuse, in the OWASP Top 10 for Agentic Applications (December 2025). In GVNR the agent cannot choose or change its own id.

3. Least privilege. Start read-only and add what the job proves it needs. Refunds up to a set amount, a staging database instead of production, a project folder instead of the whole disk. Separate credentials per environment.

4. Decide before the action. Put a check between the agent and its tools that answers every call with allow, deny, or stop and ask a person. Deny what should never happen, ask before what cannot be undone, allow the rest. A check that runs after the fact can only report the damage.

5. Set spend and loop limits. Two kinds: a total per agent per session or day, and a rate per minute. The rate catches runaways, the expensive incidents. Add loop detection: GVNR’s defaults stop an agent that repeats the same action four times in its last eight, and ask when spend passes $2 a minute or 75% of a $20 cap.

6. A person approves the irreversible. Deleting data, moving money, deploying, publishing and sending messages outside the company wait for a named human. Keep the list short: Gartner warns that approvals degrade under time pressure and approval fatigue.

7. Keep a tamper-evident record. Record who the agent acted for, the exact action, the tool, the model, the rule that answered, the verdict and the time, as fields you can query. Chain each line to the hash of the one before, so an edit or a deleted line breaks the chain and a single check shows where.

8. Review on a cadence. Each month the owner confirms the agent’s access and removes what it has not used. Each week someone reads the denies and asks: denies show where agents push, asks show where rules are too tight or too loose.

9. Write the incident playbook before the incident. Stop the agent, revoke its key, verify the record, work out what it touched, fix the rule that let it through, and write it up. Only a person should be able to resume a stopped agent.

How do I secure AI agents?

Steps 2 to 6 are the security core: identity, least privilege, a check before each action, limits, and a person on the irreversible. Those five cover the most common ways agents go wrong in the OWASP list: a hijacked goal (ASI01), misused tools (ASI02), abused identity and privileges (ASI03), unexpected code execution (ASI05) and cascading failures (ASI08). For coding agents with a shell, the step-by-step settings for Claude Code, Cursor, Codex and Gemini CLI are in my note on agents deleting databases. For agents connected to tools over MCP, start with the MCP security checklist.

How do I manage AI agent permissions?

Treat each agent like a new starter with a narrow role. Write its permissions down as scopes (what it can read, what it can change, how much it can spend), attach them to its own identity, and give it an owner who signs off each month. Make “ask a person” a permission in its own right: a finance agent can draft a payment and have it wait, rather than holding either no access or full access. When an agent changes job, change its scopes: editing the prompt changes nothing it can reach.

How do you red team AI agents?

AI red teaming means attacking your own agent before someone else does, and with an agent the thing to attack is what it can do: its tool calls, its spending, its deletes. Use the OWASP Top 10 for Agentic Applications as the test list: hide instructions in a document to hijack its goal (ASI01), push it to misuse a tool (ASI02), get it to act with someone else’s identity or privileges (ASI03), and try to make it run code it should not (ASI05). Aim each attack at the check that sits before the action, and record its verdict. Every attack that reaches a tool is a rule to write, and every one that ends in deny or ask is evidence for your review in step 8.

How does this map to NIST AI RMF, ISO/IEC 42001 and the EU AI Act?

The NIST AI Risk Management Framework 1.0 (January 2023) organises the work into four functions: Govern, Map, Measure and Manage. ISO/IEC 42001:2023 is the first international standard for an AI management system; ISO’s own summary lists responsibilities, risk management, lifecycle controls, monitoring and continual improvement among its requirements (ISO/IEC 42001 explained). The EU AI Act sets legal duties, the heaviest for high-risk uses.

For high-risk systems, Article 12 requires automatic event logs, Article 14(4)(e) a stop button or similar procedure, and Article 26 oversight by competent people and logs kept for at least six months.

When do the EU AI Act rules apply to AI agents?

The Act entered into force on 1 August 2024. Prohibited practices and the AI literacy duty applied from 2 February 2025, rules for general-purpose AI models from 2 August 2025, and the transparency rules, such as telling people they are talking to a machine, from August 2026, according to the European Commission. The Digital Omnibus, Regulation (EU) 2026/1744, moved the high-risk duties: 2 December 2027 for Annex III uses such as employment, credit, education and critical infrastructure, and 2 August 2028 for AI built into regulated products.

Most coding and research agents are not high-risk. An agent that screens job applicants or decides who gets credit is. Either way, the nine steps are the evidence an auditor or a customer’s security team will ask for.

What should an AI agent governance platform do?

Whatever you buy or build, check six things: an identity and owner per agent, a decision before each action runs, coverage of every agent whichever AI it uses, per-agent limits on spend, rate and repeats, irreversible actions routed to a named person, and a record you can verify without trusting the vendor.

That is what GVNR does on your own machine. It works with Claude Code, Cursor, Codex, Gemini CLI and any other MCP client. Where an agent offers a hook that runs before each tool call, the decision is enforced; Claude Code’s hook is wired today, and over MCP the agent has to ask. It is free and source-available under FSL-1.1-ALv2, so each release becomes Apache 2.0 two years later (code on GitHub). It does not judge whether an answer is true. It decides what an agent may do, and keeps the receipt.

The same three answers sit behind the credential and banking builds in my forward deployed work.

Questions people ask

What is AI agent governance?

The rules and controls that decide what each AI agent may do, who it acts for, and how you prove what it did.

What is an AI agent governance framework?

A repeatable set of controls applied to every agent: inventory, identity, least privilege, a check before each action, limits, human approval for the irreversible, a tamper-evident record, regular review and an incident playbook.

How do I secure AI agents?

Give each agent its own identity and the least access its job needs, check every action before it runs, cap spend and repeats, and make irreversible actions wait for a person.

How do I manage AI agent permissions?

Write them as scopes tied to the agent’s own identity, start read-only, make “ask a person” a permission, and have the owner review them monthly.

Does the EU AI Act apply to AI agents?

Yes, depending on the use. Transparency rules apply now; high-risk duties such as logging and human oversight apply from 2 December 2027 for Annex III uses.

What is the difference between AI risk management and AI compliance?

AI risk management is the ongoing work of finding, measuring and reducing what can go wrong with an AI system; the NIST AI RMF splits it into Govern, Map, Measure and Manage. AI compliance is proving you meet a specific rulebook. For EU AI Act compliance on high-risk uses, that means automatic logs (Article 12), a way to stop the system (Article 14) and logs kept for at least six months (Article 26). The nine steps above serve both: the controls cut the risk, and the record is the evidence.

If you need agent governance designed or built into your product, write to me at m@tawfik.tv.