An enterprise AI governance plan is only as good as the controls that run when a request arrives. Most plans are policy documents: principles, committees, a review cadence. They read well and enforce nothing. This post is for the engineers who end up running the controls, and it lists what the plan has to contain so that each line of policy maps to something in the platform that can refuse a request, cap a budget or write a record.<\/p>\n
Each control in the plan needs an owner and an enforcement point. The owner is a person, not a council. The enforcement point is the component that applies the control at request time: the gateway, the budget service, the guardrail layer, the audit sink. If a control has no enforcement point, the plan should say so plainly and treat it as a gap, not a policy. A useful format is a control inventory with four columns: the control, who owns it, where it is enforced, and the evidence it produces.<\/p>\n
Teams that call providers directly cannot be governed, only audited after the invoice. The plan should put one gateway between every application and every model provider, with keys issued per team rather than per developer. Routing policy lives there: which requests may use the expensive model, which go to the cheaper one, and which run on a self-hosted model because of the data they carry. So does provider fallback, so a cap or an outage degrades a feature instead of taking it down.<\/p>\n
A budget that only alerts is a report. The plan should set a hard cap per team and per environment that returns an error when it is reached, with the alert as a warning on the way up. The spend incident that triggers most governance work is a weekend test script or a runaway agent; a hard cap turns that from a five-figure surprise into a failed job. Include the cached-input and output rates in the budget maths, because output tokens cost several times more than input on every current model.<\/p>\n
Input and output guardrails belong in the request path, not in a prompt. The plan should list which checks run before a request reaches a model (data classes that may not leave, injection patterns) and which run on the way back (actions that need a human, outputs that may not be shown), and what happens when a check fails. Every block should leave a record, because the next question from security will be how often it happens.<\/p>\n
Agents that call tools against production systems need a permission model: which tools, which systems, which actions, and which of those need a person in the loop. The plan should write that model down per agent and enforce it in the tool layer rather than trusting the model to obey a system prompt. Alongside it, a per-action audit record: the input, the tool call, the output and the decision, kept somewhere an auditor or a customer can read. When a customer asks what your agent did with their data, this record is the answer; without it, the answer is a guess.<\/p>\n
Models and prompts change, and a governance plan has to know which version served which request. That is model governance: registration, an evaluation set that gates releases, and a rollback that has been exercised rather than written. It is its own discipline and we cover it on the MLOps and model operations<\/a> page; the governance plan should reference that process rather than restate it.<\/p>\n The plan should state, per data class, which models may receive it and where those models run. Regulated data may be allowed to reach a model in your own cloud region and never a vendor endpoint; public data may go anywhere. The gateway enforces the boundary, which is the only place it can be enforced consistently.<\/p>\n Leave out the ethics preamble, the maturity model and the framework crosswalk. They have their place in a board paper. In the plan the engineers run, every page should answer who, where, and what evidence. If a line cannot be turned into a control with an owner and an enforcement point, it is not governance yet.<\/p>\nData boundaries<\/h2>\n
What to leave out<\/h2>\n