IMDA / Model AI Governance Framework for Agentic AI

The agent takes incorrect actions, such as scheduling appointments on the wrong date or producing flawed code. The harm depends on the domain: flawed code can introduce exploitable vulnerabilities; wrong medical appointments can affect patient outcomes.

Before development or deployment, assess risk (likelihood × impact) against benefit using agent-specific factors: domain error tolerance, access to sensitive data, access to external systems, scope and reversibility of actions, autonomy level, task complexity, third-party involvement, and system complexity. Not all use cases suit agents; some are better served by deterministic workflows. Evaluate and formally accept residual risk.

Require human approval before high-stakes actions (editing sensitive data, final decisions in high-risk domains), irreversible actions (deletion, external communications, payments), outlier behaviour, and user-defined thresholds. Calibrate approval level to risk tier rather than reviewing everything.

Test overall task execution, tool calling (right tools, permissions, inputs, order), and robustness to errors and edge cases — in realistic environments, at scale, repeatedly, and across varied datasets to surface low-probability, high-impact behaviour.

Prompt the agent to reflect on whether its plan adheres to instructions and to summarise its understanding and request clarification before proceeding.

Roll out progressively — trained users first, restricted tools/protocols first, low-risk systems first — expanding as controls are validated.

Log and monitor on multiple layers (user–agent, agent–tool, model reasoning); define programmatic thresholds, anomaly detection, and proportionate interventions from scheduled review up to halting execution; integrate with observability platforms; feed insights back into evaluation.

Train integrating users on relevant use cases and restrictions (e.g. no confidential data), prompting best practices, the agent's range of actions, and common failure modes (hallucination, error loops), with ongoing refreshers.

The agent takes actions outside its permitted scope or authority — for example, executing an action without escalating for human approval when required by explicit instruction, company policy, or SOP.

Grant agents only the minimum tools and data access needed for their task, enforced through robust authentication and authorisation rather than instructions.

Give each agent a unique, cryptographically verifiable identity, tied to a supervising agent, human user, or department; record the capacity in which it acts; issue and track all identities (and permissions) from a centralised system, removing identities no longer required.

Make authorisations scoped, time- or session-bound, non-transferable, and least-privilege by default, with explicit escalation paths. An agent's permissions must never exceed those of its authorising human user; record all delegations.

Prefer structural, rule-based safeguards over prompt-layer instructions for higher-risk actions: enforce tool restrictions at the tool layer; build mandatory sequences into the workflow rather than relying on instructions.

Require human approval before high-stakes actions (editing sensitive data, final decisions in high-risk domains), irreversible actions (deletion, external communications, payments), outlier behaviour, and user-defined thresholds. Calibrate approval level to risk tier rather than reviewing everything.

Test entire workflows (reasoning, tool calls, intermediate steps), not only final outputs; verify the agent follows SOPs and routes for human approval when required, including attempting disallowed actions to confirm restrictions work.

Deny actions when approval infrastructure fails (supervisors unreachable, no approval policy exists); maintain termination and fallback solutions for catastrophic malfunction.

Record agent actions, decisions, plans, and interactions across all components; ensure problematic trajectories cannot be deleted; make every action attributable to an agent identity.

Biased LLM output translates into biased actions — e.g. unfair vendor selection in procurement, grant disbursement, or hiring decisions, particularly when handling groups of different profiles and demographics.

Require human approval before high-stakes actions (editing sensitive data, final decisions in high-risk domains), irreversible actions (deletion, external communications, payments), outlier behaviour, and user-defined thresholds. Calibrate approval level to risk tier rather than reviewing everything.

Accompany recommendations in sensitive workflows with the data considered, key factors, confidence, and noted uncertainties, so reviewers can evaluate rather than defer.

Test overall task execution, tool calling (right tools, permissions, inputs, order), and robustness to errors and edge cases — in realistic environments, at scale, repeatedly, and across varied datasets to surface low-probability, high-impact behaviour.

Track human override rates (low rates may signal rubber-stamping) and review response times (short times may signal automation bias); use analytics to identify outlier reviewers whose decision patterns deviate from the norm.

Enable users to report overrides and wrong actions, and feed this back into improving agent processes.

Agent actions lead to exposure or manipulation of personally identifiable or confidential information (customer details, trade secrets, internal communications). Causes include attackers exploiting the agent, or the agent failing to recognise data as sensitive. Agents typically have broader data access than plain LLMs and can not only leak but also wrongly modify data. Risk increases when the agent has persistent memory storing sensitive data across sessions.

Grant agents only the minimum tools and data access needed for their task, enforced through robust authentication and authorisation rather than instructions.

Do not grant write access to tables in sensitive databases unless strictly required; prefer read-only scopes.

Configure tools to require strict input formats; enforce least privilege per tool via authentication and authorisation.

Verify access controls hold across user roles and domains, including multi-turn indirect queries and comparisons between restricted and authorised accounts.

Keep sensitive data out of the agent's context entirely where feasible (e.g. trusted execution environments, opaque reference identifiers), eliminating leakage by design.

Restrict which agents can read or write shared context, memory, and intermediate outputs.

Log and monitor on multiple layers (user–agent, agent–tool, model reasoning); define programmatic thresholds, anomaly detection, and proportionate interventions from scheduled review up to halting execution; integrate with observability platforms; feed insights back into evaluation.

Declare in the UI at the point of interaction that users are dealing with an agent; disclose the agent's authorised range of actions and decisions, how user data is collected/stored/used (with consent where needed), the user's own responsibilities, and human contact points for escalation.

A compromised or malfunctioning agent disrupts the systems it interacts with — e.g. deleting a production codebase or overwhelming external systems with requests.

Design mechanisms to take agents offline and limit blast radius on malfunction; run high-risk tasks (e.g. code execution) in self-contained environments with limited network and data access.

Monitor and intervene during execution: rate limits to prevent excessive tool use, input validation, and injection detection — including re-prompting for approval when previously whitelisted patterns appear materially riskier.

Grant agents only the minimum tools and data access needed for their task, enforced through robust authentication and authorisation rather than instructions.

Roll out progressively — trained users first, restricted tools/protocols first, low-risk systems first — expanding as controls are validated.

Deny actions when approval infrastructure fails (supervisors unreachable, no approval policy exists); maintain termination and fallback solutions for catastrophic malfunction.

Due to hallucination or semantic misalignment (incorrect interpretation of user intent), the agent creates a plan that cannot achieve the task, contradicts user intent, or drifts from an earlier plan.

Prompt the agent to reflect on whether its plan adheres to instructions and to summarise its understanding and request clarification before proceeding.

For process-driven tasks, constrain the agent to follow defined SOPs and workflows rather than letting it define every step itself.

Require human approval before high-stakes actions (editing sensitive data, final decisions in high-risk domains), irreversible actions (deletion, external communications, payments), outlier behaviour, and user-defined thresholds. Calibrate approval level to risk tier rather than reviewing everything.

Record agent actions, decisions, plans, and interactions across all components; ensure problematic trajectories cannot be deleted; make every action attributable to an agent identity.

Build reflection/verification into workflows: self-review against evidence, LLM-judge critique, metrics-based checks (faithfulness, relevance), output-format validation, and bounded retries with termination on repeated failure.

The agent hallucinates non-existent tools, calls the wrong tools, calls the right tools with wrong inputs, or calls tools in a biased manner.

Configure tools to require strict input formats; enforce least privilege per tool via authentication and authorisation.

Test overall task execution, tool calling (right tools, permissions, inputs, order), and robustness to errors and edge cases — in realistic environments, at scale, repeatedly, and across varied datasets to surface low-probability, high-impact behaviour.

Grant agents only the minimum tools and data access needed for their task, enforced through robust authentication and authorisation rather than instructions.

Monitor and intervene during execution: rate limits to prevent excessive tool use, input validation, and injection detection — including re-prompting for approval when previously whitelisted patterns appear materially riskier.

Log and monitor on multiple layers (user–agent, agent–tool, model reasoning); define programmatic thresholds, anomaly detection, and proportionate interventions from scheduled review up to halting execution; integrate with observability platforms; feed insights back into evaluation.

Direct or indirect prompt/code injection (e.g. malicious instructions embedded in websites, documents, or third-party chatbot responses) redirects the agent towards harmful actions, including data exfiltration. Exposure increases with the agent's access to external, untrusted systems such as the open web.

Systematically identify attack paths (memory poisoning, tool misuse, privilege compromise) and map how untrusted data moves through the system; update the threat model regularly.

Monitor and intervene during execution: rate limits to prevent excessive tool use, input validation, and injection detection — including re-prompting for approval when previously whitelisted patterns appear materially riskier.

Allow agents to interact only with whitelisted MCP servers, websites, and external services.

Conduct regular red teaming against the agentic system, including indirect prompt injection scenarios specific to the use case, to validate guardrail effectiveness.

Prefer structural, rule-based safeguards over prompt-layer instructions for higher-risk actions: enforce tool restrictions at the tool layer; build mandatory sequences into the workflow rather than relying on instructions.

Configure agents to hand control to the user when sensitive data (passwords, API keys) must be entered.

New agent communication protocols are poorly deployed or compromised — e.g. an untrusted MCP server containing code that exfiltrates user data. Supply chain risk from third-party tools and skills falls in this category.

Allow agents to interact only with whitelisted MCP servers, websites, and external services.

Sandbox all agent-initiated code execution.

Use standardised protocols where applicable (e.g. agentic commerce protocols for transactions); consider the MCP layer as a governance point — filtering sensitive data, logging agent-to-system interactions.

Clarify obligations (security, performance guarantees, data protection) in contracts with model developers, platform providers, and tool hosts; reassess deployment where gaps exceed risk tolerance.

An attacker or upstream error corrupts the agent's persistent memory or knowledge base, causing the agent to act on poisoned context in current and future sessions.

Keep sensitive data out of the agent's context entirely where feasible (e.g. trusted execution environments, opaque reference identifiers), eliminating leakage by design.

Structure and version source information (context graphs, metadata on version/expiry) so agents deterministically use only current, relevant documents.

Restrict which agents can read or write shared context, memory, and intermediate outputs.

Systematically identify attack paths (memory poisoning, tool misuse, privilege compromise) and map how untrusted data moves through the system; update the threat model regularly.

Log and monitor on multiple layers (user–agent, agent–tool, model reasoning); define programmatic thresholds, anomaly detection, and proportionate interventions from scheduled review up to halting execution; integrate with observability platforms; feed insights back into evaluation.

Agents act at machine speed and volume, making it difficult for oversight mechanisms to detect and prevent unauthorised actions in real time. Continuous human approval requirements lead to automation bias and alert fatigue.

Require human approval before high-stakes actions (editing sensitive data, final decisions in high-risk domains), irreversible actions (deletion, external communications, payments), outlier behaviour, and user-defined thresholds. Calibrate approval level to risk tier rather than reviewing everything.

Log and monitor on multiple layers (user–agent, agent–tool, model reasoning); define programmatic thresholds, anomaly detection, and proportionate interventions from scheduled review up to halting execution; integrate with observability platforms; feed insights back into evaluation.

Keep approval requests short, clear, and digestible — not raw logs — while surfacing associated risk and confidence scores.

Design agents to monitor other agents in real time, flagging anomalies and inconsistencies.

Deny actions when approval infrastructure fails (supervisors unreachable, no approval policy exists); maintain termination and fallback solutions for catastrophic malfunction.

A mistake in one step propagates and amplifies across later steps, producing outsized impact — e.g. a hallucinated inventory figure driving downstream over- or under-ordering.

Build reflection/verification into workflows: self-review against evidence, LLM-judge critique, metrics-based checks (faithfulness, relevance), output-format validation, and bounded retries with termination on repeated failure.

Define triggers for change review (model updates, tool modifications, domain shifts, performance degradation, regulatory changes) and categorise changes by risk — light review for minor prompt refinements through to immediate re-risk assessment for critical changes.

Test entire workflows (reasoning, tool calls, intermediate steps), not only final outputs; verify the agent follows SOPs and routes for human approval when required, including attempting disallowed actions to confirm restrictions work.

Require human approval before high-stakes actions (editing sensitive data, final decisions in high-risk domains), irreversible actions (deletion, external communications, payments), outlier behaviour, and user-defined thresholds. Calibrate approval level to risk tier rather than reviewing everything.

Record agent actions, decisions, plans, and interactions across all components; ensure problematic trajectories cannot be deleted; make every action attributable to an agent identity.

Uncontrolled proliferation of agents without centralised management, leading to provenance issues, incompatibility between agent generations, orphaned permissions, and shadow agents.

Give each agent a unique, cryptographically verifiable identity, tied to a supervising agent, human user, or department; record the capacity in which it acts; issue and track all identities (and permissions) from a centralised system, removing identities no longer required.

Define accountability across the agent lifecycle for key decision makers (goals, permitted use cases, governance), product teams (design, implementation, testing, monitoring, user education), cybersecurity teams (guardrails, red teaming, threat modelling), and users (responsible use, training, reporting). Maintain adaptive governance capability to track technology developments.

Make authorisations scoped, time- or session-bound, non-transferable, and least-privilege by default, with explicit escalation paths. An agent's permissions must never exceed those of its authorising human user; record all delegations.

Define triggers for change review (model updates, tool modifications, domain shifts, performance degradation, regulatory changes) and categorise changes by risk — light review for minor prompt refinements through to immediate re-risk assessment for critical changes.

Agents working together fail through **miscoordination** (interpreting user intent differently and pursuing divergent goals), **conflict** (agents optimising competing objectives, e.g. a support agent issuing refunds a revenue-protection agent blocks), or **collusion** (emergent coordinated behaviour, e.g. pricing agents converging on higher prices).

Require agents to communicate through structured schemas (typed function calls) rather than free text, reducing unintended instruction passing.

Test at the multi-agent system level — not only per agent — for emergent behaviour, competitive dynamics, and the impact of one compromised agent on others.

Restrict which agents can read or write shared context, memory, and intermediate outputs.

Structure agents around functional boundaries (e.g. IT helpdesk vs HR self-service) with separately scoped tools and permissions, rather than one all-powerful agent.

Design agents to monitor other agents in real time, flagging anomalies and inconsistencies.

With multiple non-deterministic agents (or one complex agent with feedback loops), possible outcomes grow exponentially, producing emergent behaviour that cannot be predicted from testing components individually. Hardest to anticipate when agents cross system or organisational boundaries without white-box access.

Roll out progressively — trained users first, restricted tools/protocols first, low-risk systems first — expanding as controls are validated.

Log and monitor on multiple layers (user–agent, agent–tool, model reasoning); define programmatic thresholds, anomaly detection, and proportionate interventions from scheduled review up to halting execution; integrate with observability platforms; feed insights back into evaluation.

Test at the multi-agent system level — not only per agent — for emergent behaviour, competitive dynamics, and the impact of one compromised agent on others.

Treat system complexity (multiple agents, autonomous handoffs, feedback loops) as a risk-increasing factor requiring tighter governance pathways and escalation.

Design mechanisms to take agents offline and limit blast radius on malfunction; run high-risk tasks (e.g. code execution) in self-contained environments with limited network and data access.

Agents provided or operated by external parties are difficult to observe and control — it is unclear what they are grounded on, what they infer from conversations, or how data is stored and used. Cross-platform actions by vendor-embedded agents may not have been exhaustively tested.

Clarify obligations (security, performance guarantees, data protection) in contracts with model developers, platform providers, and tool hosts; reassess deployment where gaps exceed risk tolerance.

Require disclosures on agent capabilities and data handling; request scoped API keys, per-agent identity tokens, and logging of tool calls and access history. Where lacking, use alternatives or scope down the use case.

Restrict vendor-embedded agentic features to within their own ecosystems unless cross-platform actions are explicitly assessed and approved.

Before development or deployment, assess risk (likelihood × impact) against benefit using agent-specific factors: domain error tolerance, access to sensitive data, access to external systems, scope and reversibility of actions, autonomy level, task complexity, third-party involvement, and system complexity. Not all use cases suit agents; some are better served by deterministic workflows. Evaluate and formally accept residual risk.

Human reviewers over-trust agents that have performed reliably, approve without genuine evaluation, or lack the domain expertise to assess agent actions (e.g. non-engineers reviewing "vibe-coded" output). Anthropomorphic design can worsen this. Oversight degrades silently over time.

Track human override rates (low rates may signal rubber-stamping) and review response times (short times may signal automation bias); use analytics to identify outlier reviewers whose decision patterns deviate from the norm.

Train overseers to identify common failure modes (inconsistent reasoning, outdated policy references); ensure reviewers have the domain expertise to evaluate agent actions; note that chain-of-thought is not a faithful explanation.

Keep approval requests short, clear, and digestible — not raw logs — while surfacing associated risk and confidence scores.

Require approvers to record written justification before approving or rejecting high-risk agent actions.

Accompany recommendations in sensitive workflows with the data considered, key factors, confidence, and noted uncertainties, so reviewers can evaluate rather than defer.

As agents take over entry-level and routine tasks that traditionally train staff, basic operational knowledge erodes. Users may no longer be able to perform critical processes manually when agents malfunction or become unavailable.

Identify the core capabilities of each job affected by agents and provide sufficient training and work exposure so staff retain foundational skills and can run critical processes manually.

Train integrating users on relevant use cases and restrictions (e.g. no confidential data), prompting best practices, the agent's range of actions, and common failure modes (hallucination, error loops), with ongoing refreshers.

Deny actions when approval infrastructure fails (supervisors unreachable, no approval policy exists); maintain termination and fallback solutions for catastrophic malfunction.