SAIF / Secure AI Framework

Risk that adversaries cause a model to produce incorrect inferences by slightly perturbing inputs, leading to reputational or legal challenges and triggering downstream security or privacy failures.

Ensure that AI models are made robust to adversarial inputs through adversarial training and testing techniques in the context of their intended application.

Risk that adversaries reduce the availability of ML systems by issuing resource-exhausting queries, rendering the service unavailable to legitimate users.

Ensure that only authorised users and endpoints can access AI system resources, and that rate limiting and load balancing are in place to prevent service exhaustion.

Risk that model output is not appropriately validated or sanitised before being passed to downstream systems or users, exposing the organisation to reputational, security, and user safety harms.

Ensure that model outputs are blocked, nullified, or sanitised before being passed to applications, extensions, or users.

Ensure that AI models are made robust to adversarial inputs through adversarial training and testing techniques in the context of their intended application.

Risk that training or retraining data is maliciously altered — through deletion, modification, or injection of adversarial data — to degrade model performance, skew outputs, or install hidden backdoors.

Ensure that poisoned or sensitive data is detected and removed or remediated in training and evaluation datasets.

Ensure that secure-by-default frameworks, libraries, software systems, and hardware components are used for AI development and deployment to protect confidentiality and integrity of AI assets.

Ensure that all data, models, and code used to produce AI models are verifiably integrity-protected during development and deployment.

Ensure that internal access to models, weights, and datasets in storage and production is minimised through least-privilege access controls.

Ensure that all data, code, models, and transformation tools used in AI applications are inventoried and tracked.

Risk that an adversary tampers with a model's source code, dependencies, or weights through supply chain or insider attacks, introducing vulnerabilities or unexpected behaviours including persistent architectural backdoors.

Ensure that secure-by-default frameworks, libraries, software systems, and hardware components are used for AI development and deployment to protect confidentiality and integrity of AI assets.

Ensure that all data, models, and code used to produce AI models are verifiably integrity-protected during development and deployment.

Ensure that internal access to models, weights, and datasets in storage and production is minimised through least-privilege access controls.

Ensure that all data, code, models, and transformation tools used in AI applications are inventoried and tracked.

Risk that adversaries make unauthorised modifications to components used for deploying a model — via supply chain attacks or exploitation of known vulnerabilities in serving infrastructure — altering model behaviour in production.

Ensure that secure-by-default frameworks, libraries, software systems, and hardware components are used for AI development and deployment to protect confidentiality and integrity of AI assets.

Risk of unauthorised appropriation of an AI model — by exploiting storage or serving infrastructure — resulting in intellectual property theft, replication of functionality, and potential downstream security and privacy harm.

Ensure that all data, code, models, and transformation tools used in AI applications are inventoried and tracked.

Ensure that internal access to models, weights, and datasets in storage and production is minimised through least-privilege access controls.

Ensure that secure-by-default frameworks, libraries, software systems, and hardware components are used for AI development and deployment to protect confidentiality and integrity of AI assets.

Risk that adversaries clone or recreate a model by systematically analysing its inputs, outputs, and behaviours — typically via API abuse — enabling imitation products or facilitating targeted adversarial attacks on the original model.

Ensure that only authorised users and endpoints can access AI system resources, and that rate limiting controls prevent excessive API querying that could enable reverse engineering.

Risk that vulnerabilities in software interacting with AI models — such as plugins, libraries, or applications — are exploited by attackers to gain unauthorised access, introduce malicious code, or compromise system operations, posing broad threats to user trust, privacy, and security.

Ensure that the least-privilege principle is applied as the upper bound on permissions for integrated components and plugins, minimising the number of tools and actions permitted.

Risk that adversaries cause a model to execute commands injected inside a prompt — exploiting the boundary between instructions and input data — leading to unintended model behaviour, data leakage, jailbreaks, or downstream harm when combined with other risks.

Ensure that adversarial queries to AI models are blocked or restricted through robust input filtering and validation.

Ensure that AI models are made robust to adversarial inputs through adversarial training and testing techniques in the context of their intended application.

Ensure that model outputs are blocked, nullified, or sanitised before being passed to applications, extensions, or users.

Risk that user data is collected, retained, processed, or shared beyond what is permitted by applicable policies, creating legal and regulatory exposure.

Ensure that all user data from AI applications — including prompts and logs — is stored, processed, and used in compliance with user consent and applicable policies.

Ensure that users are informed of relevant AI risks through disclosures, and that transparency and control mechanisms are provided for the use of their data in AI applications.

Risk that private or confidential data — including memorised training data, user interactions, system prompts, or data passing through integrated systems — is disclosed through querying of the model or agent. In agentic systems this risk is amplified, as agents may have privileged access to emails, files, and credentials, enabling large-scale exfiltration through tools and external channels.

Ensure that technologies minimising, de-identifying, or restricting use of PII data in training or evaluating models are applied.

Ensure that all user data from AI applications — including prompts and logs — is stored, processed, and used in compliance with user consent and applicable policies.

Ensure that model outputs are blocked, nullified, or sanitised before being passed to applications, extensions, or users.

Ensure that users are informed of relevant AI risks through disclosures, and that transparency and control mechanisms are provided for the use of their data in AI applications.

Ensure that the least-privilege principle is applied as the upper bound on agent permissions, with contextual and dynamic access to tools and data, to minimise the volume of sensitive information an agent can access or disclose.

Ensure that user approval is required for any agent actions that alter user data or act on the user's behalf, preventing unsanctioned disclosure of sensitive information through agent tools.

Ensure that an agent's actions, tool use, and reasoning are logged and auditable, enabling detection of unauthorised data access or disclosure through agent channels.

Risk that models infer sensitive personal information — such as gender, political affiliation, or health status — not contained in training data, causing privacy violations and potential harm to data subjects even when derived from public inputs.

Ensure that all data used to train and evaluate models is authorised for the intended purposes, and that data likely to lead to sensitive inferences is identified and managed.

Ensure that model outputs are blocked, nullified, or sanitised before being passed to applications, extensions, or users.

Risk that a model is trained or fine-tuned on data that is not authorised for that use — including data lacking user consent, unlicensed copyrighted material, or legally restricted data — resulting in legal and ethical liability.

Ensure that all data used to train and evaluate models is authorised for the intended purposes, covering consent, licensing, and regulatory requirements.

Ensure that unauthorised or sensitive data is detected and removed or remediated in training and evaluation datasets.

Risk that the absence of continuous adversarial testing, vulnerability management, threat detection, and incident response leaves AI systems exposed to undetected security and privacy failures across all risk areas.

Ensure that security and privacy improvements are identified through self-driven adversarial attacks on AI infrastructure and products.

Ensure that production infrastructure and products are proactively and continually tested and monitored for security and privacy regressions.

Ensure that internal or external attacks on AI assets, infrastructure, and products are detected and alerted on in a timely manner.

Ensure that a defined and exercised incident response process is in place to manage AI security and privacy incidents.

Risk that the absence of organisational policies, education, product governance, and risk management processes leaves AI security and privacy risks unmanaged and residual risk unmeasured across the organisation.

Ensure that easy-to-understand AI security and privacy policies and education are published for users.

Ensure that comprehensive AI security and privacy policies and education are published for employees.

Ensure that all AI models and products are validated against established security and privacy requirements before deployment.

Ensure that residual AI risk is inventoried, measured, and monitored across the organisation on an ongoing basis.

Risk that a model-based agent executes unintended actions — whether through accidental misalignment in task planning or malicious manipulation via prompt injection, poisoning, or evasion — causing harm to organisational reputation, user trust, security, and safety. Severity scales directly with the agent's level of autonomy and the breadth of permissions granted.

Ensure that the least-privilege principle is applied as the upper bound on agent permissions, with contextual and dynamic access to tools and actions, to constrain the blast radius of accidental or malicious rogue actions.

Ensure that user approval is required for any agent actions that alter user data or act on the user's behalf, preventing unsanctioned real-world consequences from rogue agent behaviour.

Ensure that an agent's actions, tool use, and reasoning are logged and auditable, enabling timely detection of and response to rogue behaviour.

Ensure that agent outputs are normalised and sanitised before rendering, to prevent malicious content from being executed within the user's application context.