MindForge / Emerging Risks and Opportunities of Generative AI for Banks: A Singaporean Perspective

Risk that training or input data systematically underrepresents or misrepresents certain individuals or groups, leading to biased model outputs, discriminatory decisions, or reputational harm.

Ensure that the data does not contain systematic, unfair or harmful representation of certain groups based on attributes like race, gender, age, etc.

Ensure that the data selected for training, validation and testing is sufficiently representative of certain groups based on attributes like race, gender, age, etc.

Risk of AI models generating outputs that cause harm, disadvantage, or offence to individuals or groups, resulting in discriminatory outcomes, erosion of trust, and potential legal liability.

Ensure that the AI system outputs do not contain systematic, unfair or harmful representation of certain groups or individuals.

Risk that AI system outputs or usage conflict with the organisation's core values or societal norms, leading to reputational damage, stakeholder distrust, and ethical harm.

Ensure that a risk categorization of AI systems is defined, includes 'prohibited' category and systems classified as 'prohibited' are not developed or used.

Ensure that the intended use of an AI system is documented and suitable guardrails or monitoring is put in place to ensure the actual use is aligned with the intended use.

Risk that the high energy consumption and carbon emissions associated with training and operating large language models undermine the organisation's ESG commitments and corporate social responsibility objectives.

Ensure that the resources used to train and deploy AI systems are used in accordance the sustainable limits set by the organization.

Ensure that for AI models requiring high energy consumption, alternative lower resource models are considered and used if they meet the business requirements.

Risk that AI-generated content is used to deceive or manipulate users into actions they would not otherwise take, causing consumer harm, loss of informed consent, and regulatory exposure.

Ensure that the AI systems that generate deceptive content, impersonations or purposely false information that may trick or mislead users are not developed or used.

Ensure that appropriate mechanisms are put in place to prevent users being tricked or mislead into taking certain actions without fully understanding the consequences.

Risk that AI systems produce harmful, hateful, discriminatory, violent, or otherwise offensive content, causing direct harm to individuals and reputational damage to the organisation.

Ensure that the AI system outputs do not contain harmful content.

Ensure that the AI system outputs do not contain offensive content.

Ensure that the AI system outputs do not contain violent content.

Ensure that the AI system outputs do not contain racist content.

Ensure that the AI system outputs do not contain sexist content.

Ensure that the AI system outputs do not contain profane content.

Risk that insufficient training and education on Generative AI risks leaves staff and users unable to identify, escalate, or mitigate AI-related harms, increasing the organisation's exposure to operational, reputational, and compliance failures.

Ensure that AI users and staff dealing with the AI operations are trained in AI literacy.

Ensure AI systems are accessed only by users with the necessary awareness and understanding of the unique risks involved in such systems.

Risk that the organisation lacks sufficient oversight or contractual control over third-party AI system or model providers, resulting in inability to govern model changes, ensure compliance, or assign liability for adverse outcomes.

Impose required risk management processes on the development of third-party AI systems and models.

Ensure stable and transparent upgrade process of AI systems and models from third-party providers.

Risk that the absence of AI governance frameworks, controls, and accountability structures for AI use cases leads to unmanaged risks, untraceable decisions, and inability to assign responsibility for harmful outcomes.

Establish AI governance framework with central inventory, risk classification and role definitions

Maintain the profile of the AI system, which includes AI system's purpose, intended use and relevant parties of the AI system, as input into the AI risk management system that controls deviations.

Ensure that for the algorithms and models used, the applicable legal, organizational and technical requirements are met, suitable documentation exists, and an approval process exists.

Risk that insufficient human-in-the-loop oversight prevents timely detection and correction of AI failures, particularly for high-risk outputs requiring human validation, leading to unchecked harmful outcomes.

Ensure adequate human oversight to AI system processes, including the ability for human correction or intervention in the event of exceptions or when generating content with risk levels requiring human validation.

Ensure adequate monitoring exists to trigger human intervention in case anomalies, dysfunctions and/or unexpected performance is detected.

Risk that the absence of accessible feedback channels and formal recourse pathways leaves individuals harmed by AI outputs without remedy, and removes accountability from system developers and operators.

Ensure a clearly accessible feedback mechanism within the AI system is available to all users and affected parties. Establish a defined recourse pathway — a documented, step-by-step process for individuals to contest, escalate, or seek remediation for harmful or biased outputs.

Monitor and track submitted feedback and conduct periodic audits of feedback patterns to identify systemic bias, recurring harmful outputs, or gaps in recourse effectiveness. Implement automated flagging for high-severity complaints that triggers immediate escalation.

Embed consequence management into developer and operator accountability frameworks — including KPIs, performance reviews, or contractual obligations tied to harmful output rates and recourse resolution times.

Risk that undefined or unvalidated accuracy requirements for AI use cases lead to deployment of systems that produce unreliable outputs, resulting in poor decision-making, financial loss, and erosion of stakeholder confidence.

Sufficient user testing has been performed to evaluate AI system output accuracy with respect to the intended use.

Ensure that the intended use of an AI system is monitored at a regular frequency.

Risk that the inability to trace the origin, lineage, and rights associated with training and test data undermines auditability, regulatory compliance, and the organisation's legal right to use the data.

Ensure that all datasets used by the AI system have acceptable data provenance, including ownership, origin and lineage (transformations between original sources and the AI system)

Risk that the black-box nature of AI models prevents meaningful explanation of how outputs are derived, impairing auditability, user trust, regulatory compliance, and the ability to identify and correct errors.

Ensure that suitable explainability methods (either white-box or black-box) are deployed along with the AI system.

Ensure that the technical explainability methods are adequately evaluated to access their performance, limitations and whether they are fit for purpose.

Ensure that the relevant parties (e.g., users) are trained in how to use and interpret model explainability, as well as, understand its limitations.

Risk that AI systems mimicking human-like characteristics cause users to place unwarranted trust in outputs or mistakenly believe they are interacting with a human, leading to impaired judgement, manipulation, and erosion of informed consent.

Ensure that users are informed in understandable and accessible manner whenever they interact with an AI system, instead of a human.

Ensure that the AI system does not intentionally deceive the users into thinking they interact with a human, instead of an AI system.

Ensure that users are aware that they interact with an AI system, instead of a human.

Risk that model hosting and data processing activities occur outside permitted geographic jurisdictions, resulting in breach of data sovereignty regulations, regulatory sanctions, and loss of customer trust.

Ensure that the jurisdictional requirements governing the hosting and processing of data involved in the AI system are be identified and documented before deployment.

Where possible, ensure that the actual locations of model hosting and all data processing activities (including subprocessors) are compliant with applicable requirements. If not possible, ensure that contracts with model providers and subprocessors include binding obligations specifying the permitted geographic locations for model hosting, data processing, and storage, and must require providers to notify the organisation of any changes to those locations before they take effect.

Risk that ambiguous ownership of data used to train or generated by AI models exposes the organisation to intellectual property disputes, privacy violations, and legal liability.

Ensure that all datasets used by the AI system have documented data ownership, respond to relevant data regulations, rights to data for AI use, licenses and copyrights.

Ensure that the necessary rights to use exists over the AI system outputs.

Risk that data is transferred to or stored on systems not authorised under licensing terms or organisational policies, resulting in data security breaches, compliance violations, and contractual liability.

Establish data access and usage policies applicable to each AI system data asset to ensure that data compliance obligations are known, attributable to their source, and available to inform access controls, approval decisions, and audit activity.

Ensure that data assets at rest are be protected in a manner proportionate to their data policy classification.

Ensure that the new and existing approved uses of each data asset remain current and governed over time — so that new tools, systems, or transfer requirements are assessed against applicable obligations before being permitted,

Risk that AI system behaviour or outputs fail to meet applicable legal, regulatory, or organisational requirements, exposing the organisation to enforcement actions, fines, and reputational harm.

Ensure that the AI system complies with the regulatory requirements set out in the EU AI Act

Risk that AI system inputs or outputs violate intellectual property rights held by third parties, exposing the organisation to litigation, financial damages, and reputational harm.

Ensure the development and use of an AI system does not violate the Intellectual Property rights owned by another individual, organisation or entity.

Ensure that suitable automated techniques are put in place to automatically detect common types of outputs that infringe third-party intellectual property.

Risk that AI-generated content cannot be protected by copyright or trademark due to legal uncertainty around AI authorship, leaving the organisation's outputs vulnerable to unrestricted use by competitors.

Ensure that content produced by an AI system without material human change is marked as AI-generated; and content produced by an AI system with material human change has a named human owner who is accountable for reviewing and approving the final output, and whose contribution establishes authorship over it.

Ensure that the human authorship required to establish IP protection over AI-assisted content is genuine, identifiable, and evidenced — and that the organisation can demonstrate a defensible authorship chain for any content it asserts ownership over.

Ensure an independent, automated mechanism exists for identifying AI-generated content that has not been declared as such — reducing reliance on author self-declaration and discovering cases where AI involvement was not disclosed voluntarily.

Risk that misclassified or insufficiently protected personal and sensitive data is processed by AI systems without legal or ethical justification, resulting in privacy violations, regulatory penalties, and harm to data subjects.

Ensure that personal or sensitive data is not processed without appropriate protections as a result of being misclassified or unrecognised at entry — recognising that in a customer-facing AI system, personal data will routinely enter through channels that were not designed to handle it, such as free-text user prompts.

Ensure that no personal or sensitive data is processed by the AI system without a prior, explicit determination that a valid legal or ethical basis exists for doing so — and that this determination is made deliberately rather than assumed.

Risk that the absence of clear data retention and deletion policies for AI system data leads to prolonged storage of personal or sensitive information, resulting in privacy breaches, regulatory non-compliance, and inability to honour data subject rights.

Define a retention policy for each data asset associated with the AI system, specifying the maximum period for which personal, sensitive, or confidential data may be retained, the basis for that period, and the deletion or anonymisation action required at the end of it.

Ensure that data subject rights under applicable regulation can be exercised in the context of the AI system, and that where full technical erasure is not achievable — for example where data has been incorporated into model weights — this limitation is documented, legally assessed, and disclosed appropriately.

Ensure that retention policy commitments are operationally executed rather than existing only on paper — and that deletion is complete across the full data landscape of the AI system, not just the primary storage location.

Risk that AI models generate ungrounded or factually incorrect outputs presented as truth, leading to misinformation, reputational damage, erosion of public trust in AI systems, and harm to affected individuals or groups.

Ensure that the degree to which an AI system hallucinates is fit for purpose.

Ensure that monitoring and necessary logging is in place to allow analyzing the AI system hallucinations.

Ensure that suitable hallucination detection methods are deployed along with the AI system.

Risk that AI models present uncertain, contested, or false information with unwarranted confidence, impairing users' ability to exercise independent judgement and leading to flawed decisions based on unreliable outputs.

Ensure that the system outputs distinguish between information that is verifiable and attributable and information that is uncertain, inferred, contested, or unverifiable — and must communicate that distinction to the user in the output itself.

The uncertainty disclosed by a AI system in its outputs must accurately reflect the actual level of uncertainty present in those outputs, as determined by periodic independent assessment.

Risk that training data does not adequately represent the geographic, cultural, or operational context of the AI system's deployment, leading to inaccurate outputs, poor decisions, and failure to meet the system's intended objectives.

Ensure that the data selected for training is sufficiently representative of the geographical and cultural context under which the AI system will be used.

Ensure that the data selected for training and validation/testing is aligned with the indented goal of the AI system.

Risk that the absence of systematic, ongoing monitoring of AI system performance, usage patterns, and compliance allows undetected degradation, misuse, or deviation from intended purposes, ethical guidelines, and regulatory requirements.

Ensure that the relevant operational metrics (e.g., request latency, throughput, error rates, and resource utilization) are monitored.

Ensure that the relevant metrics capturing security and privacy aspects are monitored.

Ensure that functional metrics capturing data quality and model performance are monitored.

Ensure that the AI system actual use conforms to the intended use, ethical guidelines and regulatory requirements.

Risk that low-quality, noisy, or incomplete training data degrades model performance, resulting in unreliable outputs, increased remediation costs, and diminished fitness for purpose.

Ensure that the data selected for training, validation and testing has attributes that are of the right age in a specific context of use.

Ensure that the data selected for training, validation and testing is sufficiently representative of the real-world conditions under which the AI system is (or will) be used.

Ensure that the data selected for training, validation and testing is not noisy.

Ensure that the data selected for training, validation and testing has attributes that correctly represent the true value of the intended attribute, concept or event in a specific context of use.

Ensure that objects (of the real world) occur only once as a record in a dataset.

Ensure that the data used to train the model is not leaked into the data used to evaluate the model (or vice versa).

Ensure that all the required records/data values/attributes and metadata in the dataset are present.

Ensure that all the required records/data values/attributes and metadata in the dataset collected over the relevant period.

Ensure that the data source provides data that is factual and has attributes that are correct and of high quality.

Ensure that the data has attributes that are free from contradiction and are coherent with other data in a specific context of use.

Risk that training data becomes outdated as real-world conditions evolve, causing progressive degradation in model accuracy, the development of ingrained biases, and unreliable outputs that undermine business decisions.

Ensure that the data selected for training, validation and testing has attributes that are of the right age in a specific context of use.

Ensure that the model performance does not decay due to concept drift.

Risk that model outputs fail to meet required accuracy and performance thresholds, rendering the system unfit for its intended purpose and leading to flawed decisions, financial loss, and loss of stakeholder confidence.

Ensure that the model outputs meet the required performance thresholds.

Ensure that the model outputs are consistent.

Ensure that the AI system performance robust to input noise or semantically equivalent variations of the input data.

Ensure that the AI system performance evaluation is repeatable.

Risk that unanticipated usage patterns exploit the broad capabilities of generative models, causing outcome instability, unexpected failure modes, and unreliable outputs that harm users or the organisation.

Ensure monitoring mechanisms is implemented to detect and anomalies, and a process exists to periodically review them.

Risk that the complexity of Generative AI services outpaces the organisation's business continuity and disaster recovery capabilities, leading to prolonged service disruptions and inability to maintain critical operations.

Ensure appropriate mitigations and monitoring is implemented to prevent users from conducting excessive and uncontrolled inferences.

Over Reliance of the AI system on an individual model provider, without a backup plan in case of disruptions.

Risk that technology, cost, or resource constraints prevent the organisation from meeting the architectural requirements needed to govern AI models across deployment environments, resulting in security gaps, compliance failures, and ungovernable systems.

The ability to inspect and/or ensure data security in transit (e.g., support for SSL/ TLS encryption to secure data transmission).

The ability to inspect and/or implement user authentication and authorisation protocols.

The ability to perform automated model validation (e.g., necessary APIs are exposed).

The ability to inspect and/or implement backup and recovery plan to protect the model and the data.

The ability to control, maintain and update the AI system, including bug fixes, security patches, and feature enhancements.

The ability to monitor system operation and logs for automated processing.

Risk that consumers or employees inadvertently use Generative AI for inappropriate or illegal activities, exposing the organisation to legal liability, regulatory sanctions, and reputational damage.

Ensure that the unintended use of an AI system is documented and suitable guardrails or monitoring is put in place to ensure the actual use is not violating the unintended use policy.

Risk that malicious actors deliberately introduce corrupted or adversarial data into training pipelines or operational inputs, compromising model integrity and producing harmful, inaccurate, or exploitable outputs.

Ensure that data is thoroughly validated and verified before it is used (e.g., data passes data quality requirements before including in the training dataset or into the RAG knowledge base).

Ensure that the AI system either does not automatically ingest new data while in use, or that appropriate controls are in place to access data quality and validate the data.

Store data in a secure manner, and maintain access controls to limit who/what can update data stores.

Insure that the model has sufficient robust when handling potentially tainted inputs (e.g., through adversarial training or via secure handling of possible prompt injections).

Ensure monitoring mechanisms is implemented to detect and anomalies before the data is ingested into the system.

Risk that a malicious party with access to foundational model components deliberately alters model behaviour, leading to unpredictable, harmful, or exploitable outputs that undermine system integrity and user safety.

Ensure least-privilege access to foundational model weights, APIs, and fine-tuning pipelines. Enforce MFA, role-based permissions, and audit logs on all model-touching operations.

Cryptographically sign and hash model weights at each stage (training, fine-tuning, deployment). Verify signatures at load time; reject any artefact whose hash has changed unexpectedly.

Risk that adversarial prompts bypass AI system guardrails or filters, enabling malicious actors to generate prohibited content, extract sensitive information, or cause the system to act outside its intended boundaries.

Perform security testing to assess the degree of system vulnerability to prompt injections, and understand the system's limitations.

Ensure that the degree that sufficient mitigation measures are implemented to detect prompt injection attacks.

Risk that de-identified data released during normal AI operations can be re-linked to individuals through inference or cross-referencing, resulting in privacy breaches, regulatory violations, and harm to affected data subjects.

Possibility of de-identified records/ data being able to be re-identified mostly with malicious intent. This risk is related to “model inference attacks” but is distinct in that it refers to data released in the normal course of operations, whereas model inference attacks imply the use of deliberately designed inputs.

Risk that AI model outputs or development processes inadvertently expose sensitive, confidential, or personal data to unauthorised parties, whether through normal usage, prompt injection attacks, or training data memorisation, causing privacy violations and regulatory liability.

Define and maintain a list of PII and sensitive information that the AI system should not reveal.

Ensure that the degree to which the AI system is susceptible to reveal PII and sensitive information is fit for purpose. This includes: (i) unintentional data exposure during normal usage, (ii) targeted attacks by an malicious users (e.g., via prompt injection), and (iii) data leakage via training data.

Ensure that the system prompt used by the AI system is not leaked to the malicious users.

Risk that adversaries exploit the model's natural language interface to extract information about individuals in the training data through crafted inputs, compromising data subject privacy and exposing the organisation to regulatory penalties and legal claims.

Perform security testing to assess the degree of system vulnerability to prompt injections, and understand the system's limitations.

Ensure that the adversarial de-anonymization attack accuracy is below the allowed thresholds.