HIGH RISK AI SYSTEMS

WHAT ARE HIGH-RISK AI SYSTEMS?

High‑risk AI systems are AI‑based systems whose deployment can significantly affect human health, safety, or fundamental rights such as equality, privacy, and access to justice. They are not defined by the underlying technique (for example, neural networks vs rules), but by the severity of potential harm and the sensitivity of the domain in which they operate, such as credit scoring, policing, migration control, or critical infrastructure.

Importance of High‑Risk AI Systems

High‑risk AI systems increasingly mediate access to housing, loans, education, welfare, employment and even physical liberty, making their errors or biases materially consequential for individuals. Because they can also amplify historical discrimination and entrench opaque decision‑making at scale, legislators use the “high‑risk” label to trigger strict duties of risk assessment, documentation, transparency, human oversight, and ongoing monitoring.

History and Evolution

The idea of distinguishing “high‑risk” applications emerged as AI moved from low‑stakes recommender systems into safety‑critical and rights‑sensitive settings, highlighting that some uses required more than soft ethics. The EU AI Act crystallised this into a formal risk pyramid with explicit categories of unacceptable, high, limited, and minimal risk, and other jurisdictions (including the US, China, India, and Singapore) have begun to adopt similar functional distinctions in laws, policies, and guidelines.

OFFICIAL DEFINITION OF HIGH-RISK AI SYSTEMS

There is no single universal definition of high‑risk AI, but most legal and policy instruments converge on a common idea: systems whose use can seriously affect health, safety, or basic rights are “high‑risk” and must satisfy heightened safeguards.

High-Risk AI Systems in Indian Legislation

Digital Personal Data Protection Act, 2023

The DPDP Act, 2023, does not define “high‑risk AI systems” directly, but it does create the category of “significant data fiduciaries”, which captures entities whose processing activities pose higher risks to data principals’ rights and therefore attract enhanced obligations such as impact assessments and audits. In practice, large‑scale AI deployments that profile, score, or make consequential decisions about individuals are likely to fall within this category, making DPDP an indirect but important foundation for regulating high‑risk AI in India.

Artificial Intelligence (Ethics and Accountability) Bill, 2025 – Section 5(2)

The Artificial Intelligence (Ethics and Accountability) Bill, 2025, proposes the first dedicated legislative framework for AI in India, adopting a risk‑layered architecture inspired by the EU AI Act by distinguishing prohibited, high‑risk, and lower‑risk systems. Section 5 is devoted to “High‑Risk Artificial Intelligence Systems”; sub‑section 5(2) requires that any person intending to deploy a high‑risk AI system must first submit it to the proposed Ethics Committee for Artificial Intelligence, along with documentation on its purpose, data sources, safeguards against bias, and mechanisms for human oversight and grievance redress.

Functionally, section 5(2) establishes an ex ante authorisation regime for high‑risk AI uses in sensitive domains such as surveillance, law enforcement, employment, healthcare and credit, making deployment contingent on prior ethical and technical scrutiny. Commentators argue that this makes the Bill structurally similar to the EU high‑risk regime, while also raising questions about institutional capacity, regulatory capture, and the risk that “approval” could normalise controversial applications so long as they pass procedural checks.

High‑Risk AI Systems in Official Government Report(s)

India AI Governance Guidlines

India’s AI Governance Guidelines are a non‑statutory national framework that set out how AI should be developed and deployed in India, with a strong, explicit focus on restricting high‑risk AI systems.

The India AI Governance Guidelines were issued by the Ministry of Electronics and Information Technology (MeitY) on 5 November 2025 as a principle‑based, risk‑based framework for “safe and trusted AI.” They aim to maximise developmental and economic gains from AI while mitigating risks such as bias, discrimination, exclusion and lack of transparency, and they are designed to remain agile and technology‑neutral.

High‑risk AI in the Guidelines

The Guidelines state that India “does not allow unrestricted deployment of high‑risk AI systems” and instead adopts a risk‑based, evidence‑led and proportionate governance approach. They recognise that some AI applications—particularly those that can cause significant harm to individuals or society—require stronger safeguards, and they emphasise that sectoral regulators (for example in finance, health, telecom, and law enforcement) remain responsible for enforcement and oversight within their existing legal mandates.

High‑risk AI is not set out in a fixed annex list, but the Guidelines highlight domains where risk is especially acute:

• bias and discrimination in access to services or opportunities,
• exclusion of vulnerable groups from essential digital infrastructures, and
• opaque systems that affect rights or safety without meaningful transparency or recourse. For such cases, they call for robust risk‑management processes, human oversight, and safeguards tailored to the specific sector and application.

Institutional and techno‑legal architecture

The Guidelines sit within a broader techno‑legal framework being developed by the Government of India. They propose or reference institutions such as:

• an AI Governance Group (AIGG) to coordinate policy,
• an AI Safety Institute (AISI) to test and evaluate systems, and
• technical mechanisms such as regulatory sandboxes and incident‑reporting channels for AI harms.

Importantly, the Guidelines themselves do not create new statutory appeal bodies or audit mandates; instead they rely on existing laws like the Information Technology Act and the Digital Personal Data Protection Act, with the expectation that these will be refined and complemented over time. A phased action plan (short‑, medium‑ and long‑term) envisages developing India‑specific risk‑classification frameworks, refining liability regimes, and progressively strengthening regulatory measures as AI technologies and use cases evolve.

NITI Aayog

NITI Aayog’s work on Responsible AI is the main place where “high‑risk” AI thinking first appears in Indian policy, even though it stays at the level of principles and recommendations rather than binding law.

Where “high‑risk” appears in NITI Aayog documents

NITI Aayog has a multi‑document Responsible AI series:

• Working Document: Towards Responsible #AIforAll (Part 1, 2020) – early draft identifying that some AI use‑cases in certain sectors should be treated as “high‑risk” and subject to specific requirements.
• Principles for Responsible AI (2021) – sets seven principles (safety and reliability, inclusivity and non‑discrimination, equality, privacy and security, transparency, accountability, protection of positive human values) but focuses on ethics rather than a formal high‑risk list.
• Approach Document for India: Part 2 – Operationalising Principles for Responsible AI (2021) – explicitly recommends a risk‑based regulatory mechanism and suggests that government may mandate responsible‑AI practices for high‑risk AI use‑cases.
• Responsible AI for All: Adopting the Framework – A Use Case Approach on Facial Recognition Technology (2022) – applies the framework to FRT and, drawing on EU debates, treats facial‑recognition systems as paradigmatic high‑risk use‑cases requiring the highest level of compliance.

How NITI Aayog defines and classifies high‑risk AI

The Part 2 Approach Document is clearest: it suggests classifying AI systems by the level of risk they pose into unacceptable, high, and low or minimal risk, explicitly inspired by global models such as the EU AI Act. Whether an AI system is “high‑risk” depends on its intended purpose, the severity of possible harm, and the probability of its occurrence, with both factors combined in a risk‑matrix. The document proposes that India adopt a risk‑based regulatory mechanism where “the greater the potential for harm, the more the regulatory scrutiny attracted by the relevant AI system,” and that policy tools like sandboxing and controlled deployments be used for higher‑risk use‑cases.

In this scheme, high‑risk AI use‑cases are those that can cause significant harm to individuals or society—for example, affecting rights, safety, or access to essential services—and the paper suggests that government may mandate responsible‑AI practices for high‑risk AI use‑cases. It also recommends that a Central Expert Team (CET) or similar body be tasked with identifying such high‑risk use‑cases as the ecosystem evolves.

The 2022 Responsible AI for All – FRT use‑case paper then operationalises this thinking by treating facial recognition technology as a high‑risk application: it maps specific harms (inaccuracy, discrimination, chilling effects, surveillance) and sets out detailed recommendations to maximise benefits while minimising risks, explicitly referencing the EU’s classification of FRT as high‑risk with stringent compliance requirements.

Relation to India AI Governance Guidelines

NITI Aayog’s Responsible AI series laid the conceptual groundwork for India’s later AI Governance Guidelines, which adopt a risk‑based, principle‑based approach and explicitly state that India will not allow unrestricted deployment of high‑risk AI systems. In that sense, NITI’s documents are best read as pre‑legislative policy blueprints: they introduce high‑risk AI categories, recommend risk‑tiered regulation, and test these ideas on concrete use‑cases like facial recognition, which later inform the Union government’s techno‑legal framework for AI.

FUNCTIONAL ASPECT OF HIGH-RISK AI SYSTEMS

Functionally, a system tends to be treated as high‑risk where it significantly shapes or determines consequential decisions about individuals, it operates in critical infrastructure, or it performs safety functions in regulated products such as medical devices or vehicles. A crucial nuance, reflected, for example, in Article 6(3) EU AI Act, is whether the AI merely performs narrow, non‑decisive tasks (such as formatting or spell‑checking) or materially influences the outcome; only the latter is generally captured by high‑risk classifications.

Core Functional Criteria for High-Risk Classification

The functional classification of a high-risk AI system hinges on its role in a decision-making process. The core principle, as refined by the EU AI Act, is that a system's function is high-risk if it is more than a mere procedural tool. The critical test is whether the AI's output "materially influences" a consequential outcome. This distinguishes a spell-checker from a CV-screening algorithm that rejects candidates. A system's operation in critical infrastructure or as a safety component in a regulated product further solidifies its high-risk designation.

India's Approach to Functional High-Risk Classification

India's framework does not have a single, equivalent statutory test to the EU's Article 6(3). However, the same functional logic of "consequential influence" is being built through a combination of data protection law, sector-specific guidelines, and the 2025 India AI Governance Guidelines. The concept of an AI system "materially influencing" an outcome is functionally addressed by focusing on systems that make or inform decisions that significantly affect individuals.

The DPDP Act, 2023 and the "Significant Data Fiduciary"

The Digital Personal Data Protection Act, 2023, creates a functional risk classification via the category of Significant Data Fiduciary (SDF) . Much like the concept of "high-risk controllers" under GDPR, the SDF framework embodies India’s risk-based approach to data protection. An entity can be designated as an SDF by the Central Government based on factors such as the volume and sensitivity of personal data processed, the risk of harm to data principals, and the potential impact on the sovereignty and integrity of India. This functional designation directly applies to AI systems.

The India AI Governance Guidelines (2025) and the AI (Ethics and Accountability) Bill, 2025

The MeitY Guidelines explicitly adopt a risk-based, evidence-led, and proportional governance approach, declaring that they "do not allow unrestricted deployment of high-risk AI systems". The functional criteria for high-risk are articulated through specific "areas of risk" that focus on the nature of the harm a system can cause, rather than its underlying technology. The Guidelines identify seven such areas, which serve as a functional classification:

(i) malicious use,

(ii) bias and discrimination,

(iii) transparency failures,

(iv) systemic risk,

(v) loss of control,

(vi) national security threats, and

(vii) risk to vulnerable groups.

The proposed Artificial Intelligence (Ethics and Accountability) Bill, 2025, creates an even more direct functional test. Section 5(2) establishes an ex ante authorisation regime for high-risk AI systems operating in sensitive domains such as surveillance, law enforcement, employment, healthcare, and credit.

TYPES OF HIGH-RISK AI SYSTEMS

High‑risk AI systems can be grouped by the sectors they affect, the functions they perform in decision‑making, and the legal triggers that turn a system into “high‑risk” (for example, being listed in a statute, or materially influencing outcomes). These categories are used by legislators, regulators, and courts to decide which systems must go through stricter risk management, documentation, and oversight processes. Under Chapter III, Section 1, Huergo explains that the AI Act classifies systems as “high‑risk” through two main routes which are AI that is a safety component of products covered by EU harmonisation legislation (Annex I), and AI whose intended purpose falls within one of the high‑risk use areas listed in Annex III.

Sector‑based types

• Justice and law enforcement: Predictive policing, crime‑hotspot mapping, suspect ranking, and facial recognition are treated as high‑risk because they can lead to wrongful stops, arrests, and surveillance of marginalised groups. The EU AI Act lists several law‑enforcement uses as high‑risk in Annex III, and puts some biometric uses in an even stricter “unacceptable risk” category.
• Credit, finance, and social protection: Credit‑scoring, fraud detection and welfare‑eligibility scoring are high‑risk because they control access to essential financial and social services. Controversies such as SCHUFA scoring and the Apple Card case show how opaque models can become gatekeepers to economic opportunities, triggering data‑protection and anti‑discrimination scrutiny.
• Employment, education, and opportunities: Hiring algorithms, CV screeners, exam‑proctoring, and student‑scoring systems are treated as high‑risk in the EU AI Act because they shape access to jobs and education. In US and Indian debates, they are framed as high‑impact or high‑risk tools that can embed historic labour and educational bias into data‑driven scoring.
• Health, medical devices, and public health: Clinical decision‑support, diagnostics, and AI embedded in medical devices are high‑risk because errors can directly threaten life and bodily integrity. Under EU product‑safety law, AI that serves as a safety component of regulated devices is automatically high‑risk under the AI Act.
• Critical infrastructure and essential services: AI used to manage power, water, transport and other critical infrastructure is high‑risk because failures can cause widespread physical harm or disruption. Annex III of the EU AI Act expressly covers AI for managing and operating critical infrastructure, and similar worries appear in US and Chinese cyber‑physical security policy.
• Migration, asylum, and border control: Visa‑risk tools, asylum‑screening algorithms and border‑surveillance AI are high‑risk because they affect freedom of movement and protection from refoulement. The EU AI Act puts these systems in Annex III and subjects them to high‑risk obligations, while restricting public visibility of their entries in the EU database for security reasons.

Function‑based types

• Fully automated decision‑making systems: Systems that take binding decisions without human review (like automated loan denials or welfare sanctions) are the paradigm of high‑risk AI because they can directly cause harm without meaningful human intervention. Data‑protection laws such as the GDPR impose special safeguards here, including rights to explanation and human review.
• Decision‑support with strong influence: Judicial, medical and policing tools labelled “decision‑support” can still strongly steer human choices by ranking cases, flagging “risk” or proposing defaults. The EU AI Act and India’s draft AI (Ethics and Accountability) Bill 2025 treat such systems as high‑risk whenever they materially influence outcomes, even if a human signs off at the end.
• Safety components of regulated products: AI that functions as a safety component in products such as industrial machinery, vehicles, medical devices or toys is automatically high‑risk in the EU framework. This covers embedded AI, often invisible to users, whose malfunction can cause serious harm; commentators such as Huergo note that their high‑risk status derives from the underlying sectoral product‑safety law, so AI and product share a single conformity‑assessment chain.

Legal‑trigger‑based types

• Explicit lists (Annex / schedule models): The EU AI Act’s Annex III is the clearest example of a legally fixed high‑risk list: if an AI system’s intended purpose matches a listed use (biometric identification in policing, credit‑scoring, educational scoring, migration risk assessment, critical infrastructure management), it is presumed high‑risk and must meet Chapter III, Section 2 obligations. This offers legal certainty but can lag behind new applications, so Article 7 allows the Commission to update Annex III over time.
• Material‑influence tests (Article 6(3) nuance): Article 6(3) EU AI Act allows certain AI systems near Annex III domains to be excluded from the high‑risk regime if they do not materially influence decisions and only perform narrow auxiliary tasks such as formatting or simple flagging. Huergo stresses that this carve‑out is meant for tools that stay at limited support functions, not systems that meaningfully shape outcomes; similar distinctions appear in policy debates about tools like SUVAS, which supports translation within high‑risk domains like courts but is not itself treated as high‑risk.
• Indian proposals (AI Bill 2025, section 5(2)): India’s Artificial Intelligence (Ethics and Accountability) Bill, 2025 would allow rules or schedules to designate “high‑risk” categories; any system within them must undergo prior review by an Ethics Committee, especially when used in surveillance, law enforcement, credit, employment or health. Section 5(2) requires providers/deployers to submit documentation on purpose, training data, bias‑mitigation and human‑oversight, creating a legal trigger: being notified as “high‑risk” activates mandatory pre‑deployment scrutiny.
• US “rights‑impacting” and “safety‑impacting” categories: US federal policy under OMB M‑24‑10 does not use the term “high‑risk AI system,” but requires agencies to flag AI use‑cases as “rights‑impacting” or “safety‑impacting” in their inventories, effectively creating high‑risk categories. These use‑cases must undergo heightened risk management, testing and oversight, even though there is no single statutory high‑risk definition.

APPEARANCE IN OFFICIAL DOCUMENTS

The category of “high‑risk AI systems” is no longer confined to legislative text; it now appears across multiple layers of regulatory registers, inventories, and observatories that record how such systems are designed, deployed, and monitored. These instruments vary in their legal force and epistemic focus: some are binding, forward‑looking compliance registries, while others are descriptive inventories intended to support coordination and oversight across government. Together, they determine who must disclose high‑risk AI use, what must be disclosed, and who can see it, thereby shaping how researchers, litigants, and the public can analyse these systems.

In India, “high‑risk AI systems” are recognised in official policy, but there is not yet a single, binding public registry dedicated to them. The India AI Governance Guidelines, issued through the Press Information Bureau and NeGD, explicitly state that India “will not allow unrestricted deployment of high‑risk AI systems” and propose an India‑specific risk‑assessment and classification framework for AI deployments across sectors. These Guidelines treat high‑risk AI as a distinct category for regulatory attention, especially in domains like deepfakes, algorithmic discrimination, national security, and systemic risk, but at this stage, they operate as a governance framework rather than a statutory registry.

Institutionally, the Guidelines envisage new apex bodies such as the AI Safety Institute (AISI) and the India AI Governance Group (AIGG), which are tasked with testing and evaluating AI systems, conducting risk assessments, advising regulators, and coordinating a “national, federated AI incident reporting mechanism” for AI‑related harms. This incident system is intended to collect reports of failures and harms, with special priority for high‑risk domains, effectively functioning as a harm‑centric database rather than a catalogue of all high‑risk systems.

The proposed Artificial Intelligence (Ethics and Accountability) Bill, 2025, adds a second layer of documentation. It establishes a statutory Ethics Committee for Artificial Intelligence, a multi‑stakeholder body empowered to develop ethical guidelines, monitor compliance, and investigate AI‑related harm, and requires high‑risk AI deployments designated under the Bill to be notified to and reviewed by this Committee. Although the Bill does not yet create a public‑facing “high‑risk AI registry”, the Ethics Committee’s review and case records would operate as an official internal database of high‑risk AI uses and harms, accessible to the government and potentially to courts and auditors, thereby formalising high‑risk AI as an object of state record rather than just policy discourse.

Taken together, these instruments mean that in India, high‑risk AI systems appear in three kinds of official documents:

• policy frameworks (India AI Governance Guidelines) that define high‑risk domains and call for a federated incidents database;
• legislative proposals (AI Ethics and Accountability Bill 2025) that require high‑risk systems to be filed with and scrutinised by an Ethics Committee;
• and sectoral guidance, where regulators such as CERT‑In and NCIIPC are identified as key data creators for cyber and infrastructure‑related AI risks.

RESEARCH THAT ENGAGES WITH HIGH-RISK AI SYSTEMS

Research on engaging with high-risk AI in India's justice system is an increasingly large body of research that is also internally fragmented across various disciplines. However, it is connected by a common institutional focus: a justice system in structural crisis. As of January 2024, there are 34 Supreme Court judges functioning at full strength, 783 High Court judges working against the sanctioned strength of 1,114, and 22,677 judges in subordinate courts. The main reasons for delays in the administration of justice are the backlog of cases, lack of numbers, and procedures. As of September 2025, India's judicial system is said to have a backlog of more than 50 million cases, which would take human judges more than 300 years to resolve at the current rate.

Use Of AI Tools Like SUPACE In Indian Judiciary: Constitutional Safeguards For Due Process And Judicial Independence

Contribution

This article provides one of the earliest doctrinal analyses of SUPACE as a judicial‑support AI tool and examines whether its deployment is compatible with constitutional guarantees of due process and judicial independence. It contributes to the high‑risk AI debate by arguing that, even where AI is framed as “assistive,” tools like SUPACE can have high‑risk characteristics because they operate in a structurally overburdened justice system and may influence judicial reasoning in opaque ways.

Content

Nagpal, Uppal and Harshita describe how SUPACE uses machine‑learning techniques to ingest case files, identify relevant precedents and generate summaries for judges of the Supreme Court of India. They note that while these features promise efficiency and better case management, they also raise concerns about bias in training data, lack of transparency in how results are prioritised, and the possibility of undue reliance by judges pressed for time. The authors argue that, given the stakes of constitutional adjudication and criminal matters, SUPACE should be treated as a high‑risk AI system requiring strong ex ante safeguards, including clear documentation of its design, independent validation, and the preservation of meaningful human oversight in the form of informed, not merely formal, judicial review of its outputs.

The Role of Artificial Intelligence in Modern Courts: A Tool of Efficiency or a Threat to Justice?

Contribution

This open‑access working paper surveys judicial AI initiatives globally and asks whether the use of AI in courts enhances efficiency at the cost of fairness and legitimacy. It contributes to high‑risk AI discourse by proposing criteria for when court‑focused AI should be considered high‑risk, emphasising impacts on due process, equality of arms and public trust rather than purely on technical sophistication.

Content

The authors review examples of AI in courts across several jurisdictions, including tools for document management, outcome prediction and case triage, and highlight the particular challenges faced by under‑resourced systems such as India’s. They argue that AI systems, which prioritise cases, suggest outcomes or structure the information that judges see, can have high‑risk effects because they alter the epistemic environment in which decisions are made, even if they do not issue binding rulings themselves. The paper recommends that judicial AI be classified as high‑risk whenever it can materially influence access to a fair hearing, and calls for independent audits, impact assessments and strong explanation rights tailored to the judicial context.

Impacts and ethics of using Artificial Intelligence (AI) by the Indian police

Contribution

This empirical and normative study examines how AI tools are used by Indian police forces and evaluates their ethical and social implications. It offers an Indian law‑enforcement‑specific account of high‑risk AI, focusing on applications like facial recognition, crime analytics and surveillance systems that can significantly affect liberty, privacy and non‑discrimination.

Content

The article documents the adoption of AI‑driven tools, including automated facial recognition, predictive crime‑mapping, and large‑scale data‑integration platforms used for investigations and crowd management. It identifies key concerns such as inaccuracies in facial recognition, lack of clear legal frameworks, potential for mass surveillance, and the reinforcement of existing social biases through historically skewed policing data. The authors argue that such systems qualify as high‑risk AI because they are deployed in coercive contexts where errors or biases can directly lead to wrongful suspicion, arrest or harassment, and advocate for robust legal safeguards, human rights impact assessments and independent oversight before further expansion of AI in policing.

Artificial Intelligence in the Indian Judiciary: SUPACE, SUVAS, and the Limits of Assistive Automation

Contribution

This policy‑oriented study situates SUPACE and SUVAS within the broader e‑Courts ecosystem and questions the narrative that such tools are purely “assistive.” It advances the notion that, in a system facing massive backlogs and severe capacity constraints, assistive AI can become effectively high‑risk because its recommendations and translations are likely to shape outcomes even if formal decision‑making authority remains with human judges.

Content

The paper explains how SUVAS is used to translate judgments and orders between English and Indian languages, while SUPACE assists in legal research and case summarisation, both aimed at improving access to justice and reducing delay. Drawing on interviews and official documents, the author(s) argue that the opaque nature of these systems and the lack of publicly available audit mechanisms make it difficult to assess their error rates, biases or distributional impacts. The piece warns that the combination of technological opacity and institutional pressure to clear arrears can turn these tools into de facto high‑risk AI systems, and recommends stronger transparency commitments, pilot‑phase evaluations, and explicit judicial guidelines on appropriate use and limits of AI assistance.

HIGH RISK AI SYSTEMS IN INTERNATIONAL INSTRUMENTS

International instruments do not always use the exact term “high‑risk AI system”, but they converge on a shared idea: AI uses that pose serious risks to human rights, democracy, safety, or the environment must be subject to heightened duties of risk assessment, oversight and accountability. These soft‑law standards shape how domestic legislators identify and regulate high‑risk AI, even where they remain non‑binding.

UNESCO Recommendation on the Ethics of Artificial Intelligence, 2021

The UNESCO Recommendation on the Ethics of AI, adopted unanimously by 193 Member States in November 2021, is the first global standard‑setting instrument on AI ethics. It does not create a closed list of “high‑risk AI systems,” but builds a risk‑sensitive framework around ten core principles, including proportionality and do‑no‑harm, safety, security, human oversight, transparency and fairness.

The Recommendation requires that the use of AI systems “must not go beyond what is necessary to achieve a legitimate aim” and that risk assessments be used to prevent harms, including unwanted safety risks and vulnerabilities to attack. It explicitly calls for Ethical Impact Assessments for AI systems that pose potential risks to human rights, requiring broad testing (including, where appropriate, real‑world testing) before such systems are released onto the market. In high‑risk contexts, the Recommendation demands robust oversight mechanisms to identify, prevent, mitigate and account for the impact of AI systems on human rights, the rule of law and inclusive societies, and it contains a breakthrough provision discouraging the use of AI for social scoring and mass surveillance.

OECD AI Principles and AI Risk‑Management Frameworks

The OECD AI Principles, endorsed by OECD and partner countries, promote trustworthy AI and call on governments to develop interoperable, risk‑based approaches to AI governance. While the Principles do not list high‑risk use cases, they urge states to pay special attention to AI deployments that affect safety, critical infrastructure, and fundamental rights, and to ensure that those developing and deploying AI are accountable for their systems’ proper functioning.

Building on these Principles, the OECD has developed tools such as the Framework for the Classification of AI Systems and the High‑Level AI Risk Management Interoperability Framework, which help policymakers characterise AI systems and structure risk‑management practices. These frameworks emphasise that AI systems with significant impact on individuals or society require more intensive risk assessment, documentation, and monitoring, and they stress properties like robustness, safety, transparency, explainability, and fairness as essential for high‑impact uses. In practice, they provide a conceptual template for identifying and governing high‑risk AI across jurisdictions, even though they stop short of creating binding obligations.

Council of Europe Framework Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law

In the May of 2024, the Council of Europe adopted the Framework Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law, the first international treaty on AI. Unlike the EU AI Act’s tiered risk‑pyramid, the Convention takes a more contextual risk‑based approach, requiring continuous assessment of the risks posed by particular AI systems in light of their context, stakeholders, and potential human‑rights impacts.

The Convention obliges state parties to establish oversight and accountability mechanisms to prevent and mitigate risks to human rights, democracy and the rule of law arising from AI systems, with particular attention to high‑impact deployments in areas such as law enforcement, justice, and public administration. It does not enumerate high‑risk sectors in an annex, but instead requires that risk assessments and safeguards be proportionate to the system’s potential adverse effects, effectively treating certain applications as high‑risk whenever they have significant implications for rights and democratic processes.

European Union

EU Database for High‑Risk AI Systems (EUDB): Article 71 of the EU AI Act requires the European Commission, in collaboration with Member States, to set up and maintain an EU database containing information about high‑risk AI systems listed in Annex III and certain other registered systems. Providers (or authorised representatives) must enter the information specified in Annex VIII Sections A and B before placing a high‑risk system on the market, while public‑sector deployers must enter deployment‑related data under Section C, making the EUDB a binding, pre‑market registry for high‑risk AI. Most entries are publicly accessible and machine‑readable, with only certain security‑sensitive law enforcement and migration systems kept in a secure, non‑public segment.

The EU AI Act takes a risk-based approach to high-risk AI systems (primarily described in ANNEX III) comprising several sectors such as law enforcement, migration, employment, access to essential services and administration of justice. Nonetheless, the framework also identifies significant carve-outs from article 6(3), so that systems are not labelled high risk where they merely perform well-defined procedural tasks, or if they assist rather than substitute for human decisions or solely identify patterns without affecting outcomes. This shows that classification is not automatic but rather contingent on the concrete effects of the AI system on people and fundamental rights, strengthening a nuanced and context-sensitive regulatory approach.

Singapore: Model AI Governance Frameworks and Agentic AI Guidelines

Singapore does not use the exact legal term “high‑risk AI system”, but its official guidance effectively targets high‑risk and high‑impact AI through its Model AI Governance Frameworks and more recent guidance on agentic AI. These instruments translate abstract international principles into detailed, operational guidelines for organisations deploying AI in sensitive, high‑impact contexts such as finance, healthcare, and public‑facing services.

The original Model AI Governance Framework (second edition, 2020), issued by the Personal Data Protection Commission (PDPC) and IMDA, sets out four pillars: (1) internal governance structures and measures, (2) determining the level of human involvement in AI‑augmented decision‑making, (3) operations management, and (4) stakeholder communication. While formally voluntary, the framework has been effectively made mandatory for regulated financial institutions via the Monetary Authority of Singapore’s (MAS) Technology Risk Management and AI‑model risk guidelines, which require board‑level oversight, documented AI risk‑management processes, and rigorous validation before deployment of high‑impact models in credit, fraud detection and other critical functions.

In 2024–2026, Singapore extended this approach to generative and agentic AI, publishing a Model AI Governance Framework for Generative AI and a Model Governance Framework for Agentic AI that identify key risks such as uncontrolled autonomy, tool abuse, and complex delegation chains. These frameworks recommend bounding an AI agent’s autonomy and tool access based on risk, implementing lifecycle testing and monitoring, and ensuring meaningful human oversight at critical checkpoints, particularly for agents that can take actions affecting individuals’ rights or safety. Singapore also supports technical governance through AI Verify, an open‑source testing framework that allows organisations to generate standardised governance and performance reports for their AI systems, including those that would qualify as high‑risk under EU‑style regimes.

Taken together, Singapore’s guidelines demonstrate a soft‑law, implementation‑focused model for governing high‑risk AI: instead of a statutory high‑risk list, they operationalise risk through governance pillars, lifecycle controls, and sector‑specific obligations, especially in finance and other critical sectors.

Other Countries and International Guidance

Several other jurisdictions and international frameworks similarly operationalise high‑risk AI, even when they do not always use that exact label.

Peru

Law No. 31814 (2023): Peru’s AI law establishes a risk‑based regime with prohibited and high‑risk categories, requiring AI developers and operators to implement strict data‑quality, transparency, human‑oversight and documentation measures for systems classified as high‑risk. Draft implementing regulations further specify requirements for risk classification, lifecycle documentation and accountability measures within the national digital‑transformation system.

United States

NIST AI Risk Management Framework: Although NIST is a US agency, its AI Risk Management Framework is widely used internationally as a voluntary standard. It organises AI risk management into four functions—govern, map, measure, and manage—and is frequently cited as a way to operationalise high‑risk AI obligations, including those in Article 9 of the EU AI Act, which requires a risk‑management system for all high‑risk AI systems but leaves the method open. Many organisations subject to EU rules adopt the NIST AI RMF internally to generate the evidence trail that notified bodies expect when reviewing high‑risk systems.

Federal AI Use Case Inventories: In the OMB Memorandum M‑24‑10 and the Advancing American AI Act, the federal agencies must maintain and publish AI use‑case inventories that identify systems as “rights‑impacting” or “safety‑impacting,” categories that functionally map onto high‑risk AI. These inventories list the system’s purpose, development stage, data characteristics, and risk‑management measures, and are aggregated across government, creating a decentralised but consolidated disclosure mechanism for high‑risk uses, even though the phrase “high‑risk AI system” is not itself a statutory term.

California and frontier AI safety laws

Recent US state‑level measures, such as California’s AI safety laws, focus on “frontier” or highly capable AI models and establish enhanced safety, incident‑reporting and oversight duties for developers whose models can pose systemic or catastrophic risks. These measures do not always use the phrase “high‑risk AI systems,” but they clearly target a high‑risk class of systems based on capability and potential impact, requiring rapid reporting of serious incidents to state authorities and creating enforcement powers for attorneys‑general.

These experiences show that, beyond EU‑style annexes, high‑risk AI is increasingly being concretised through guidelines, risk‑management frameworks, and sectoral rules—with Singapore’s model governance frameworks and Peru’s Law 31814 offering clear examples of risk‑tiered obligations, and NIST’s AI RMF serving as a widely adopted method for implementing high‑risk AI controls across borders.

WAY AHEAD: ADDRESSING DATA CHALLENGES IN HIGH-RISK AI SYSTEMS

The convergence of urgency in regulation, the rise of AI incidents, and the emergence of stakeholder consensus create a critical window for action between 2025 and 2026. The time for debate about the ethics of AI is over, and it is time to take action to protect sensitive data and stakeholder trust, with several drivers accelerating at once, including the continued march of regulation, public interest in AI data practices, and competitive differentiation based on responsible AI practices.

The way forward is best understood as consisting of six interlocking pillars, each based on stakeholder recommendations and/or institutional literature.

Establishing A Robust Data Governance as the Non-Negotiable Foundation

The most significant factors for data governance in the AI system, according to the 2024 State of Data Intelligence report, are data quality (42%), security (40%), and analytics (40%). As another industry expert explains, "AI systems thrive on clean, contextualised, and accessible data for the users and for the developers. Without this, the businesses continue to face the risk of inefficiencies they aim to eliminate."  

For data governance, organisations need to establish a foundation in three areas: visibility and inventory (where AI is used and what data it touches), controls (access and ethics), testing and validation (how to ensure AI works well and within risk bands) and finally, how to monitor AI to ensure it remains effective and within risk bands.  

The OECD’s flagship report on governing with AI for 2025 proposes a three-part model for good governance: first, ‘enablers,’ which include governance structures, digital infrastructure, data management, finance, and workforce skills; second, ‘guardrails,’ including rules, accountability, and transparency; and third, ‘engagement,’ including citizens, civil society, and businesses.

Deploying Privacy-Enhancing Technologies (PETs) and Federated Learning

According to the Cloud Security Alliance, organisations should invest in Privacy-Enhancing Technologies such as federated learning and differential privacy as mitigating factors to achieve compliance and innovation, along with ISO/IEC 42001 and EU AI Act Risk Tiers.

Federated learning is a machine learning approach that is structurally significant for a way forward. It is a machine learning approach that enables artificial intelligence models to be learned across decentralised devices or servers that contain local data samples. The data is not required to be transferred or stored in a centralised location. It is a privacy and security-friendly approach for collaborative artificial intelligence development across borders in a manner that is compliant with national legal constraints.

Regulatory agencies are already in the process of piloting this. Swissmedic, along with other agencies, proposes to utilise federated learning to improve TRICIA, a machine learning approach for improving artificial intelligence for evaluating incoming reports of critical incidents related to medical devices.

Scaling Synthetic Data: With Strong Governance

Synthetic data can result in better outcomes if organisations focus on effective governance, transparency, and multistakeholder engagement, bridging the gap between the developers and users of the technology and the business leaders, lawyers, and policy makers who influence its adoption. Every stakeholder has a unique governance role to play, which no one else can do. The most critical thing to focus on is investing in data traceability. A strong system of provenance helps organisations trace the entry point of synthetic data and reduces risks of bias and AI autophagy. The WEF suggests to policymakers and business leaders that the best way forward with synthetic data governance is to develop context-aware standards, acknowledging the unique characteristics of synthetic and simulated data, and engaging with privacy and AI regulators to ensure alignment with emerging frameworks. In financial services, there is emerging evidence, and the 2023-2025 pilots by the FCA resulted in 60% similarity in data for fraud detection, which improved the model by 15%.

HIGH-RISK AI SYSTEMS ARE ALSO KNOWN AS

In India, the term 'high-risk AI systems' is mainly used with the same name; there is not yet a fixed legal label like in the EU AI Act. Government documents and media refer directly to 'high-risk AI systems' when talking about AI uses that can seriously affect safety, rights, or critical sectors. The new India AI Governance guidelines talk about high-risk domains and bar the unrestricted deployment of high-risk AI systems, but they do not rename them. The broader regulatory discussion is framed around 'responsible AI', 'AI governance guidelines', 'risk - based approach', and in data-protection law, around 'significant data fiduciaries' under the DPDP Act, which is a related risk category for data controllers, not a separate name for high-risk AI systems.