Algorithmic Bias

WHAT IS ALGORITHMIC BIAS

Algorithmic bias refers to systematic and repeatable errors in the outputs of algorithmic or artificial intelligence (AI) systems that result in unfair, discriminatory, or unequal results. AI systems use algorithms to learn patterns from historical data and gain insights from inputs to produce an output. Now, if the data contains any existing social inequalities or imbalances, then these biases can be reproduced and enhanced in the system’s output.

Such biases can also emerge from design choices where developers make decisions relating to how to process data and what outcomes to optimize. These seemingly neutral variables can act as proxies for sensitive attributes and result in unfair outputs.

Another source is the presence of feedback loops. In such cases, the output of an algorithm influences real-world outcomes, which are then fed back into the system as input, which again uses that biased input to reinforce or amplify existing biases.

While these sources show how algorithmic bias can produce harmful and discriminatory effects, this understanding is not complete. Algorithmic bias is usually seen as negative, often leading to unfair or discriminatory outcomes. But that is not always the full picture. Not all kinds of biases are harmful. In some situations, a certain kind of bias is actually needed to make the outcomes fairer. If we try to remove all biases and make systems completely “neutral”, we might end up making things worse for marginalised people. This becomes very clear in the healthcare sector. Researchers at the National Institutes of Health (NIH) have proposed a framework to repurpose bias as a tool for equity.  Many so-called “neutral” algorithms fail to diagnose or predict diseases in minority groups properly. For example, a system that ignores race might underestimate diabetes risk in Asian patients or fail to correctly identify skin conditions in people with darker skin. For example, in diabetes screening, Asian patients may face higher risks at the same BMI, but a race-blind algorithm cannot adjust for this, leading to underdiagnosis. This highlights a key tension that a process can be technically fair but still produce unfair results. To fix this, researchers have suggested using what we can call “corrective bias.” This means intentionally designing systems to pay extra attention to groups that have been historically overlooked. Studies show that when algorithms take factors like race into account, they can make better and more accurate decisions. In law, this idea is similar to affirmative action. Treating everyone the same does not always lead to fairness, so systems may intentionally favour disadvantaged groups to balance inequalities. In this sense, some bias is not only acceptable but necessary for achieving real equality.

OFFICIAL DEFINITION OF ALGORITHMIC BIAS

‘ALGORITHMIC BIAS’ AS DEFINED IN LEGISLATION

Currently, India does not have a single authoritative legal definition of “algorithmic bias” in any enacted legislation. However, several existing statutes contain provisions that, read purposively, create a framework for addressing algorithmic bias.

Legal Provision Relating to ‘Algorithmic Bias’

The provisions relating to algorithmic bias are complemented by constitutional guarantees of equality and non-discrimination, and emerging policy frameworks that seek to govern artificial intelligence systems.

Constitutional Guarantees

Article 14 guarantees equality before the law and equal protection of the law. Algorithmic systems producing disparate outcomes that disproportionately burden marginalized groups may violate this guarantee.

Article 15 prohibits discrimination on grounds of religion, race, caste, sex, or place of birth. AI systems that replicate or amplify discrimination along these lines engage constitutional scrutiny.

Article 21 protects the right to life and personal liberty, interpreted to include the right to privacy and dignity. Opaque algorithmic decisions affecting entitlements or liberty implicate these rights.

Digital Personal Data Protection Act, 2023 (DPDP Act)

The DPDP Act does not explicitly define ‘algorithmic bias,’ however, it provides a statutory framework regulating automated data processing, which forms the basis of algorithmic systems. It focuses on lawful processing, consent and accountability rather than directly addressing discriminatory algorithmic outcomes.

Section 2(x) defines “processing” to include operations performed on personal data, including collection, storage, use and dissemination, implicitly bringing algorithmic decision-making systems within the scope of this regulation.

The Act requires that personal data be processed only for lawful purposes, with consent of legitimate use, in a manner ensuring accuracy and security. Section 8(3) imposes an obligation on Data Fiduciaries to ensure that when personal data is used for making individual decisions, presumably including automated or algorithmic decisions, such data must be kept accurate, consistent, and complete. Furthermore, Section 11 grants data principals the right to access information about processing, creating a limited transparency obligation regarding algorithmic logic.

This requirement of data accuracy and correction indirectly mitigates algorithmic bias by enabling individuals to rectify erroneous data that may produce unfair outcomes. In sum, while the Act regulates the inputs and processes of algorithmic systems, it does not explicitly regulate their outputs, thereby only indirectly addressing algorithmic bias.

Information Technology Act, 2000 (IT Act)

Section 79 provides a conditional safe harbour to intermediaries. The protection is lost if an intermediary fails to comply with due diligence obligations, including the requirement to prevent bias and discrimination under the IT Rules.

Rule 4(4) of the IT (Intermediary Guidelines and Digital Media Ethics Code) Rules, 2021 requires significant social media intermediaries using automated tools for content screening to periodically review such tools for “accuracy, fairness, propensity for bias or discrimination, and impact on privacy and security”. This is the first explicit statutory articulation of algorithmic fairness standards in Indian subordinate legislation.

IT (Intermediary Guidelines) Amendment Rules, 2026 introduce a statutory definition of “synthetically generated information” (SGI) and mandate labelling, metadata embedding, expedited takedown (within three hours of notification).

Consumer Protection Act, 2019

Section 2(47) defines “unfair trade practice” to include any practice that adopts unfair or deceptive methods. Under Section 18, the Central Consumer Protection Authority (CCPA) has issued Guidelines for Prevention and Regulation of Dark Patterns, 2023, which covers algorithmic designs and AI‑driven misleading experience patterns as unfair trade practices.

Chapter VI (Sections 83‑87) imposes product liability for defective products or deficient services. An AI product or service whose algorithmic bias causes harm to a consumer (e.g., a biased credit‑scoring system or a misdiagnosing medical algorithm) may be characterised as a “defective product” under this Chapter, making developers and deployers jointly liable.

‘ALGORITHMIC BIAS’ AS DEFINED IN INTERNATIONAL INSTRUMENTS

UNESCO Recommendation on the Ethics of AI, 2021

These Recommendations highlight that the algorithm used in AI has the potential to “reproduce and reinforce existing biases.” They call on AI actors to take measures so that these biased applications and outcomes are minimized, and fairness is ensured in the outputs. Further, the Recommendations call on Member States to conduct algorithmic impact assessments, ensure the explainability of high-stakes AI decisions, and provide effective remedies for individuals harmed by biased systems.

OECD AI Principles (2019)

While the OECD Principles do not directly address algorithmic bias, they do emphasize fairness, transparency, and accountability in AI systems. They stress that AI actors must respect human-centered values throughout the AI lifecycle, including non-discrimination and equality.

They talk about how if skewed information is put into an AI system, those biases can become amplified by the algorithm and perpetuate discrimination. Therefore, AI actors have to give life to responsible AI usage. While the OECD framework is principles-based, it signals that responsible AI requires active efforts to prevent and mitigate bias.

General Data Protection Regulation (GDPR)

The GDPR establishes a normative and rights-based framework that effectively addresses the risks associated with biased automated decision-making. While not an AI specific law, it provides powerful tools for challenging algorithmic bias. Article 22 gives individuals the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects. This includes “profiling,” which is broadly defined to cover any automated analysis of personal characteristics such as performance, health or behavior.

Read together with Recital 71, the GDPR makes clear that automated decision-making must be subject to safeguards that prevent discriminatory effects, particularly those based on sensitive characteristics such as race, ethnicity, political opinion, religion, or health.

In effect, algorithmic bias under the GDPR may be understood as the production of unjustified or discriminatory outcomes through automated processing of personal data, particularly where such processing significantly affects individuals and lacks adequate safeguards such as human intervention, transparency, or contestability.

International Organisation for Standardisation (ISO)- Bias in AI Systems and AI-Aided Decision Making

The standard defines bias across the full AI system lifecycle, including data collection, training, continual learning, design, testing, evaluation, and use. It simply defines bias as the “systematic difference in treatment of certain objects, people, or groups in comparison to others.” Furthermore, it also defines automation bias as the human tendency to favour recommendations from automated systems even when they conflict with correct information. These definitions are important because they distinguish between bias that arises from technical design and the psychological bias that can lead humans to rely on flawed or biased algorithms.

EU Artificial Intelligence Act

The EU AI Act is the most comprehensive AI-specific statute globally. While it does not define ‘algorithmic bias’ as a standalone term, it integrates bias mitigation into its core requirements.

For high-risk AI systems, Article 10(2)(f) requires training, validation, and testing data to be subject to data governance practices so that possible biases do not have any negative impact on fundamental rights. Further, data outputs influence inputs for future operations, so it is necessary that data governance principles to be in place.

Article 10(5) permits processing of sensitive personal data when “strictly necessary for the purposes of ensuring bias monitoring, detection and correction,” an important carve out for effective auditing.

Additionally, Article 5(1)(c) prohibits certain practices outright, such as social scoring by AI systems on the basis of social behaviour or personal characteristics of natural persons, that would lead to detrimental or unfavourable treatment.

European Commission Staff Working Document- Impact Assessment

As the EU AI Act does not explicitly define ‘algorithmic bias,’ the accompanying European Commission document can be seen to understand biases in data or models. It clarifies that bias can originate in training data, which may contain hidden prejudice, or in the algorithms themselves, which by way of their reasoning mechanisms may prefer certain characteristics over others.

Council of Europe Framework Convention on AI (CAHAI)

Article 10 of the Convention requires the parties to ensure that AI systems respect equality and prohibit discrimination under domestic and international law.

It further requires parties to identify, assess, prevent, and mitigate risks of discrimination arising from AI systems, including indirect discrimination caused by the use of seemingly neutral proxies that have disparate impacts on protected groups. This treaty is significant because it moves beyond the general principles to impose concrete obligations on states to actively prevent discriminatory outcomes from AI.

The European Union Agency for Fundamental Rights (FRA)- Bias in Algorithms Report

In its report on bias in algorithms, the agency highlights how algorithmic systems can produce and reinforce discriminatory outcomes, particularly in high-risk domains such as predictive policing and online hate speech detection. The report emphasizes the role of feedback loops, where algorithmic predictions influence real-world outcomes, which are then fed back into the system as training data, thereby reinforcing existing biases over time.

The FRA recommends that EU institutions mandate regular bias assessments, improve guidance on the use of sensitive data, and ensure greater transparency and accountability in AI systems. It also calls for broader discrimination assessments covering multiple protected characteristics and stresses the importance of promoting linguistic diversity in AI development.

The Toronto Declaration 2018- Protecting the right to equality and non-discrimination in machine learning systems

The Toronto Declaration is a non-binding, soft law instrument that does not impose direct legal obligations. However, it derives its normative force from existing international human rights law and is addressed to both states and private actors, urging them to prevent discriminatory outcomes in the design and deployment of algorithmic systems.

It explains algorithmic bias as reinforced patterns of inequality, that arise from biased datasets, by machine learning systems. The people behind the technology bring their own biases which then reflect in the inputs given to the technology, further reflected in the output produced.

It therefore place obligation on both the state and private actors to prevent discriminatory outcomes, ensure transparency, conduct regular audits, and provide remedies to the individuals affected by these biased outputs.

Comparative Significance

Taken together, these international instruments reveal a global consensus that algorithmic bias is a governance problem that requires structured responses. Some approach it through individual rights and data protection, while others adopt a risk-based regulatory framework. For India, these instruments offer both a benchmark and a toolkit, for adaptation to local legal and social contexts.

‘ALGORITHMIC BIAS’ AS DEFINED IN OFFICIAL DOCUMENT(S) AND REPORTS

These official documents and report address the concept of ‘algorithmic bias’ through related language like ‘fairness,’ ‘non-discrimination,’ ‘bias-free AI,’ and ‘equitable outcomes’  across the following documents.

The Artificial Intelligence (Ethics and Accountability) Bill, 2025

The Artificial Intelligence (Ethics and Accountability) Bill, 2025 (Lok Sabha Bill No. 59 of 2025) proposes the first statutory definition in Indian legislation. It is important to note that it is a Private Member's Bill, which are very rarely proceeded to enactment. The legislative history of the Lok Sabha shows only fourteen Private Member's Bills passed since Independence. It is therefore more appropriately treated as a significant policy document, rather than as a legislative instrument with prospective legal force.

Clause 2(b) of the Bill defines:

““algorithmic bias” refers to the presence of systematic errors in AI systems that result in unfair outcomes.”

This Bill, though a Private Member’s Bill, represents a significant legislative attempt to codify the concept. It applies to developers and deployers of AI in sectors like law enforcement, financial credit, and employment, and mandates that AI systems “not discriminate only based on race, religion, gender, or any of them”.

While concise, this proposed statutory definition is deeply significant for regulatory research because it deconstructs the concept into two distinct measurable elements:

• Systematic Error (The Technological Cause): The legislation frames not merely as a random anomaly or a one-off glitch but as a recurring flaw within the AI models. The bill links these systematic errors directly to the development phase, acknowledging that they often stem from the data sources and methodologies used for training algorithms. To combat the root cause of these errors, the proposed law explicitly mandates that developers ensure diversity and inclusivity in training datasets.
• Unfair Outcomes (The Real World Impact): The definition explicitly connects technical flaws to tangible societal harm. Within the broader context of the bill, the threshold for an unfair outcome is mostly clearly illustrated in section 5, which says that AI systems involved in critical decision-making sectors, especially law enforcement, financial credit and employment, must not discriminate based on race, religion, gender or any of them.

Section 6(b) places the onus on the developers to ensure prevention of algorithmic bias by conducting regular audits, ensuring diversity and inclusivity in training datasets, and withdrawing systems exhibiting significant bias.

Further, Section 7 establishes a grievance redressal mechanism for affected individuals, with Section 8 imposing penalties for violation of the provisions of the Bill.

NITI Aayog - National Strategy for Artificial Intelligence (2018)

This is India's foundational AI policy document. Although it does not formally define “algorithmic bias,” it recognizes the issue in India. The document identifies fairness and bias as key concerns in AI deployment, highlighting that AI Systems trained on historical data may reproduce and reinforce existing social inequalities. It challenges the assumption that data-driven systems are inherently neutral, highlighting that datasets can reflect past discrimination and lead to biased outcomes. It frames algorithmic bias as a systemic issue arising from data and design and calls for the identification and mitigation of such bias within a broader Responsible AI framework.

NITI Aayog - Responsible AI for All, Part I: Principles for Responsible AI (2021)

In its report “Towards Responsible AI for All,” NITI Aayog addresses the risk of AI systems producing outcomes that are biased against certain individuals or groups on grounds of gender, caste, religion, or socio-economic status. The document describes how bias can arise at different stages of the AI lifecycle, including data collection, model design, parameter selection, and deployment. It also connects these concerns to the Constitution of India, particularly the principles of equality and non-discrimination. By doing so, it treats algorithmic bias not just as a technical issue, but as one that can affect fundamental rights and requires careful governance.

NITI Aayog - Responsible AI for All, Part II: Operationalizing Principles for Responsible AI (2021)

This is the companion document to Part I. It builds on the earlier policy by explaining how algorithmic bias arises in practice. It notes that AI systems are designed by humans and trained on real-world data, which means that existing social biases can enter these systems and become amplified through large-scale automated decision-making. The risk is further intensified by the “black box” nature of many AI models, which makes biased outcomes difficult to detect, explain, or challenge, thereby reinforcing unfair or discriminatory results.

The Ministry of Electronics and Information Technology’s India AI Governance Guidelines: Enabling Safe and Trusted AI Innovation (2025)

These guidelines recognize that while AI offers major benefits, it also comes with risks such as deepfakes, misinformation, and algorithmic bias. To address this, the guidelines aim to strike a balance between innovation and accountability. They require measures that actively prevent algorithmic bias by creating risk assessment frameworks suited to India and considering vulnerable groups. They also ask for a system that would report AI-related harms and track the risks arising from it.  The guidelines encourage the use of voluntary standards, audits, and commitments to reduce risks. Finally, they require human oversight and safeguards, especially in sensitive sectors, to prevent loss of control and ensure responsible use. The document is about making sure AI in India is not only advanced, but also transparent, fair, non-discriminatory, explainable, and secure.

Reserve Bank of India (RBI). (2025). Framework for Responsible and Ethical Enablement of Artificial Intelligence (FREE-AI) Committee Report.

Even though this report primarily deals with the financial sector, it also lays down principles that are relevant to AI governance across several domains. It lays seven guiding sutras, out of which one of the principles is fairness and equity. This principle focuses on the fact that AI systems must be designed, tested and deployed in a manner that ensures non-discriminatory outcomes, especially for marginalized groups. The report tells us about how AI should be regulated across different sectors in India, like healthcare, education, and governance.

The Comptroller and Auditor General of India’s Artificial Intelligence Strategy Framework (2025)

This document highlights the need for checking AI systems for bias through proper audits. It considers algorithmic fairness as a key principle, showcasing that AI should not produce unfair and discriminatory outcomes. The framework provides guidance, such as audit goals and a checklist to check whether the AI system has been correctly built. It has safeguards to reduce bias such as algorithmic fairness assessments during audit execution. These safeguards also include training data controls to avoid bias and checking for spurious correlations and unintended outcomes. It stresses that AI systems should be safe, secure, and fair.

Expert Committee on Non-Personal Data Governance Framework (Kris Gopalakrishnan Committee) - Report, 2nd Edition (December 2020)

The Report of the Committee of Experts on Non-Personal Data Governance Framework (2020), chaired by Kris Gopalakrishnan and constituted by the Ministry of Electronics and Information Technology, is not primarily focused on algorithmic bias but highlights an important related risk. The report notes that even non-personal and anonymized data can generate collective harms, including discriminatory impacts on communities. It explains that insights derived from aggregated datasets may be used in ways that enable group-level exclusion or unequal treatment, even without identifying individuals directly. While the report does not explicitly define algorithmic bias or propose a dedicated AI accountability framework, it underscores the need for governance mechanisms to address harms arising from large-scale data processing.

‘ALGORITHMIC BIAS’ AS DEFINED IN CASE LAWS

The Indian judicial system and the international courts have not made any legal definition that defines algorithm bias as a distinct legal term. But, its constituent elements that amount to discriminatory automated decision-making, a lack of transparency in algorithmic systems, and the right to challenge algorithmically generated results have increasingly been dealt with through a growing body of domestic and international case law, which has gradually defined its legal contours. Indian case laws are important in determining the constitutional norms that can be used to compare the algorithmic systems. The most direct court interpretation of algorithmic bias as a legal issue is found in international legal decisions.

Even within the Indian constitutional framework, algorithmic bias is viewed with high scrutiny in the doctrine laid down by E.P. Royappa v. State of Tamil Nadu (1974) in which the Supreme Court decided that, “arbitrariness is the very antithesis of equality.” The court further broadened the scope of Article 14 to bar any arbitrary state action, irrespective of whether it is expressly discriminatory in the grounds listed. More importantly, this standard does not involve the need to prove intentional discrimination, and it is only necessary to show that the output of the algorithm has no rational, principled basis or brings about systematically irrational outcomes to a protected group. Any AI application used by a state that gives biased output through biased training data or design may be challenged as an arbitrary classification and thus breaches Article 14, since no evidence of intentional discrimination is required.

In Justice K.S. Puttaswamy (Retd.) v. Union of India (2017) in which a nine-judge bench agreed unanimously that the right to privacy is a fundamental right in Article 21 of the Constitution, strengthening the constitutional foundation for algorithmic accountability. This ruling was based on the case challenging the Aadhaar biometric identification programme, and it was proactive in the sense that it acknowledged that technology, such as automated data processing and profiling, posed new dangers to privacy. The Court observed that profiling may result in discrimination based on factors such as religion, ethnicity or caste. The Court introduced a three-part test, which is the constitutional standard of all state actions involving automated processing of personal data: legality, legitimate aim, and proportionality. It is this proportionality framework that has become the main constitutional point of reference against which the algorithmic decision-making in India is now measured. Post-Puttaswamy, an AI tool implemented in the state that generates oppositely discriminative results without justified reasons can be challenged as a violation of Article 21.

In Maneka Gandhi v. Union of India (1978), the requirement of procedural fairness in automated processes was considered, in which the Court ordered that any process that was likely to interfere with personal liberty had to be right, just and fair. The principle has been used in automated processes that have impacted fundamental rights in the context of academic and policy discourse.  The presence of an algorithmic procedure that renders a decision impacting individual liberty, like a bail risk score, a welfare eligibility notice, or a predictive policing result, but cannot be described or challenged, may itself fail the Maneka Gandhi test of procedural fairness, even when it is enforced within a formally legitimate legal structure.

Courts all over the world have started to consider algorithmic bias as a material harm that can be legally sued.  One of the US district courts, Mobley v. Workday, Inc. acknowledged that AI-based hiring tools in any case of decision-making are prone to create a liability under anti-discrimination statute, and therefore, the outcomes of an algorithmic mediation should be treated as acts as opposed to the neutral technical output. In the same way, Equal Employment Opportunity Commission v. iTutorGroup (2023) demonstrated that discriminatory criteria may be explicitly integrated into algorithmic systems, with an automated hiring system being programmed to block out those who are older than a specific age, thus leading to unlawful elimination of a protected group, which would violate the Age Discrimination in Employment Act.

The case of Mary Louis v. SafeRent Solutions (US Federal Court, 2024) explains how an algorithmic bias may emerge even when the data it operates on appears to be neutral. Here, a tenant screening algorithm came up with a set of safety scores called SafeRent Scores that discriminated against Black and Hispanic applicants, especially those who benefited from housing vouchers. The bias was due to the fact that the system was based on credit history data, which is influenced by structural inequalities that existed before, and it did not consider mitigating factors like the significant rental support through public housing vouchers. The acknowledgement by the court that such a system might be considered subject to the Fair Housing Act highlights one important aspect of algorithmic bias: that even facially neutral automated systems can be used to create discriminatory results when trained on biased data, have no contextual sensitivity, and lack any real mechanisms to be reviewed by human beings.

In the European context, R (on the application of Bridges) v. Chief Constable of South Wales Police addressed the use of automated facial recognition (AFR) software by police. In this case, it was held that the police had not considered whether the facial recognition technology was unfair towards individuals based on their race or gender. The case established the requirement of a positive pre-deployment bias verification duty for public authorities using algorithmic surveillance technology. The requirement was based on the Public Sector Equality Duty and Article 8 of the European Convention on Human Rights. Furthermore, in NJCM et al. v. The State of the Netherlands (SyRI Case), decided in 2020 by the District Court of The Hague, it has been held that large-scale data analytics in welfare systems can inadvertently create links based on bias, especially in relation to certain groups of people. The Court held that the system was unlawful under Article 8 of the European Convention on Human Rights because of a lack of transparency in relation to data collection and processing, data aggregation, and a lack of verifiable safeguards that made it impossible to evaluate whether or not it had been used in a non-discriminatory manner. This illustrates that opaque decision-making processes are not only a technical issue but a legal one because if people are not able to evaluate decision-making processes, it can lead to bias.

No Indian court has directly ruled on an algorithmic bias claim yet. However, the domestic and international case law discussed above shows the legal standards and principles, such as constitutional equality, privacy, procedural fairness, and positive obligations to verify non-discrimination, that together make up the framework in which algorithmic bias is being defined and challenged more and more.

INTERNATIONAL EXPERIENCE

European Union - EU AI Act

As addressed in Section 2.2.5 above, the EU AI Act addresses bias through a combination of high-risk system obligations, prohibited practices, and data governance requirements. From a comparative perspective, the EU model is notable for a risk-based architecture that places heavy obligations on systems with the highest potential for fundamental rights violations. Article 5(1)(c)'s categorical prohibition of social scoring, which is the most extreme form of institutionalized algorithmic bias. The explicit purpose in Article 10(5) permitting processing of sensitive data for bias correction is a data protection exception created specifically to enable algorithmic auditing.

United States - Colorado SB 21-169/ Colorado AI Act

Colorado's Senate Bill, signed into law in 2024 and effective February 2026, is the first US state law to formally define ‘algorithmic discrimination.’

Section 6-1-1701 (1)(a) defines it as any use of artificial intelligence systems that result in people being treated unfairly on the basis of protected characteristics such as age, race, sex, disability, religion, etc.

The law requires developers and deployers of high-risk AI systems to implement risk management frameworks to prevent algorithmic discrimination, conduct annual impact assessments, and provide transparency notices to consumers. By placing duties on both creators and users of AI, the model spreads accountability across the supply chain.

Brazil - AI Bill No. 2338/2023 (translated to English)

The Brazilian AI Bill, approved by the Federal Senate in December 2024 and currently before the Chamber of Deputies, contains some of the most detailed provisions on discrimination and bias of any national AI legislation outside the EU. It distinguishes between direct discrimination, which is explicitly treating someone unfavorably because of personal characteristics, and indirect discrimination, which occurs when a seemingly neutral rule or practice ends up placing a specific group at a disadvantage. This distinction is crucial because many forms of algorithmic bias are indirect, where they arise from training data or design choices that produce biased results without any overt intent.

Furthermore, it requires algorithmic impact assessments to include evaluation of risks to fundamental rights, including discrimination risks, as part of the mandatory pre-deployment review for high-risk systems.

By linking classical equality law and AI-specific regulation, Brazil offers a model for other jurisdictions that seek to bridge the gap between anti-discrimination frameworks and new technology.

New Zealand - Glossary of AI Terms (Digital.govt.nz)

New Zealand's digital government agency (digital.govt.nz) maintains an official Glossary of AI Terms as part of its Responsible AI Guidance for the Public Service. It explains that bias in AI models typically comes from 2 sources: design of the models (i.e. the assumptions of the developers that build the code or the algorithm) and the training data used.

Though not legally binding, it gives public administrators a practical understanding of bias, helping them implement fair AI practices in government services.

Comparative Analysis

A comparative analysis indicates that international approaches to algorithmic bias are more explicit and structured than the current Indian framework. The European Union’s AI Act adopts a risk-based regulatory model, imposing binding obligations on high-risk systems, while jurisdictions such as the United States (Colorado AI Act) and Brazil’s AI Bill go further by explicitly defining ‘algorithmic discrimination.’ In contrast, India does not yet have a dedicated statutory framework addressing algorithmic bias. While the Artificial Intelligence (Ethics and Accountability) Bill, 2025, exists, it has not yet come into effect. Currently, these concepts are instead inferred through broader principles of data protection, information technology and constitutional guarantees such as equality under Article 14. This results in a more fragmented and indirect approach to regulating algorithmic harms.

RESEARCH THAT ENGAGES WITH ‘ALGORITHMIC BIAS’

There has been a significant increase in the number of studies conducted on algorithmic bias by academic institutions, policy organizations and independent researchers. These studies have focused to see how algorithmic system reproduces structural inequality in terms of empirical evidence. They provide with both regulatory and technical solutions to address the issue of algorithmic bias in multiple sectors such as criminal justice.

ProPublica investigation on COMPAS (Angwin el al., ProPublica, 2016)

In 2016, ProPublica published a significant investigation by Julia Angwin and her reporting team into the COMPAS risk assessment algorithm utilized throughout the U.S. criminal justice system. Their research compared data sets of classified recidivism risk scores to actual recidivism rates among different racial groups. They concluded that a disproportionate number of black crime defendants were assigned high-risk levels in relation to crime defendants who were similarly located but assigned a low-risk classification at a higher rate than white crime defendants. Their research demonstrated that there are examples of algorithmic systems that generate disparate outcomes based on race without a stated intention, and that lack of visibility into proprietary algorithms is one of the critical impediments to holding them (and the organizations that implement them) accountable.

Amazon Recruitment Algorithm (Reuters,2018)

In the report by Reuters in 2018, the Amazon recruitment tool was investigated. The tool used an AI-based approach to filter through the resumes. The AI was trained on the data collected over the past ten years. The report found that the AI tool was biased towards discriminating against resumes with gender-related keywords, such as women’s groups or educational institutions. The above-mentioned report serves as an example of how inequality can arise in an algorithm.

Bail Prediction and Risk Assessment Studies(2024)

Current academic literature on AI-based bail prediction systems, as analyzed in recent academic research published in 2024, reveals that these systems have the tendency to reproduce the existing disparities in the past judicial decisions. This includes the differences in the bail decisions of socio-economic and demographic groups. The literature reveals that the bias in the algorithm is not due to the algorithm’s design but due to the practices followed in the institutions.

Welfare Algorithm Bias- CNAF Litigation (France, 2024)

Recent studies, based on litigation cases against the Caisse Nationale des Allocations Familiales (CNAF) in France (2024), analyze the efficacy and implications of algorithmic systems for detecting fraudulent activities in welfare administration. The algorithmic system provides risk scores to welfare recipients based on various socio-economic parameters such as income levels, employment status, and benefit utilization. The results show that the algorithmic system over-surveilled already disadvantaged sections of society, including women, disabled persons, and single parents.

Biometric Bias in Facial Recognition (Manjang v Uber Eats, UK, 2023-2024)

In the case of Manjang v. Uber Eats, the problem of bias was also present. The case was based on the application of facial recognition technology for identity verification, which was found to perform badly for people with darker skin. As a result, they were denied access to work opportunities. The problem of biometric bias was also present, where the accuracy of the biometric data was not the same for all, thus causing direct socio-economic effects.

Regulatory Gap Analysis – Vidhi Centre for Legal Policy (Chatterjee & Ravindran, Indian Express, 2019)

Sohini Chatterjee and Sunetra Ravindran, in an article published in The Indian Express in 2019, discuss the limitations of existing legal frameworks in dealing with algorithmic decision-making. The article is written from a policy and legal perspective and discusses constitutional and administrative law structures. The article concludes that existing legal frameworks are incapable of dealing with decision-making entities that are not human and highlights the lack of a comprehensive legal regime on algorithmic bias. The authors suggest that there is a need to evolve a special legal and institutional framework to ensure accountability in algorithmic decision-making.

State Use of Algorithms – NALSAR University of Law (Tripathi, Tech Law Forum, 2020)

In his article written for the Tech Law Forum at NALSAR in 2020, Harsh Tripathi discusses the use of algorithm-based systems. In this article, the author uses a legal analytical approach to discuss the issues of opaqueness, lack of accountability, and the problem of discrimination. The author argues that algorithm-based systems used by the government may reflect existing biases, especially when used by the government in a diverse society like India.

Algorithmic Bias in Employment – UC Law SF Business Law Journal (Dailey, 2025)

In an article published in the UC Law SF Business Law Journal, Jordan Dailey, in an article published in 2025, explores the concept of algorithmic bias in the context of employment, particularly in the hiring process. The research, through its doctrinal and case analysis, has highlighted the impact of biased data on the hiring process. The research has proposed several mitigation strategies to overcome the problem of algorithmic bias in the corporate world.

Gendered Algorithmic Bias in India – Indian Journal of Law and Technology (Chandak, 2024)

In her article published in the Indian Journal of Law and Technology in 2024, Sejal Chandak explores the phenomenon of algorithmic discrimination in employment on the basis of gender in India. The study has adopted an interdisciplinary approach and analyzed the subject both from the legal and technological perspectives. The study reveals the role of algorithmic systems in perpetuating gender discrimination and the lack of an all-encompassing anti-discrimination framework in India.

Synthesis and Gaps in Research

The existing literature suggests that algorithmic bias is a socio-technical issue that occurs during the entire lifecycle of artificial intelligence systems. Although international literature shows robust empirical findings on algorithmic bias across different domains, Indian literature suggests that algorithmic bias needs to be contextualised with socio-economic and constitutional paradigms. However, there are many gaps, especially in terms of the lack of India-based empirical literature and the absence of standardized methodology for the detection of algorithmic bias.

CHALLENGES

As detailed in Section 4, algorithmic bias presents a complex web of conceptual, technological and institutional obstacles. These challenges span the lifecycle of algorithmic systems, where bias is present from the stage of data collection to the outputs generated.

India currently lacks a binding statutory definition of algorithmic bias. While the Artificial Intelligence Bill gives a definition, there is a lack of a universally accepted legal standard creating regulatory uncertainty and hindering enforcement. Existing legal instruments, such as the DPDP Act or the IT Act, do not mandate algorithmic fairness or impact assessments. While they introduce data governance obligations like accuracy and transparency, they do not impose safeguards against discriminatory algorithmic outcomes. This creates a regulatory gap where the inputs to these systems are governed, but the outputs remain largely unregulated.

Further, the ‘thinking’ undertaken by these algorithms to produce the outputs are not made available to the public, making the process opaque and creating a situation of ‘Black-box’ algorithms. Daksh Report on “Algorithmic Accountability  in the Judiciary” notes that these opaque tools can violate procedural due process. Moreover, regulators and courts currently lack the technical expertise to audit algorithmic systems or adjudicate bias claims.

WAY AHEAD

To address the algorithmic bias effectively India needs a clear and forward looking approach that aims at transparency and accountability.

There should be an enactment of a clear definition of algorithmic bias, building on the 2025 bill and aligning with the international standard mentioned above. Further, a precise risk based analysis is needed, as to what are the risks of algorithmic bias, the ways in which it can be mitigated and the remedies that affected people can have. The high risk algorithms should disclose the sources of the data, the logic behind the model and the results of the audit. This would allow the Indian regulatory framework to be aligned with global standards such as the EU AI Act and GDPR Regulations.

The judges, regulators and the public officials should be trained in AI governance. There should be an investment in interdisciplinary research on bias in the Indian contexts. The state should ensure that individuals have the right to contest automated decisions that affect their fundamental rights before a human reviewer.  Those affected people can seek for the compensation for algorithmic bias.

The companies that adopt a transparent and auditable practices should be acknowledged in public, given tax incentives and provided certificates, so that it can motivate other companies as well to be transparent in their practice. India has an opportunity to come at par with the developed countries by enacting a legislation that comprehensively governs Artificial Intelligence.

---

Contributors & review status

Curated / edited bynot yet recorded

Peer reviewpending

Project reviewpending

Expert reviewpending

Page created 2026-08-21  ·  Last updated 2026-08-23