Is AI Ethical?

The Prohibition of Discrimination and AI

The prohibition of discrimination is a fundamental right that enjoys constitutional protection. A series of international instruments (e.g., the Charter of Fundamental Rights of the European Union) declares the right to non-discrimination, and domestic law treats it with particular prominence: the general provision of Article XV of the Fundamental Law is given concrete form by Act CXXV of 2003 on Equal Treatment, while the subject has an extensive literature and numerous court decisions have been handed down on the topic of discrimination. It is therefore well established what, in ordinary circumstances, conflicts with the requirement of equal treatment and the prohibition of discrimination. However, the spread of artificial intelligence has created new conditions in the fight against discrimination. The two examples below illustrate why it is problematic if artificial intelligence is not properly prepared to uphold the equal treatment that we expect from decision-makers in all cases.

The COMPAS case centres on the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) risk assessment system, used by courts in the United States to calculate a recidivism probability score for offenders through predictive analytics, with a view to assessing the risk of future reoffending. The system is thus intended to assist courts with concrete decision-making recommendations. The COMPAS system itself can generate three predictive scores for the individual concerned: the risk of pretrial release; and the probability of general and violent recidivism. To train this software, the results of questionnaires completed by persons in pre-trial detention and data from the criminal record database were used as a basis. Although this AI solution does not explicitly include data referring to racial factors, it emerged during practical use that its internal decision-making mechanism assigns significant weight to the ethnic background of the defendant. Once it became clear that COMPAS, based on its training data, was more likely to flag Black defendants as high-risk than white defendants, the system’s use came under widespread criticism and challenge. As a result, the version challenged for discrimination was banned. However, a partially modified version of the software continues to be used in the American criminal justice system (safeguard elements have been built into the algorithm to better diversify results, and a significant advance is that the current software no longer assigns “decisive” significance to the offender’s skin colour).

In the Amazon case, the discriminatory problems arising from Amazon’s AI-based recruitment software stemmed from the system having been trained on ten years of CV data in which successful male career trajectories were overrepresented. Consequently, the software favoured male candidates and disadvantaged women by assigning them lower scores. Amazon ultimately discontinued use of the software, as it was unable to guarantee freedom from gender-based bias.

The cases above also highlight the problems of algorithmic bias — specifically, that incorrect steps can occur at various stages of the algorithm’s development (and afterwards) that ultimately lead to a “biased algorithm.” These cases have all triggered wide-ranging debate about the transparency and fairness of the operational logic of algorithms within the legal system. It is clear that the situation must be addressed, since problems of the kind described above are unacceptable if the deployment of AI systems becomes widespread — these systems may, in certain cases, operate irrespective of geographical boundaries, making it impossible to foresee the range of people affected by the software. Furthermore, since retrospective legal remedies do not, in given cases, address mass-scale harm, there is a need for specifically in-process technical, organizational, and logical controls.

Legal and Other Solutions

The problem is approached from a legal perspective by the European Union’s Artificial Intelligence Regulation (AI Act), which is the world’s first comprehensive AI regulation. In addition to reaffirming the prohibition of discrimination at the level of fundamental principles, the AI Act also establishes a specific new procedure — the fundamental rights impact assessment — which must be carried out during the IT development phase, i.e., in a preventive manner. This is intended, insofar as possible, to prevent AI tools that infringe any fundamental right (including the prohibition of discrimination) from entering active use.

It is obvious, however, that there are also solutions beyond legal instruments that prudent AI operators are obliged to employ. The examples above make clear that the potential bias of AI software can easily be caused by the training data. It is difficult to foresee this situation in advance, since the volume of training data does not allow for adequate prior human review. Examining the balance of the dataset used for training is possible using statistical methods, but this may also mean artificially modifying the content of records generally considered to be of public authority.

It is likewise obvious that developers do not intentionally train software with discriminatory data; nevertheless, the “black-box” problem inherent in algorithms is present, meaning that it is simply not always foreseeable what conclusions the software will draw. Furthermore, training data carries historical, geographical, economic, political, and religious aspects that may influence the AI system’s conclusions. On this basis, the prudent AI operator is obliged to take the following steps:

• Data examination and cleansing: Datasets must be carefully examined to the extent possible and cleansed of historical prejudices and biases. This is one of the most difficult steps.
• Built-in controls: During the development of AI systems, it is necessary to incorporate control mechanisms that prevent the software solution from making certain types of decisions under any circumstances.
• Transparency and explainability: The decisions of AI systems must be transparent and explainable. Clients must be given the opportunity to understand why and on what basis the solution reached its decision, though the AI system’s operator is not required to disclose trade secrets in the disclosure.
• Human oversight: This is a control mechanism proportionate to operational risk. The decisions of AI systems must be placed under oversight commensurate with the system’s risk classification, and humans must have the ability to override the AI system’s decisions when necessary.
• Ethical guidelines and training: Operators must develop ethical guidelines and provide regular training for employees on the use of AI systems and on ethical considerations.

The regulation and practical application of AI-based systems are going through an extremely turbulent period. Alongside the AI Act, the growing number of software applications and AI solutions deployed will presumably provide sufficient experience to determine whether the current regulation is adequate and whether operators of AI software will be able to ensure the non-discriminatory operation of these systems.

The next instalment of this article series examines liability questions arising from the use of AI systems.

Reviewed by: dr. Laura Bikki Kovácsné, dr. Bernadett Bocsi, and dr. András Bencsik

Related articles:

The AI Act and the Financial Sector

What Counts as AI?

---

[1] https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm