Bias and discrimination in algorithms – where do they go wrong?
Bias and discrimination in algorithms – where do they go wrong?
This is the start of a new blog series, which critically addresses the use of algorithms and machine learning in our society, highlighting the issues that emerge, what can be done to address these issues – from a technological and from a societal point of view, as well as periodically informing about the findings from the research project FairAlgos. This opening blog tackles the question of how algorithms can be biased in the first place. What does it mean to talk about algorithmic discrimination?
As big data analytics and machine learning technologies are improving, and the technological infrastructures are becoming cheaper, automated decision making through algorithms and machine learning sees increased use in a wide range of settings. In policing and the judicial system, in commercial settings such as credits or marketing, in smart cities, or in welfare systems – automating technologies are in use in all of them, either assisting decisions, or taking them independently. Once in operation, the technologies are expected to be more accurate in their decision making than the human counterparts they are replacing, while also being faster in their operation and thus more efficient. Certainly, most of these promises hold true in normal situations. And yet, there are many examples where choices and decisions have been made based on the outcome of algorithmic systems, that were fundamentally discriminatory and biased. Journalists have shown that in the US-justice system, algorithms predicted black people a higher risk to re-offend than white people, for similar offenses. In other cases, algorithms were only showing high-paid job online advertisements to white men and not to women or people of colour. Predictive policing algorithms were directing law enforcement agencies to neighbourhoods with a high proportion of people from racial minorities, irrespective of the level of crime. Facial recognition algorithms – in combination with bad police work – led to the arrest of an innocent black man. And the new algorithm that is supposed to classify the employment chances of job seekers in Austria, discriminates – amongst others – against single moms.
These examples do not mean that these algorithms are purposefully built to discriminate against women, people of colour, or minorities. Instead, they are the outcome of different processes that lead to algorithms making decisions that are biased. The researchers Batya Friedman and Helen Nissenbaum[1] have defined three different types of bias that can be included in algorithms: pre-existing bias, technical bias, and emergent bias.
Pre-existing biases are the ones that are prevalent within our societies. They have their roots in social institutions, in practices and attitudes, and can emerge from the society in its entirety or solely reflect the personal biases of individuals responsible for the design of the system, such as the system designers. A pre-existing bias can be included consciously and deliberately. Mostly, they are however included implicitly and unconsciously, sometimes even in the best of intentions. An example of pre-existing bias is the lower score for women regarding their employability in the context of the algorithm of the Austrian public employment service. While this decision might reflect the current state of the labour market, it is nonetheless based on a pre-existing bias and discrimination against women. A bias that is represented in the training data that is used to teach the algorithm – in this case that women, especially with children – are less likely to get a (high-payed) job. The same goes for the criteria that are implemented and designed into the algorithmic decision-making structure. Although it might be an accurate representation of the current state of the labour market, that women are less likely to succeed, as with the training data, this is the outcome of a discriminative labour market. Defining this into the algorithmic structure further reinforces and materializes this bias.
The second type of algorithmic biases are inherent of the technology itself. Technical biases usually emerge within the design process of the algorithms and are the results of limitations of computer tools such as hardware, software, and peripherals. But they also can emerge when system designers attempt to digitalise human constructs, when they are trying to make fundamental human aspects machine readable. In short when they quantify the qualitative. More simple examples of technical bias are the representation of rankings on computer screens, where those who are lower in the ranking will be neglected more quickly, as they simply do not fit on the computer screen right away. This is the case in search engines. More problematic technical biases emerge when face recognition algorithms are used to identify individuals solely based on grainy and unclear CCTV images. In this case, a mismatch is very likely to happen, even more so if the individuals are not the standard white male. But this bias can also emerge in cases when system designers quantify aspects that are not easily quantifiable, such as the potential for love and attractiveness, which are used in the matching algorithms for dating websites.[2]
The third type of algorithmic biases are emergent biases. These do not exist (per se) in the technology straight “out of the box.” Instead, they emerge because of the interaction between the technology and the users. The smarter, the more interactive, the more self-sustained that technology becomes, the more common emergent biases are. A prominent example here is the artificial intelligence chatter bot named Tay, developed by Microsoft and designed to communicate with human users via Twitter. The bot was meant to mimic a teenage girl and came pre-programmed with “teen girl speak”, with the intend to learn ideas and ways of communicating through interacting with human users. Within less than a day, Tay became a neo-Nazi racist and sexist. Microsoft didn’t program Tay to be a white supremacist, but they did program her to learn from human users. And when human users are trolls and act like white supremacists, Tay quickly adapted to mimic her human companions.
All these are just some of the examples which show that algorithms and machine learning are quite often wrong in their decision-making, don’t always behave as expected, learn from the pre-existing biases in our society and basically function well for the standard they are trained on – which is mostly male and white, but tend to be more inaccurate when targeting minorities and non-white male individuals. This is highly problematic, in particular as an international approach towards regulating algorithms and machine learning is not progressing. Instead, there is a strong trend towards the self-regulation of the technology industry – a trend that so far has failed to deliver. In the wake of the increased focus on systemic racism and discrimination in our societies, this lack of accountability and willingness to properly address the problem is not acceptable.
In the midst, we as VICESSE have started to address the problem from an academic/political perspective, mainly through two national research projects. The project AlgoJus addresses the intersection between algorithmic decisions and human decisions. The FairAlgos project specifically looks at the potential of bias and discrimination in machine learning algorithms in imaging and how to implement fairer decision-making processes, in the technology as well as in the practice of using these technologies, in cooperation with the Computer Vision Lab of the Technical University of Vienna. Furthermore, we also support the Austrian campaign against the so called AMS-algorithm – the algorithm that is in use in the Public Employment Service Austria and classifies the employment chances of job seekers. In the course of these projects, a series of blogs will further address the problems which occur in our societies through the use of algorithms, as well as highlight some of the advances that are made to mitigate these problems.
[1] See Friedman, Batya; Nissenbaum, Helen (1996): Bias in computer systems. In ACM Trans. Inf. Syst. 14 (3), pp. 330–347. DOI: 10.1145/230538.230561.
[2] Roscoe, Philip; Chillas, Shiona (2014): The state of affairs. Critical performativity and the online dating industry. In Organization 21 (6), pp. 797–820. DOI: 10.1177/1350508413485497.