2.4.1 Non-discrimination and the Prevention of Bias

The potential for algorithmic bias or discrimination is a well-documented issue. Much public discussion in this area has focused on gender, race and socio-economic inequality³⁰⁴. However, disability, including mental health and psychosocial disability, ‘has been largely omitted from the AI-bias conversation’.³⁰⁵ Whitaker and colleagues have argued that ‘patterns of marginalization [concerning disability] are imprinted in the data that shapes AI systems, and embed these histories in the logics of AI’.³⁰⁶ For example, Ben Hutchinson and colleagues at Google, demonstrated that social attitudes casting disability as bad and even violent – particularly in regard to mental health – were encoded in AI systems designed to detect hate speech and identify negative/positive sentiment in written text.³⁰⁷ The ‘machine-learned model to moderate conversations’, according to Hutchinson and colleagues, classifies texts which mention disability and particularly references to mental health as more ‘toxic’, while ‘a machine-learned sentiment analysis model rates texts which mention disability as more negative’.³⁰⁸ Such studies highlight how biased datasets create biased algorithms, which can have major consequences for people’s lives, as the next example shows.

This case study is revealing. Mr Behm was seemingly harmed due to data to which he was never given access. Nor does it appear that Mr Behm had an easily accessible opportunity to contest, explain or investigate the test outcome. Cathy O’Neil argues that this type of algorithmic ‘red-lighting’ has the potential to ‘create an underclass of people who will find themselves increasingly and inexplicably shut out from normal life’.³¹¹

One response to biased algorithmic systems has been to focus on creating un-biased datasets. Datasets could be made more diverse, the argument goes, to capture diverse human experiences. This would avoid negative consequences for people who, through the various human and circumstantial complexities in their lives, are considered ‘statistical outliers’ for whom algorithmic decision systems are ill-equipped. This approach is certainly warranted in some circumstances, where the need for good quality and representative data can help avoid biased or discriminatory outcomes.

However, the aim of creating unbiased datasets will be insufficient in many situations. Meredith Broussard criticises this approach as being commonly ‘technochauvinist’ in nature.³¹² Techno-chauvinism refers here to the false view that technological solutions provide ‘appropriate and adequate fixes to the deeply human problem of bias and discrimination’.³¹³

Representative and high-quality datasets will be important in some instances but Broussard’s criticism suggests that there is a second major category of discrimination at play: namely, discrimination perpetuated by human systems and institutions using data concerning mental health. Examples might include insurance companies discriminating against people based on data showing that they accessed mental health services at one time,³¹⁴ or police and border authorities discriminating against people based on non-criminal data concerning an individual’s engagement with mental health services, as the next example shows.

This example suggests that no amount of ‘unbiased datasets’ will offset the discriminatory premise of various digital initiatives, which are designed to intervene in the lives of persons with lived experience and disability on an unequal basis with others.

Discriminatory impacts are also more likely as algorithmic and data-driven technologies are applied in settings affecting marginalised populations. This includes settings in which there are broader constraints on individuals’ agency, including cumulative effects of disadvantage. This could include ethnic and racial minorities, people facing involuntary psychiatric intervention, families or individuals facing housing insecurity, returning veterans, people with addiction, the previously or presently incarcerated, and migrants and asylum-seekers.³¹⁸ LLana James points to these concerns when she asks: ‘How will data labelled as Black, poor, disabled or all three impact a person’s insurance rates?³¹⁹ James argues that current laws (writing in Canada) do not appear to protect health service recipients and patients and calls for updated data laws to protect against ‘the perils of monetized data and the discriminatory algorithms they are generating’.³²⁰

Regarding insurance, Access Now have expanded on James’ point regarding exclusion and insurance-based discrimination, noting that:

This point is concerning in the mental health context, and the emergence in recent years of partnerships between insurance companies and mental health technology companies³²² and other insurance company initiatives concerning mental health-related data warrant scrutiny.³²³

The likelihood of disability-based discrimination will be compounded when data scientists, technologists, tech entrepreneurs, clinical innovators, and so on, are not aware of the potential for discrimination using data concerning mental health. Consider the following claims being made in the ‘emotion recognition’ industry in China.

AI Now Institute, an interdisciplinary research centre examining artificial intelligence and society, have called for a ban on technology designed to recognise people’s emotions concerning ‘important decisions that impact people’s lives and access to opportunities’.³²⁶ This scope would surely extend to automated forms of diagnosis or proxy-diagnosis of cognitive impairments or mental health conditions. The proposed ban could apply to decisions concerning hiring, workplace assessment, insurance pricing, pain assessments or education performance. AI Now base their recommendation on two concerns: 1) the often-baseless scientific foundations of emotion recognition, and 2) the potential for bias and discrimination in the resulting decisions.³²⁷

Emotion or ‘affect’ recognition technology such as facial recognition technology raise important issues for this report. At least three points relating to non-discrimination and the mental health context are worth noting:

There are important differences between the motives and claims of commercial actors who are promoting affect or emotion recognition technology, and that of biometric monitoring in clinical studies conducted by mental health researchers—and there are major differences in the regulatory frameworks affecting each. In general, health research is far more tightly regulated, with more entrenched infrastructure for upholding ethical research involving humans. Although not without serious problems, including the interference of private industry with academic research,330³³⁰ and the growing reliance of universities on the private sector to fund research,331³³¹ the scholarly health research infrastructure appears better developed for the purposes of ethical oversight, than many private sector uses of affect recognition technologies, including where those technologies are sold to government agencies, such as police agencies and border authorities.

Nevertheless, there are many examples of overlap between commercial and clinical activities in the digital mental health context,³³² and public scrutiny is required of clinical or scholarly claims about what behaviour can convey about a person’s inner-world. It is not possible in this report to examine the claims being made about psychiatric biometric monitoring. Instead, the aim in this section is to draw attention to poorly understood potential for discrimination and bias in the use of such technologies in the mental health and disability context.

Concerns about discrimination need not relate to algorithmic systems. For example, a coalition of Australian organisations representing people with lived experience and psychosocial disability called for a suspension of a national electronic health records scheme, citing fears of discrimination if personal mental health histories were stolen, leaked or sold.³³³ Previous case studies cited throughout the report demonstrate how such discrimination might occur.

One important step to preventing harms caused by data concerning mental health and disability – whether leaked, stolen or traded – is to strengthen non-discrimination rules concerning mental health and psychosocial disability.³³⁴ National discrimination laws may require amendments to ensure that discrimination on mental health grounds by online businesses is covered.³³⁵ Such amendments would be consistent with the goals and legislative history of anti-discrimination laws and would remove ambiguity regarding the status of websites, social media platforms and other online businesses.³³⁶ Remedies for individuals who are aggrieved by discriminatory behaviour and practices are also likely to require strengthening, including ensuring substantive, verifiable, auditable standards of non-discrimination in the use of algorithmic and data-driven technologies