1.3 Digitising Involuntary Psychiatric Intervention and Other Coercive Measures

Unlike data concerning physical health, data concerning mental health can be used to initiate state-authorised coercive interventions in certain cases. This possibility adds important legal, social and political dynamics to this discussion. Although the majority of people who access some kind of service for mental health reasons will access those services voluntarily, a small but significant minority of people will be subject to involuntary psychiatric intervention – typically involving the person being detained in hospital and treated against her/his/their wishes – under existing mental health-related legislation.

1.3.1 AI-based Suicide Alerts and Self-harm Surveillance

Government agencies, social media companies, not-for-profits, health services, and others have begun using machine learning and artificial intelligence in suicide prevention, including in efforts to pre-emptively identify people who may self-harm.52 In some cases, these technologies appear to have been used to activate police powers to detain people for the purposes of involuntary psychiatric intervention.

Facebook/Meta’s algorithmic responses also encourage peer-responses from among the person’s user-network by drawing their attention to the person’s apparent distress.55 These measures were developed after some form of consultation with suicide attempt survivors and experts on suicide prevention (though few details are available).56 Facebook/Meta provides some information about the algorithmic process behind the interventions,57 and has described the ethical issues with which programmers grappled.58

However, there remains little information about what precisely is meant by a ‘wellness check’ (including whether location data are shared with first-responders). Nor is there publicly available research as to the accuracy, scale or effectiveness of the initiative. What Facebook/Meta does with the information following each apparent crisis is also unclear.

Police appear to be the first-responders undertaking ‘wellness checks’. Facebook/Meta has therefore drawn criticism for failing to grapple with the reality of anti-Black racism in the US and the prevalence of police violence in their encounters with distressed individuals, particularly Black, Indigenous, people of colour. For example, Joshua Skorburg and Phoebe Friesen write:

The US is by no means alone on such patterns of police violence.⁶⁰

From a legal and regulatory perspective, suicide prediction in medical systems is governed by health information laws, medical practice and clinical governance regimes, and research regulations that require transparency and peer review. Flawed as these frameworks may be, AI-based suicide prediction on social media platforms, as Mason Marks points out, ‘typically occurs outside the healthcare system where it is almost completely unregulated, and corporations often maintain their prediction methods as proprietary trade secrets’.⁶¹ To remedy this, Marks recommends several steps to improve people’s safety, privacy and autonomy, including:⁶²

The use of individual and population monitoring in efforts to prevent suicide or efforts to promote its use, are likely to increase in coming years. In December 2020, the US National Suicide Prevention Lifeline administrator recommended that the US Government authorise geo-location systems to pin-point the exact location of all callers by 2022.⁶³ Leah Harris has criticised this recommendation, warning that ‘Mad and disabled advocates who have experienced mental health crisis intervention, and even some crisis service providers, worry that geolocation would serve to further entrench coercion in mental health and crisis response systems, replicating problematic aspects of [the US emergency services line] 911’.⁶⁴ The impact of automated surveillance of callers on rates of involuntary psychiatric interventions, police involvement in crises, citizens’ willingness to report to such services, and so on, remains unknown.

1.3.2 ‘Digitising mental health law

Some governments have sought to digitise processes of involuntary psychiatric intervention.⁶⁵

Proponents suggest the electronic forms and digital platforms will improve access to care, reduce errors, and improve information sharing, which ultimately reduces the distress of the individuals and prevents delays in the provision of healthcare.69 However, some mental health services users have raised concerns about the unknown impact of the digitised process on people subject to orders, which are potentially serious and warrant closer attention.⁷⁰

In recent years, the processing of data about those subject to involuntary psychiatric intervention through electronic records systems has harmed people with lived experience in some cases. Concerns about police agencies sharing data concerning self-harm were raised in Canada, where municipal police collated non-criminal information about individuals who had self-injured or attempted suicide.71 The information was then circulated to US border authorities, who used the information to deny several Canadians entry into the US. (This example will be discussed at page 52). The ease with which people’s sensitive data concerning involuntary treatment can be accessed by various government departments, has raised concerns about the potentially unlawful uses of that data.

The SIM program will be discussed later in the report in sections on accountability and privatisation (page 58). SIM also appeared to have disproportionate impacts along lines of race and class (discussed at page 69)

1.3.3 Power and Coercion in Mental Health

Other areas of law around the world overtly discriminate against people with lived experience and psychosocial disability, which adds to the sensitivity of data concerning mental health. Discrimination in law could include preventing a person with a mental health diagnosis from holding public office, migrating into particular countries, and working in particular professions.⁷⁹ Indeed, some countries continue to criminalise suicide attempts. For example, Section 226 of Kenya’s penal code states that ‘any person who attempts to kill himself [sic] is guilty of a misdemeanour’.80 Around 20 countries still criminalise suicide attempts, according to a 2021 report by the International Association for Suicide Prevention and United for Global Mental Health.81 Automated suicide alert programs must therefore be applied with extreme caution (see section on Non-Discrimination and Equity below)

There are also well-established examples where people with psychosocial disabilities and mental health diagnoses are occasionally subject to political scapegoating and public scare campaigns that attract intrusive and discriminatory proposals for state intervention.

One category of biometric monitoring technology more broadly, known as ‘anomaly detection’, may have repercussions for people with psychosocial disabilities and lived experience if used in public surveillance. According to one report, automated surveillance systems are designed undertake ‘automatic detection and tracking of unusual objects and people’.84 The literature on anomaly detection, according to a report for the ACLU, is ‘full of discussion of algorithms that can detect people or behaviours that are “unusual,” “abnormal,” “deviant,” or “atypical”’.85 (See above the discussion about the politics of terminology in the mental health context, including characterisations of ‘deviance’ and ‘abnormality’, page 13).86 The authors warn that identifying statistical deviance is not a negative thing per se but when it ‘shades into finding deviant people it should raise alarms’,87 and given the historical and current exclusion of, and social hostility against, people with lived experience and psychosocial disability, and other disabilities, such as intellectual and cognitive disabilities, this is particularly troubling

A less experimental form of electronic monitoring at the intersection of mental health and criminal justice is the use of monitoring of people in forensic mental health services using global positioning system (GPS). Electronic monitoring devices record and regularly transmit data on a person’s location via devices fixed to his or her body. Some GPS devices, such as devices affixed to a person’s wrist or ankle, can be linked to blood-alcohol monitors.⁸⁸

More issues concerning involuntary psychiatric interventions and computer technology will emerge in coming years, raising pressing questions. Will monitoring devices be imposed in involuntary psychiatric interventions in the civil context, such as ‘community treatment orders’? Should algorithmic technologies be used at all in coercive crisis responses? How might these concerns relate to broader efforts in recent years to reduce and eliminate coercion in mental health settings, and to debates about ‘abolishing versus reforming’ involuntary psychiatric interventions?⁹⁶

The future of algorithmic and data-driven technologies in coercive state interventions remains uncertain—but imagined futures are guiding activity today. As one industry publication that promoted technology in healthcare stated

This imagined future is one possibility. Others will reject this vision of expanded risk predictions and technology-facilitated coercion, and instead promote the development of open and co-operative crisis support relationships that are enhanced by selective use of digital technology. These contested futures suggest that the power dynamic caused by coercion in mental health services must be a part of the discussion concerning ‘digital mental health’ measures today