1.3.1 AI-based Suicide Alerts and Self-harm Surveillance

Government agencies, social media companies, not-for-profits, health services, and others have begun using machine learning and artificial intelligence in suicide prevention, including in efforts to pre-emptively identify people who may self-harm.52 In some cases, these technologies appear to have been used to activate police powers to detain people for the purposes of involuntary psychiatric intervention.

Facebook/Meta’s algorithmic responses also encourage peer-responses from among the person’s user-network by drawing their attention to the person’s apparent distress.55 These measures were developed after some form of consultation with suicide attempt survivors and experts on suicide prevention (though few details are available).56 Facebook/Meta provides some information about the algorithmic process behind the interventions,57 and has described the ethical issues with which programmers grappled.58

However, there remains little information about what precisely is meant by a ‘wellness check’ (including whether location data are shared with first-responders). Nor is there publicly available research as to the accuracy, scale or effectiveness of the initiative. What Facebook/Meta does with the information following each apparent crisis is also unclear.

Police appear to be the first-responders undertaking ‘wellness checks’. Facebook/Meta has therefore drawn criticism for failing to grapple with the reality of anti-Black racism in the US and the prevalence of police violence in their encounters with distressed individuals, particularly Black, Indigenous, people of colour. For example, Joshua Skorburg and Phoebe Friesen write:

The US is by no means alone on such patterns of police violence.⁶⁰

From a legal and regulatory perspective, suicide prediction in medical systems is governed by health information laws, medical practice and clinical governance regimes, and research regulations that require transparency and peer review. Flawed as these frameworks may be, AI-based suicide prediction on social media platforms, as Mason Marks points out, ‘typically occurs outside the healthcare system where it is almost completely unregulated, and corporations often maintain their prediction methods as proprietary trade secrets’.⁶¹ To remedy this, Marks recommends several steps to improve people’s safety, privacy and autonomy, including:⁶²

The use of individual and population monitoring in efforts to prevent suicide or efforts to promote its use, are likely to increase in coming years. In December 2020, the US National Suicide Prevention Lifeline administrator recommended that the US Government authorise geo-location systems to pin-point the exact location of all callers by 2022.⁶³ Leah Harris has criticised this recommendation, warning that ‘Mad and disabled advocates who have experienced mental health crisis intervention, and even some crisis service providers, worry that geolocation would serve to further entrench coercion in mental health and crisis response systems, replicating problematic aspects of [the US emergency services line] 911’.⁶⁴ The impact of automated surveillance of callers on rates of involuntary psychiatric interventions, police involvement in crises, citizens’ willingness to report to such services, and so on, remains unknown.