Part 2 - Themes for Responsible Public Governance

To draw out the various issues raised by the rise of automation in the mental health context, the remainder of this report is ordered around the following inter-related themes, or ethical and political values:

These themes commonly appear in global discussions about algorithmic technology and data governance.¹⁸⁸ However, we add the following caveats. By ordering our discussion under these themes, we are not suggesting that a statement of ‘principles’ is needed for the mental health and disability context – quite rightly, ‘principles documents are frequently challenged as toothless or unenforceable.’¹⁸⁹ At its worst, looking at ethical themes or values can draw attention away from broader questions of justice, power, finance and politics that drive the recent growth of algorithmic and data-driven technologies, as discussed above. It should be acknowledged that the themes we have tentatively listed risk narrowing the focus of public debate to questions of procedural safeguards – for example, by zeroing in on auditing processes designed to achieve fairness, accountability and transparency. This narrow focus can divert attention from more fundamental questions, such as whether or not certain systems should be built at all, whether because they have proven harms or because they have unproven benefits.

However, given the striking lack of research on the politics of mental health and disability-related automation,¹⁹⁰ these themes provided a way to frame our discussion using themes that are common in broader public discussion about algorithmic and data-driven technology, even as these themes may need to be re-framed. Finally, the value of the discussion is dependent upon the integration of these themes in larger public governance systems, from legislation, regulation, the work of professional associations, the advocacy of civil society organisations, activism, quality journalism, and everyday practices designed to support people or communities in crisis.¹⁹¹

2.1 Privacy

The right to privacy is longstanding in international human rights law and aims to protect a citizen from unlawful interference with their private life and correspondence.¹⁹³ Yet, the contemporary communication ecosystem allows vast amounts of data to feed into technologies for use in surveillance, advertising, healthcare decision-making, and many other sensitive contexts.¹⁹⁴ Complex forms of government and private sector monitoring exist that draw on sophisticated technologies to trace individuals and detect their behaviour, networks, consumption and so on. The role of data concerning mental health in these processes is poorly understood.

Clearly, privacy over individuals’ and communities’ data concerning distress and mental health is vital. Failure to attend to privacy issues could have multiple negative consequences: it may shape individuals’ willingness to disclose their distress (for example, privacy concerns may undermine a person’s confidence in seeking support); data may be used to discriminate against individuals, families or groups, which could be used in the context of insurance, employment, housing, credit ratings and so on; data may be sold or monetised without an individual’s consent, including being used to inform manipulative advertising practices; and may enable identity theft and health system fraud. People may avoid disclosing or accessing a service for fear that their medical biography or data on their mental and personal life may be used to their disadvantage in the future—which crucially undermines trust, which is so essential to therapeutic engagement and the seeking of support.

Alternatively, many people will simply be unaware of the risks. Indeed, most consumers have low awareness about the implications of data sharing practices within the larger communication eco-system.¹⁹⁵

The proliferation of direct-to-consumer apps and therapy platforms highlights the expansion of digital mental health initiatives in marketised form. Nicole Martinez-Martin has written that ‘the consumer domain presents particularly vexing issues for trust, because the frameworks for accountability and oversight, as well as the mechanisms for data protection and assuring safety and effectiveness, are still evolving’.¹⁹⁸

2.1.1 Ad-Tech and Predictive Public Health Surveillance

Data and analytics in advertising can exploit behavioural biases and create consumer exploitation or be used in political targeting on an unprecedented scale.199 Sensitive information about people who are potentially in distressed states can be used by private companies to manipulate people into buying certain services or products.

Conversely, a similar approach to targeted advertising can be used in public health initiatives that seek to direct people who may be in distress to particular mental health services. The aim of such initiatives is to provide ‘pre-emptive’ support to assist people to access support, particularly those who may be averse to accessing formal services, as the following example shows.

The intent of the Good Thinking targeting service, which is described as a ‘precision prevention initiative’ by its developers, is to provide benefit to individuals in apparent distress. This aim aligns with the UK Government’s Green Paper on ‘Advancing our health: prevention in the 2020s’, which describes a move towards ‘proactive, predictive and personalised prevention’.²⁰⁶

Serious questions may be raised about such programs given that they seem to largely target people from outside the formal healthcare system who have not consented to or necessarily intentionally sought out health care services. Although the advertisements for Good Thinking are targeted at those who appear to be searching for support in relation to distress, one component of the targeted digital advertising ‘[t]argets users whose behaviour, demographic and location suggests they are a potential service users [sic] – a “passive” audience.²⁰⁷

Data-based targeting and automated profiling was discussed by the World Health Organisation in its report, Ethics and Governance of Artificial Intelligence for Health, which noted its ambiguous potential:

A prominent example of a data-driven preventive health monitoring initiative that faced a severe public backlash was the ‘suicide watch radar’ app that was trialled in the UK.

Automated profiling such as the Samaritan’s ‘Radar’ app clearly engages the new generation of data protection laws. The EU’s General Data Protection Regulation (GDPR), for example, defines automated profiling as ‘any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements.²¹¹ Other parts of the GDPR would bear on how such profiling could be used. Article 22, for example, states that ‘the data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.²¹² Bernadette McSherry has noted that predicting aspects of a person’s mental health appears likely to fall within the ambit of this Article.²¹³

2.1.2 Privacy and Monetisation of Sensitive Personal Data

Privacy issues are compounded by the increasing monetisation of health and other forms of personal data. Online mental health initiatives are emerging in an internet that is increasingly dominated by profit-driven information flows. Some mental health websites or apps and affiliated third-party companies are treating the personal data of users as a commodity and tracking them for marketing or other commercial purposes.²¹⁴ This may occur as an explicit business decision by a private company that provides direct-to-consumer services for those in distress, or may occur inadvertently where a service provider is unaware of the way third-party trackers are operating on their platforms. ‘Third-party trackers’ which are sometimes described as ‘tracking cookies’ or ‘trackers’, are elements of websites that are created by parties other than the developers of the website the person is currently visiting; this would include providers of advertising, analytics and tracking services.²¹⁵

Knowledge of a user’s distress could, at a minimum, allow companies to advertise specific treatments, services, or financial products, as noted previously. It could also be sold to other interested parties, such as insurers, as discussed later in the report (see Non-Discrimination and Equity). Some have suggested that data concerning the health of individuals will be more lucrative than the sale of particular health products. Nick Couldry and Ulises Ali Mejias have argued that this likelihood is evident in Amazon’s recent moves in the US to open an online pharmacy:

Not only may data extraction be used to predict the person’s distress in order to match them to an advertised product, but another broader function may be to shape the person’s experience and behaviour in order to direct them to existing advertisement/ products. Zuboff refers to this shaping of human experience and behaviour when highlighting the emergence of ‘behavioral futures markets’.²¹⁹ This may be evident in the Cerebral app, noted above, which reportedly pushed platform users toward shorter appointments and more prescriptions in ways that potentially ‘accelerat[ed] the psychiatric prescribing cascade’.²²⁰

It is clear that technologies are now being designed to push ‘users’ to access services or products aligned with business interests tied to the technology;²²⁵ or to enforce conditional welfare and social benefit rules in government-funded services in ways that erode care,²²⁶ as will be discussed later in the report.

Privacy International’s finding that mental health websites sell visitor information to third parties highlights a striking fact: it is becoming harder to access mental health support without that access being digitally recorded in some way. The likelihood of such information moving beyond the discrete and relevant digital repositories of one service is increased by the massive and interconnected flow of data in today’s communication ecosystem. A report for the Consumer Policy Resource Centre notes the implications of the ease with which data can be transported:

This concern extends to accessing physical services given that the monitoring power of smartphones through location-tracking can potentially show the frequency and types of healthcare services an individual accesses.²²⁸ Some mental health initiatives that introduce a major digitalised or virtual component have explicitly prioritised privacy as a key component of appropriate support.

Go Mental by Josh Muir in Science Gallery Melbourne’s MENTAL. Photo by Alan Weedon.

2.1.3 Data Theft and Data Trafficking

As the amount and value of personal data stored online proliferates, data theft and trafficking will continue to occur. In 2017 in the US, for example, a mental health service provider in Texas notified 28,434 people whose data were allegedly stolen by a former employee.²³⁰ However, by far the most extreme case concerning mental health was reported in Finland in October 2020

Such incidents raise questions around the security required to protect people’s privacy relating to mental health, distress and disability, to digitally store and process sensitive personal data (of which more is discussed in the Safety and Security section below). According to William Ralston, the example from Finland is particularly troubling because Finland is regarded as having among the most advanced electronic health policy and governance frameworks in the world.²³⁵ Questions also arise about the security methods in place for technologies that are operating outside the formal healthcare context, such as the vast selection of mental health apps operated by private companies collecting personal data through people’s smartphones. Indeed, the private company in Finland that was hacked, Vastaamo, was the largest private mental health operator in the country, and investigations is underway at the time of writing to determining where responsibility for the data breach lies.²³⁶

2.1.4 Privacy and Discrimination

Privacy can be closely linked to issues of discrimination in the mental health context because a person’s mental health status – such as their psychiatric diagnosis or record of encounters with health services – can be used in ways that are leveraged against them, including by potential employers, insurers, and state agencies

This troubling case study highlights how privacy laws can be used to protect the sharing of data concerning people’s mental health. However, privacy laws in many countries were generally written prior to the explosion of algorithmic and data-driven technologies and are therefore unlikely to provide robust protection for people’s data concerning health in many places. One challenge is that privacy law and policy in various countries contain different definitions of ‘personal data’ and ‘sensitive personal data’, which means various forms of data concerning a person’s mental health, distress and disability may or may not be protected

2.1.5 Data Protection Law

It is generally agreed that robust data protection laws can provide a more comprehensive framework compared to privacy law for protecting a range of forms of personal data, and can also include additional rules for categories like health and research data. The EU’s GDPR is an influential example of data protection rules designed to remedy gaps caused by fuzzy definitions of what constitutes personal data; it specifies steps any organisation or agency handling ‘personal data’ must take in order to uphold the right to privacy. This includes ‘sensitive personal data’, which extends to ‘data concerning health’. In the US, the California Consumer Privacy Act takes a similar direction, though the scope of the GDPR is broader.²⁴⁰

The GDPR explicitly does not include the term ‘health data’ and instead uses the broader phrase ‘data concerning health’. Schneble and colleagues argue that this important distinction ‘opens the door to indirect and inferred health data falling within the scope of the GDPR’, and therefore strengthens its application outside as within the formal healthcare system. It is too early to determine the extent to which Schneble and colleagues are correct.

Others remain sceptical that even leading data protection laws like the GDPR can sufficiently protect people in the mental health context against the full range of the harms that may arise. Nicole Martinez-Martin and colleagues, for example, refer to the risk of misuse of data that is used to infer things about the health of an individual, and stated that:

Even where harmful or potentially harmful practices are identified and found to be violating data protection laws, the success of those laws is dependent on the capacity of authorities to enforce compliance, which remains an issue with the GDPR.²⁴² In any case, the GDPR is complex and only applies to organisations based in the EU. More conceptual and regulatory work is required to better define and regulate ‘data concerning health, mental health and disability’ and their use in automated profiling, to address issues of ‘indirect, inferred, and invisible health data’

Regarding law more generally, just as there is a risk of idealising technology’s promise, so there is of law: vigilance is required as to whether law reinforces unjust power relations. For example, if regulatory regimes to protect privacy are characterised by light-touch, pro-industry approaches that are designed to ease market authorisation of digital mental health services and products, this may promote the spread of cheap (if limited) software to replace more expensive, expert, and empathetic professional support, and disrupt care service provision.243 Regulation should aim to reduce all forms of domination, but there is always a risk that it will fail and/or reinforce domination.244 Some legal scholars have argued that laws governing privacy, data protection, and consumer protection have failed to govern the platform dominance of major technology corporations. Further, such laws have contributed to the massive expansion of big technology corporations into market-like structures that distort social relations and convert individuals into ‘users’—a resource to be mined for data and attention.²⁴⁵

More work is required to bring together those working on algorithmic and data-driven technology in response to disability and distress, with those who are pursuing broader alternative arrangements for the governance of our digitally mediated lives and economies. Possible alternatives include collective approaches to governing data and platforms, and community-produced data resources.

2.1.6 Informed Consent

Rights of autonomy and decision-making have been a crucial concern in traditions of service user and survivor advocacy, activism, research, and so on. Informed consent, which is a key component of upholding the right to privacy but also has far broader importance, is key to rights to autonomy and decision-making, as reflected in human rights instruments, such as the Convention on the Rights of Persons with Disabilities (see articles 3 (general principles) and 12 (equal recognition before the law)). Like other human rights instruments, as UN

Special Rapporteur for the rights of persons with disabilities Gerard Quinn points out, the ‘Convention requires that consent should be informed, real, transparent, effective and never assumed’—and this is certainly the case in algorithmic and data driven developments.²⁴⁶ According to Quinn, autonomy is implicated, ‘where machine learning uses profiling and other decisions affecting persons with disabilities without their knowledge.²⁴⁷

Informed consent is particularly important with digital forms of diagnosis or proxy-diagnosis. The consequences of being diagnosed and pathologised in the mental health context, whether accurately or not, are often profound. Indeed, algorithmic and data-driven technological interventions in mental health services or in commercialised products that have a significant impact on individuals should never occur without their free and informed consent. All informed consent processes in the digital context should provide sufficient details of safety and security measures, including information about the person or entity that monitors compliance. (See also, Recommendation 5)

Overall, privacy is probably the most prominent theme in public discussion about the ethical and legal issues on data concerning people’s mental health,²⁴⁸ though this does not mean it is the most important. It could be reasonably asked whether privacy should dominate such discussion in comparison to other concerns, as it has a tendency to reduce the conversation to the level of the individual (rather than, say, social and economic underpinnings of distress, or collective claims to using data as a democratic resource rather than an individually owned artefact). Nevertheless, much work remains in applying principles of privacy to the mental health context in the digital era. This includes consideration of:²⁴⁹

Any major effort to unpack these issues requires the active involvement of those most affected. It is also now unavoidable that new government regulation and robust enforcement is needed to protect privacy in the face of algorithmic and data-driven technologies. As advocacy organisation Access Now note, ‘data protection legislation can anticipate and mitigate many of the human rights risks posed by AI [and other algorithmic technologies]’.²⁵⁰ The Access Now position echoes a growing demand by some advocates for new data laws, enforceable penalties and the resources for affected communities to be proactive in contributing to enforcement.²⁵¹ The need for law reform remains a subject of expanding scholarship that should continue to inform and be informed by developments that particularly impact people with lived experience and psychosocial disability.

2.2 Accountability

Accountability for the impacts of algorithmic systems in mental health and disability contexts must be appropriately distributed, with adequate remedies in place. Accountability measures are needed to ensure opportunities to interrogate the objectives, outcomes and inherent trade-offs involved in using algorithmic systems, and to do in a way that centres the interest of the user-subject and the broader public, not just the entity using the system.²⁵³ The appropriate attribution of responsibility and redress is not only vital for individuals who are affected but can be vital for public trust in technology-driven solutions.

This type of case study raises important questions about accountability—questions which could be generalised about any biometric monitoring concerning cognitive impairment and disability.

What must researchers do to consider the impact of the initiative, including potential harms to those designated as ‘cognitively impaired’? Should the methods for such monitoring be widely shared given the ubiquity of mobile phone use in many parts of the world, and the apparent ease with which private companies can collect data that ‘strongly correlate with cognitive health’? If the biometric monitoring used in the study was used outside experimental conditions, what safeguards would be in place to allow those designated as impaired be able to contest that designation before it was transferred to other entities? If such technologies were deployed on a larger scale, who would be responsible in the event of an adverse outcome? For example, if an app collecting sensitive personal data relating to people’s cognitive status inadvertently releases the data to a third-party because it malfunctions, or is compromised by a security flaw, who is responsible? Is it the company that owns the app, the individual programmer(s) who made the error, the service that recommended or even prescribed the app?

These are among the questions that may be asked about accountability. Public efforts to promote accountability tend to suggest that different strategies are needed at different stages in the ‘lifecycle’ of algorithmic and data-driven systems, particularly during design (pre-deployment), monitoring (during deployment), and redress (after harm has occurred).²⁵⁶ Possible strategies include:

These principles may overlap between categories of design, monitoring and redress.

A common point in efforts to achieve accountability is the need to avoid placing accountability on the technology itself rather than on those who design, develop and deploy it. Governments, companies and their business partners, researchers, developers, and users/subjects will have varying degrees of responsibility for harms depending on context. There is a vital role for individuals, advocates, and technical experts in flagging errors and demonstrating the adverse effects of various new algorithmic technologies, but there need to be forums and institutional mechanisms for these concerns to be raised and, where necessary, acted upon

2.2.1 Privatisation and Accountability

Private sector actors are playing a prominent role in designing, constructing, and operating algorithmic and data-driven technologies in the mental health context. Traditional accountability mechanisms are not always equipped to ensure these interests align with the public good, particularly where the divide between public and private entities becomes blurred.²⁶⁷ Philip Alston, former UN ‘Special Rapporteur on extreme poverty and human rights’, has written that ‘[a]ccountability is the linchpin of human rights, but privatization has rendered existing mechanisms increasingly marginal’.²⁶⁸ The information economy has arguably accelerated this process.

Accountability requires a clear definition of who is accountable and who can hold actors accountable, including effective oversight systems that can trace the conduct of actors and to assess whether standards and requirements are met. Privatisation of services, such as mental health and social services, can undermine this clarity and oversight. The rise of private sector actors in ‘social protection services’, according to Alston, has been accompanied by a ‘deeply problematic lack of information about the precise role and responsibility of private actors in proposing, developing and operating digital technologies in welfare states around the world’.²⁶⁹ Further:²⁷⁰

This lack of transparency has a range of causes, from gaps in freedom of information laws, confidentiality clauses, and intellectual property protections, through a failure on the part of legislatures and executives to require transparency, to a general lack of investigation of these practices by oversight bodies and the media. The absence of information seriously impedes efforts to hold governments and private actors accountable

The case study of ‘Serenity Integrated Monitoring’ (or ‘SIM’) in England (see Section 2.3.2), in which mental health legislation data was used to flag individuals for police intervention and exclusion from emergency psychiatric services in the UK, offers one such example. The SIM program was rolled out to 23 National Health Service mental health trusts in England despite a lack of evidence of its impact on patient safety or outcomes. Instead, the little research supporting its implementation simply demonstrated reduced costs to services. SIM was owned and run by the High Intensity Network, a private limited company that was financially supported by the ‘NHS Innovation Accelerator’ and ‘Academic Health Science Network’.²⁷¹ This latter network comprises of the ‘NHS and academic organisations, local authorities, the third sector and industry’ and seeks to ‘spread innovation at pace and scale – improving health and generating economic growth’.²⁷² After a coalition of activists called for an immediate halt to the program, the High Intensity Network appears to have closed permanently; its website was removed and its social media presence wiped.²⁷³ Activists raised concerns that the outsourcing of service provision to a private company meant the program fell between gaps of traditional accountability mechanisms. The ‘StopSIM Coalition’ wrote:²⁷⁴

Following this statement, the Royal College of Psychiatrists (UK) called for an ‘urgent and transparent investigation’ not only into the SIM program and the High Intensity Network, but also into the ‘NHS Innovation Accelerator’ program that supported it.²⁷⁵ The ‘Innovation Accelerator’ program supports several digital mental health initiatives, including remote biometric monitoring of patients in acute psychiatric wards, which are being expanded through the NHS—arguably with a similar lack of robust supporting evidence.²⁷⁶

At a policy level, governance and regulatory discussions also risk being driven by private interests. In 2019, a ‘White Paper’ titled, Empowering 8 Billion Minds Enabling Better Mental Health for All via the Ethical Adoption of Technologies, was published by the World Economic Forum and authored by the multinational corporation, Accenture, which specialises in IT services and consulting.²⁷⁷ The authors urged:

Framing ethical consideration and sufficient evidence as barriers to digitally-enabled treatment reverses the typical academic method, in which ethical review and evidence are needed before determining whether a particular treatment is beneficial and effective. Reading generously, it is possible the authors were instead suggesting more research and ethical discussion are needed to expand on promising preliminary research. Yet clearly, vigilance is needed. There must be transparency about the business models of private firms, and the motives of brokerage organisations like the World Economic Forum. Many tech vendors and other private sector actors will be seeking lucrative government contracts or angling for a predetermined path to bringing certain technologies to market. The role of such actors in developing governance systems in the digital mental health context, and the growing economy and vested interests that surround them, must be made transparent, with consideration as to the appropriateness of that involvement.

The World Economic Forum has now published two prominent reports on digital technologies in mental health services. The other prominent report, a ‘Global Governance Toolkit’, was led by the multinational accountancy/professional services company, Deloitte.²⁷⁹ As an international body, the World Economic Forum is one of the primary agenda-setting organisations today. Yet, the Forum has been criticised for operating in ways that do not align with democratic values. Christina Garsten and Adrienne Sörbom conducted a detailed ethnographic study of the World Economci Forum and concluded that it operates using ‘discretionary governance’ at the transnational level, which entails ‘the exercise of a discreet form of power and control according to the judgment of the Forum and its members’ that operates in ‘ways that escape established democratic controls’.²⁸⁰

This is not to criticise the aspirations of everyone involved in these reports, whether as contributors or advisors, many who will hold their views on digital mental health care in good faith (even as others will have solely been interested in increasing company margins). Instead, it is to highlight the increasing role of private sector actors in pushing digital technologies, including growing efforts to shape governance frameworks and institutions, and steering regulatory attention in preferred directions to reproduce and protect their business model.

2.3 Safety and security

Some commentators have raised stark warnings about safety and security in the digital mental health context.²⁸¹

2.3.1 Safety

‘Safety’ typically refers to ensuring the technology avoids unintended harms and functions as intended

In a commentary on the COVID-19 pandemic and digital mental health services, Martinez-Martin and colleagues discussed safety as a key issue. They drew attention to online counselling, emphasising the need for online counsellors to ensure safety measures for those they’re supporting, include ‘safety planning for patients who are at high risk’ as well as measures to maintain ‘professional boundaries in the newly informal virtual space […]’.²⁹⁰ The authors refer to Germany’s Digital Health Act (Digitale–Versorgung–Gesetz) as potentially offering a good model for navigating several safety concerns. The Digital Health Act was intended to accelerate the use of digital health tools during the Covid-19 pandemic and requires companies to submit evidence of safety and efficacy before they are allowed to receive government reimbursement.²⁹¹ Martinez-Martin and colleagues argue that similar ‘regulation could help to provide a more consistent system for evaluation of digital health tools and ensure that users have access to safe products’.²⁹²

2.3.2 Security

‘Security’ tends to refer to addressing external threats to data-driven systems. An example is the 2020 Vastaamo data breach in Finland, noted earlier in the report ( X X). To recap, over 30,000 people’s psychotherapy records from the Vastaamo private counselling service were hacked and used to extort victims, in what the then public prosecutor described as perhaps the largest criminal case in Finnish history in terms of the number of victims.²⁹³

Security assurances against data concerning mental health and disability must exist in the interest of protecting the integrity and confidentiality of personal data. According to Martinez-Martin and colleagues, ‘[b]ehavioral health information is a valuable commodity, and it is likely that companies will take further advantage of the lax security and privacy landscape’.²⁹⁴ Indeed, the healthcare sector is particularly attractive for those perpetrating cyberattacks, with major incidents reported worldwide.²⁹⁵ Kyriaki Giota and George Kleftaras point out that ‘[p]ersonal health information is of great value for cyber-criminals and can be used in order to obtain medical services and devices, or bill insurance companies for phantom services in the victim’s name’.²⁹⁶

Mental health apps – again, of which there are reportedly over 10,000 – appear particularly vulnerable to poor security processes. A study by Kit Huckvale and colleagues found that not one of the 79 apps in the UK NHS Health Apps Library encrypted user data stored on the phone.²⁹⁷ The apps did commonly use password security, though Huckvale and colleagues observed that this could lead a user to believe their data were secure.

One common strategy to address security concerns is to seek to anonymise, de-identify, or aggregate data where possible.²⁹⁸ Another strategy is to make clear the specific content of rights and obligations for technology developers, product manufacturers, or service providers and ‘end users’.²⁹⁹

Concepts such as ‘security by design’ have been floated to address security concerns early on.³⁰⁰ A key component of responsible governance would be ensuring that institutions that are handling data concerning mental health, distress and disability meet relevant data security requirements. Martinez-Martin and colleagues have also stressed that the ‘details of [security] measures, and who shall prescribe them and monitor compliance, will need explicit definition and should be included in the informed consent process’.³⁰¹

Security will remain a pressing task for the foreseeable future. To date, cybersecurity researchers have detected compromises in more than 100 million smart devices around the world.³⁰² It will be unsurprising to see more major breaches of data concerning mental health and disability in the near future.

Go Mental by Josh Muir in Science Gallery Melbourne’s MENTAL. Photo by Alan Weedon.

2.4 Non-Discrimination and Equity

Key themes concerning non-discrimination and equity in the mental health context, include that:

2.4.1 Non-discrimination and the Prevention of Bias

The potential for algorithmic bias or discrimination is a well-documented issue. Much public discussion in this area has focused on gender, race and socio-economic inequality³⁰⁴. However, disability, including mental health and psychosocial disability, ‘has been largely omitted from the AI-bias conversation’.³⁰⁵ Whitaker and colleagues have argued that ‘patterns of marginalization [concerning disability] are imprinted in the data that shapes AI systems, and embed these histories in the logics of AI’.³⁰⁶ For example, Ben Hutchinson and colleagues at Google, demonstrated that social attitudes casting disability as bad and even violent – particularly in regard to mental health – were encoded in AI systems designed to detect hate speech and identify negative/positive sentiment in written text.³⁰⁷ The ‘machine-learned model to moderate conversations’, according to Hutchinson and colleagues, classifies texts which mention disability and particularly references to mental health as more ‘toxic’, while ‘a machine-learned sentiment analysis model rates texts which mention disability as more negative’.³⁰⁸ Such studies highlight how biased datasets create biased algorithms, which can have major consequences for people’s lives, as the next example shows.

This case study is revealing. Mr Behm was seemingly harmed due to data to which he was never given access. Nor does it appear that Mr Behm had an easily accessible opportunity to contest, explain or investigate the test outcome. Cathy O’Neil argues that this type of algorithmic ‘red-lighting’ has the potential to ‘create an underclass of people who will find themselves increasingly and inexplicably shut out from normal life’.³¹¹

One response to biased algorithmic systems has been to focus on creating un-biased datasets. Datasets could be made more diverse, the argument goes, to capture diverse human experiences. This would avoid negative consequences for people who, through the various human and circumstantial complexities in their lives, are considered ‘statistical outliers’ for whom algorithmic decision systems are ill-equipped. This approach is certainly warranted in some circumstances, where the need for good quality and representative data can help avoid biased or discriminatory outcomes.

However, the aim of creating unbiased datasets will be insufficient in many situations. Meredith Broussard criticises this approach as being commonly ‘technochauvinist’ in nature.³¹² Techno-chauvinism refers here to the false view that technological solutions provide ‘appropriate and adequate fixes to the deeply human problem of bias and discrimination’.³¹³

Representative and high-quality datasets will be important in some instances but Broussard’s criticism suggests that there is a second major category of discrimination at play: namely, discrimination perpetuated by human systems and institutions using data concerning mental health. Examples might include insurance companies discriminating against people based on data showing that they accessed mental health services at one time,³¹⁴ or police and border authorities discriminating against people based on non-criminal data concerning an individual’s engagement with mental health services, as the next example shows.

This example suggests that no amount of ‘unbiased datasets’ will offset the discriminatory premise of various digital initiatives, which are designed to intervene in the lives of persons with lived experience and disability on an unequal basis with others.

Discriminatory impacts are also more likely as algorithmic and data-driven technologies are applied in settings affecting marginalised populations. This includes settings in which there are broader constraints on individuals’ agency, including cumulative effects of disadvantage. This could include ethnic and racial minorities, people facing involuntary psychiatric intervention, families or individuals facing housing insecurity, returning veterans, people with addiction, the previously or presently incarcerated, and migrants and asylum-seekers.³¹⁸ LLana James points to these concerns when she asks: ‘How will data labelled as Black, poor, disabled or all three impact a person’s insurance rates?³¹⁹ James argues that current laws (writing in Canada) do not appear to protect health service recipients and patients and calls for updated data laws to protect against ‘the perils of monetized data and the discriminatory algorithms they are generating’.³²⁰

Regarding insurance, Access Now have expanded on James’ point regarding exclusion and insurance-based discrimination, noting that:

This point is concerning in the mental health context, and the emergence in recent years of partnerships between insurance companies and mental health technology companies³²² and other insurance company initiatives concerning mental health-related data warrant scrutiny.³²³

The likelihood of disability-based discrimination will be compounded when data scientists, technologists, tech entrepreneurs, clinical innovators, and so on, are not aware of the potential for discrimination using data concerning mental health. Consider the following claims being made in the ‘emotion recognition’ industry in China.

AI Now Institute, an interdisciplinary research centre examining artificial intelligence and society, have called for a ban on technology designed to recognise people’s emotions concerning ‘important decisions that impact people’s lives and access to opportunities’.³²⁶ This scope would surely extend to automated forms of diagnosis or proxy-diagnosis of cognitive impairments or mental health conditions. The proposed ban could apply to decisions concerning hiring, workplace assessment, insurance pricing, pain assessments or education performance. AI Now base their recommendation on two concerns: 1) the often-baseless scientific foundations of emotion recognition, and 2) the potential for bias and discrimination in the resulting decisions.³²⁷

Emotion or ‘affect’ recognition technology such as facial recognition technology raise important issues for this report. At least three points relating to non-discrimination and the mental health context are worth noting:

There are important differences between the motives and claims of commercial actors who are promoting affect or emotion recognition technology, and that of biometric monitoring in clinical studies conducted by mental health researchers—and there are major differences in the regulatory frameworks affecting each. In general, health research is far more tightly regulated, with more entrenched infrastructure for upholding ethical research involving humans. Although not without serious problems, including the interference of private industry with academic research,330³³⁰ and the growing reliance of universities on the private sector to fund research,331³³¹ the scholarly health research infrastructure appears better developed for the purposes of ethical oversight, than many private sector uses of affect recognition technologies, including where those technologies are sold to government agencies, such as police agencies and border authorities.

Nevertheless, there are many examples of overlap between commercial and clinical activities in the digital mental health context,³³² and public scrutiny is required of clinical or scholarly claims about what behaviour can convey about a person’s inner-world. It is not possible in this report to examine the claims being made about psychiatric biometric monitoring. Instead, the aim in this section is to draw attention to poorly understood potential for discrimination and bias in the use of such technologies in the mental health and disability context.

Concerns about discrimination need not relate to algorithmic systems. For example, a coalition of Australian organisations representing people with lived experience and psychosocial disability called for a suspension of a national electronic health records scheme, citing fears of discrimination if personal mental health histories were stolen, leaked or sold.³³³ Previous case studies cited throughout the report demonstrate how such discrimination might occur.

One important step to preventing harms caused by data concerning mental health and disability – whether leaked, stolen or traded – is to strengthen non-discrimination rules concerning mental health and psychosocial disability.³³⁴ National discrimination laws may require amendments to ensure that discrimination on mental health grounds by online businesses is covered.³³⁵ Such amendments would be consistent with the goals and legislative history of anti-discrimination laws and would remove ambiguity regarding the status of websites, social media platforms and other online businesses.³³⁶ Remedies for individuals who are aggrieved by discriminatory behaviour and practices are also likely to require strengthening, including ensuring substantive, verifiable, auditable standards of non-discrimination in the use of algorithmic and data-driven technologies

2.4.2 Fairness

Fairness is a broadly shared aspiration for governing algorithmic technologies. Definitions of ‘fairness’ differ, and the notion of ‘algorithmic fairness’ itself is an increasingly expanding field of research.³³⁷ There are reportedly between 15 and 25 plausible definitions of ‘fairness’ of relevance to algorithmic technologies, each with different and often mutually exclusive emphases.³³⁸ Depending on the context, some definitions are constructed in highly technical ways, centred on data science expertise, while others draw on the common-usage (and equally broad) aims of equitable and impartial treatment.³³⁹ There is a risk that vague references to ‘fairness’ may hide important political decisions about how fairness is understood, including – importantly – who defines it.

Such choices ought to be transparent when used in the design of mental health related technologies, partly to clarify objectives, but also to highlight who may gain (or lose) political and decision-making power depending on the approach to fairness that is adopted. If addressing fairness becomes highly technical, for example, requiring the expertise of computer scientists, mathematicians, and so on, there is seemingly less scope for those most affected to determine the parameters of what is considered fair and unfair

2.4.3 Equality

Equality is another generally accepted aspiration, centred on the goal of ensuring the same opportunities and protections to people interacting with algorithmic systems. Some have taken this aim further in seeking to use algorithmic systems to ‘eliminate relationships of domination between groups and people based on differences of power, wealth, or knowledge’ and ‘produce social and economic benefits for all by reducing social inequalities and vulnerabilities.’340³⁴⁰

Any discussion of equality in the mental health context must acknowledge existing inequalities in how mental health issues play out. Psychological distress does not occur equally across society: those who are poorer, from disadvantaged, marginalised and oppressed groups are more likely to experience distress, psychosis, trauma, mental health conditions and psychosocial disabilities.

Inequalities also appear within mental health services themselves. There is inequality of access in mental health services; for example, in the UK, older people are underrepresented in talking therapies³⁴¹ and Black British men are overrepresented in involuntary psychiatric interventions.³⁴² Inequalities of experience also arise. In Aotearoa New Zealand, people with higher economic deprivation report lower satisfaction with health services compared to others, and this group disproportionately includes high numbers of people of Māori, Pacific or Asian ethnicity.³⁴³ In high-income countries, higher proportions of Black people get diagnosed, with much research and debate about why this is the case.³⁴⁴

Some would argue that algorithmic and data-driven technologies could be used to address such inequalities; for example, helping to identify inequities in service systems, or by undertaking analyses of complex socio-economic dimensions to mental health problems. Yet, there is also potential that such technologies will replicate and even exacerbate inequalities. Sarah Carr, speaking from a UK perspective, has pointed out that the higher likelihood that Black British men will be subject to involuntary psychiatric intervention may mean that algorithmic approaches to service provision could exacerbate patterns of coercive intervention against racialised minorities.³⁴⁵ Psychiatric diagnoses are already skewed with respect to race. In the US, for example, Black and minority ethnic groups tend to receive more ‘severe’ diagnostic categories (eg. schizophrenia rather than schizo-affective disorder), for diverse reasons, including mental health practitioners seeking to ensure a low-income person can qualify for housing or social security, which a more ‘severe’ diagnosis might afford.³⁴⁶

The ‘Serenity Integrated Monitoring’ program in the UK, noted earlier ( X), which involves analysing health authority data to identify people repeatedly subject to forced psychiatric treatment and referring them to a police monitoring program, has been criticised for the likelihood that it will have ‘violent consequences [that] disproportionately impact Black, Asian and minority ethnic communities’.³⁴⁷ Sage Stephanou, founder of the Radical Therapist Network (RTN), stated:

Again, the SIM program only used data-driven technology at a small but crucial point in the program. Yet, the example suggests it is necessary to move beyond vague notions of equality and fairness in efforts to ensure the prevention of harm and equal distribution of benefits of using algorithmic and data-driven technology to address distress and healing.

As a final point on equality for this section, broader structural questions may be asked about growing inequality facilitated and indeed accelerated by the algorithmic and data-driven systems that power the information economy. The impact of social, political, and economic structures on mental health is well established. Drilling down into how a particular algorithmic system promotes or threatens equal opportunities for individuals with a mental health diagnosis under current conditions may distract from the way human distress arises primarily as a consequence of poverty, precarity, violence, and trauma as a form of social suffering—for which growing inequality in many countries will continue to be a major contributor. The role of technological change in these broader trends has a crucial role in discussions about new and emerging technology aimed at ameliorating distress. These issues will be discussed later in the report, in the section concerning public interest and societal wellbeing (page 83).

2.4.4 Inclusive Design – Emancipatory? Participatory?

Current algorithmic and data-driven initiatives in the mental health context are dominated by actors that have the most power, such as large private entities and public institutions, service providers, universities, professional associations, and so on. This concentration of power can mean a lack of digital technology oriented to experience and real-life usage. Sarah Carr writes:

For a fuller discussion of this point see above, Section 1.5 “Elevating the Perspective of People with Lived Experience of Extreme Distress and Disability”. Given that every person could generate ‘data concerning mental health’, a broad cross-section of society should have the opportunity to weigh in on the use of algorithmic and data-driven technology in mental health contexts. However, those with the most at stake tend to be those who have firsthand experience of using mental health services, including those who have been subject to involuntary psychiatric interventions, received a psychiatric diagnosis, or who just live with profound distress, a mental health condition or psychosocial disability. As noted previously, this group should not be viewed merely as another ‘key stakeholder’ but as the primary affected population whose involvement should form a political and ethical orientation underlying all work in this area. ‘Inclusive design’ is often associated with notions of ‘human rights by design’, which will be discussed below regarding human rights law (X)

Photo by Milad Fakurian on Unsplash.

2.4.5 Access to Technology

Equity of access to technology is an important social good, and is often reflected in concerns with a growing ‘digital divide’ in society. Several researchers have examined this concern as it affects people with psychosocial disabilities and lived experience.³⁵⁰ For example, Til Wykes has commented in relation to mobile technology that:³⁵¹

In the UK, Dan Robotham and colleagues looked at ‘digital exclusion’ facing 241 participants with mental health diagnoses.³⁵² The researchers concluded that the ‘digital divide’ was difficult to overcome but that successful steps could be taken to improve access to digital technologies for people who lack the knowledge, skill and financial resources, and that such steps could even form part of an essential service for citizenship and community wellbeing.

Digital inclusion strategies – such as subsidising the purchase of equipment, internet billing support, education to improve digital literacy, and so on – may be required to prevent people becoming excluded from both digitised health and social services, but also from society in general.³⁵³ However, addressing digital equity may also mean ensuring that people can access entirely ‘non-digital’ resources for those who do not wish to, or cannot, use digital technological approaches to care and support. Equally, it is important to avoid generalisations about people with lived experience or psychosocial disabilities’ supposed deficiencies in digital literacy and access. False assumptions about people with disabilities’ supposed passivity and incapacity can result in paternalistic, top-down approaches that falsely presume a need for state and industry intervention.³⁵⁴ One public inquiry in Victoria, Australia, recommended that governments could help address the ‘digital divide’ in the mental health context by ‘enabl[ing] mental health and well-being services to offer people living with mental illness or psychological distress access to devices, data and digital literacy support, where it is their preference to use digital services but they are otherwise unable to do so.³⁵⁵ Elsewhere in Australia, case law has established the potential for social security recipients to use their disability support funds for internet hardware and data plans.³⁵⁶

2.5 Human control of technology

Important decisions concerning mental health that are made with algorithmic technology must be subject to human control and review. Human control over technology is key to other themes discussed in this report, including maintaining safety and security, accountability, transparency, equity and non-discrimination.

The advance of algorithmic and data-driven technologies is often presented as inevitable, with thought leaders in computer science often depicting automation as a force of nature propelled by unstoppable technological change.³⁵⁷ This view risks delegating autonomy and responsibility for such systems away from humans to some higher and abstract authority. Even terms like ‘artificial intelligence’ can imply an external intelligence that is somehow ‘out there’. Relinquishing accountability in this way is sometimes described as the ‘substitution effect’,³⁵⁸ an effect which makes it harder to ensure human control over technologies by those who are actually designing and implementing them—and those who are using and/or subject to them.

Examples exist of civil society groups successfully challenging the introduction of some types of AI, such as facial recognition. Such groups have insisted that these technologies are not inevitable and demonstrated the power of public input to change the direction of, and even halt, the use of certain technologies. Similar examples are emerging in the mental health context (see Section 1.5).

Emphasising the importance of human control in the mental health context is important given the risk that algorithmic technologies like AI become a ‘substitute decision-maker’ over both professionals and those receiving care and support, which will impact individuals’ agency and autonomy.³⁵⁹ One ethical risk in professional decision support technology in healthcare, is that clinicians defer to algorithmic technology suggestions even in the face of a contrary opinion. An overreliance on automated systems may therefore displace human agency, moral responsibility, liability and other forms of accountability. Even the use of ‘algorithm’ to describe a decision-making system has been characterised by some advocates as ‘often a way to deflect accountability for human decisions’.³⁶⁰

More broadly, there is a risk of normalising and accepting technologies which reinforce potentially inaccurate, unhelpful or harmful categorisations of people’s mental states (as discussed in the biometric monitoring section of this report).

This case study raises questions about the appropriateness (and lawfulness) of certain claims being made about individuals, including whether it is appropriate to claim that behavioural sensing can ‘reveal’ an underlying mental state or diagnosable disorder through ‘silent digital biomarkers’. The claim rests on a presumption about the capacity of computer technology, combined with psychometrics, to capture reality. Even the common claim that various technologies ‘collect’ data suggests that there is some neutral, objective information-gathering process underway. Instead, it seems more accurate to say that new forms of data concerning mental health are being created and generated. This creation and generation is not value-neutral—it is value-laden and often rests on multiple social and political claims that may be unstated. The nature of this data-generation, the meaning given to different types of data, and the often uncritical presentation of these methods as neutral forms of data ‘collection’ warrants critical scrutiny—whether in technology sales materials, media, government policy documents, or in leading scholarly journals.

Another risk is that the push to generate ever more data to improve algorithmic and datadriven solutions can undermine human action and control by distracting from the need to change existing mental health services, policies and practices. An extraordinary amount of energy can go into efforts to generate, curate, store and use data. This process can undermine action on information that already exists, particularly information highlighting existing problems of resourcing, discrimination, coercive practices, and other prominent issues in the politics of mental health.³⁶³

2.5.1 Human Review of Automated Decision

A more targeted way to ensure human control of automated decisions is to ensure that people who are subject to automated decisions that draw on data concerning their mental health should be able to request and receive human review of those decisions. The principle of ensuring human review, unlike most other themes discussed in this report, is meant as a step after the fact. This is not to endorse the use of various automated initiatives in the first place, some of which may warrant preventive interventions that modify or halt them. Instead, the principle of human review is to ensure an ex post (after the fact) mechanism of review where algorithmic decision systems are used. The principle is guided by the rationale, noted by the European Commission’s High-Level Expert Group on Artificial Intelligence, that ‘[h]umans interacting with AI systems must be able to keep full and effective self-determination over themselves’.³⁶⁴

Different technologies, and the contexts in which they are used, will require variations of the human review that is appropriate, including the strength of that review process. Some groups have argued that human review is not merely desirable but should be viewed as a right of the data subject.³⁶⁵ The European Ethical Charter on Artificial Intelligence in Judicial Systems and their Environment, for example, contains a robust approach, indicating that cases must be heard by a competent court if review is requested.³⁶⁶

2.5.2 Ability to Opt-Out of Automated Decision-Making

People must retain the ability to opt-out of algorithmic and data-driven approaches that use data concerning their mental health. This point is closely related to issues of consent and transparency (see Section 2.7). Members of the public deserve clarity about when algorithmic technologies are being used to make decisions about them using data concerning their mental health. According to a UK government report, greater clarity ‘will help the public experience the advantages of AI, as well as to opt out of using such products should they have concerns.³⁶⁷

Opting out is not always going to be clear-cut given that people are likely to interact with AI systems in numerous ways. As Fjeld and colleagues write: ‘[people’s] information may be used as training data; they may be indirectly impacted by systemic deployments of AI, and they may be personally subject to automated decisions.³⁶⁸ Attention is required to what it means to offer meaningful opportunities to exercise choice in opting out of data-driven technologies in mental health contexts.

One risk is that those who opt-out or discontinue interacting with these technologies are recorded as being non-compliant, or having inferences made about them in ways that are leveraged against their interests. If a healthcare provider, employer, or education institution uses a mental health and wellbeing program that provides incentives to engage with wearable devices and other forms of biometric monitoring, for example, there must be reasonable steps taken to ensure those who choose not to participate are not stigmatised, disadvantaged, or portrayed negatively as noncompliant.

Distorted Constellations in Nwando Ebizie in Science Gallery Melbourne’s MENTAL. Photo by Alan Weedon

2.6 Professional responsibility

This theme is mainly aimed at individuals and groups who are responsible for designing, developing, or deploying algorithmic and data-based technologies.³⁶⁹ The actions and understanding of these individuals and groups have a direct influence on the ethical, legal, social and political dimensions of technology being used in the mental health context.

2.6.1 Multi-disciplinary and Participatory Collaboration

The rise of algorithmic technologies has seen computer scientists, engineers and mathematicians increasingly enter healthcare service delivery, research and development. These new entrants may be unaware of the broader politics of mental health and may struggle to engage people with firsthand experience of accessing mental health services.

Some clinical researchers have acknowledged the gap of involving affected populations. For example, clinical psychologist David Mohr and colleagues noted that ‘[w]e have typically not done a good job of getting input from patients about their goals, needs, or preferences’.³⁷⁰ Mohr and colleagues discuss the harm this failure does to the efficacy of technologies that may be validated in research settings but then fail to work in real-world settings:

To address the mismatch between the ‘laboratory’ and real-life, interdisciplinary and applied empirical research is needed, including not just medical researchers, computer scientists, and those involved in service delivery, but also humanities scholars.

Most digital initiatives are concerned with complex social interventions—each occurs in a complex web of formal and informal relationships. Social scientists and humanities scholars are concerned with making sense of the interaction in social, cultural, environmental, economic and political contexts. They can also bring ideas to help guard against persistent reductionist habits in mental health sciences and technological industries,³⁷² including scrutinising claims about what algorithmic and data-driven technology can realistically tell us about people’s inner-lives. Resources from sociology, anthropology, philosophy, and so on, can help determine what role algorithmic and data-driven technologies might play in creating the social conditions that improve relations between people, including acceptance of human diversity and frailty that accounts for extreme mental states and mental health crises, distributes resources appropriately and maximises human flourishing.

2.6.2 Scientific Integrity and Testing Claims

Concerns have been raised in the literature about the rapid and even reckless embrace of algorithmic technologies in mental health research; computer and cognitive scientist Chelsea Chandler and her colleagues have described this recent flurry of commercial and research activity as akin to the ‘wild west’.³⁷³ They describe an urgent need in the field for ‘a framework with which to evaluate the complex methodology such that the process is done honestly, fairly, scientifically, and accurately’.³⁷⁴ The James Lind Alliance Priority Setting Partnership, in its survey of the field concluded that ‘the evidence base for digital mental health interventions, including the demonstration of clinical effectiveness and cost effectiveness in real-world settings, remains inadequate’³⁷⁵. Health Education England also surveyed algorithmic and data-driven technologies in mental health care and raised concerns about ‘spurious claims and overhyped technologies that fail to deliver for patients’.³⁷⁶ Perhaps most damning was a recent meta-review on mobile phone-based interventions for mental health by Simon Goldberg and colleagues, which surveyed 14 meta-analyses representing 145 randomised control trials involving 47,940 participants.³⁷⁷ Despite this extensive search and vast body of research, the review ‘failed to find convincing evidence in support of any mobile phone-based intervention on any outcome.³⁷⁸ This is not to suggest all digital technological approaches to mental health are unsupported by evidence or unworthy of further research. Instead, it is to caution against the hype and ‘techno-solutionism’ which pervades the field.

2.6.3 Against Hype and ‘Techno-solutionism’

Some technology vendors and clinical experts may presume to have algorithmic and data-driven solutions for mental health care but it is not always clear whether people impacted by psychological distress actually want or need them.³⁷⁹ The history of both mental health and computer sciences are littered with hubris and outlandish claims about scientific solutions to the longstanding and complex issue of human distress, mental health issues, and so on. For psychiatry and neuroscience, grandiose claims in the recent past include the purported discovery of ‘breakthrough’ biological or neurological treatments that will ‘revolutionise’ care.³⁸⁰ For technologists, claims include being able to ‘solve’ issues using AI from crime, corruption, pollution or obesity.³⁸¹ Elon Musk’s claim that his ‘AI-brain-chips company could “solve” autism and schizophrenia’³⁸² suggests that these traditions are beginning to merge.

Such overblown claims perpetuate a form of ‘solutionism’. Solutionism, or ‘techno-solutionism’, refers to the (flawed) belief that every social problem has a technological fix, and that simple technological fixes are possible for what are actually highly complex social issues.³⁸³ As well as perpetuating the belief that certain issues are amenable to being solved by technology, this type of over-hyping can easily lead to over-promising and under-delivering. On an individual level, one consequence could be to shape individuals’ and mental health professionals’ preferences and expectations about treatment.

When over-hyped initiatives fail, or misrepresent a ‘problem’, individuals may despair that these technological steps haven’t reduced their distress. On a macro-level, over-stating the evidence can alter how funding is directed and draw resources away from where they are needed most. This increases the possibility of technology monopolising limited resources.

There is also a risk with techno-utopian or ‘techno-optimist’ approaches that technology-driven solutions and fixes are presented as an unquestioned good. This is not to argue the opposite and reject technological approaches as wholly negative. Instead, it is to caution against the presentation of any digital initiative in mental healthcare as self-evidently virtuous. Such an altruistic and optimistic picture can shutdown important questions about the way problems are framed, and who benefits and who loses as a result.

For example, one widely-promoted idea that can be sidelined by uncritical optimism is that digital mental health initiatives are cost-effective. There may be some evidence supporting this claim from individual initiatives. However, Jacqueline Sin and colleagues examined claims about cost-savings in a systematic review of ‘entirely web-based interventions that provided screening and signposting for treatment, including self-management strategies, for people with [common mental disorders] or subthreshold symptoms’.³⁸⁴ Many interventions promised low cost of service relative to face-to-face support, which was then used to suggest it could be expanded and delivered to larger populations; yet the review identified that ‘no data were available regarding estimated cost-effectiveness and only 1 paper included economic modeling’.385³⁸⁵ Advocacy organisation Privacy International has likewise argued that there remains little evidence that AI will necessarily lead to more efficient healthcare systems, despite a widespread assumption – boosted by technology vendors – that this will be the case.386³⁸⁶

Another commonly-held view is that computational monitoring, measuring and evaluation of people necessarily affords access to knowledge about individuals, including their inner states. Instead, computational technology may well get in the way of understanding people, including the unique experience of each new person in crisis or distress who deserves to be heard fully.

The assumption that there is always or even often a technological fix for any problem is highly likely to be misplaced regarding various aspects of humane and effective responses to supporting people in severe distress, trauma, mental health crises, and so on. Hence, there is a need not only to mitigate against proven and potential harms, but also to establish sufficient standards to highlight unproven benefits that remain clouded by hype and solutionism, as noted previously.

2.6.4 Responsible Design, Including Consideration of Long-Term Effects

Technologists and mental health professionals clearly play an important role in shaping the ethical, social and political dimensions of emerging technologies in the mental health context. France’s Artificial Intelligence Strategy describe researchers, engineers and developers as ‘architects of [our] digital society’.³⁸⁷ Others have challenged the way this group of professionals has been elevated to such a lofty status, and suggest this view risks conceding undue power to data scientists, engineers and the like.³⁸⁸ Regardless, and as articulated in the Université de Montréal Declaration for a Responsible Development of Artificial Intelligence, professionals have a clear role in ‘exercis[ing] caution by anticipating, as far as possible, the adverse consequences of [algorithmic systems] by taking the appropriate measures to avoid them.³⁸⁹

Such attention is arguably missing from contemporary research in the mental health context. As noted, a survey by Piers Gooding and Timothy Kariotis on research that used algorithmic and data-driven technology in mental health initiatives, found that 85% of the studies did not appear to consider how the technologies could be appropriated in negative ways, despite some of the technologies raising serious legal and ethical issues. One possible solution to this ‘blind spot’ – at least in the scholarly field – is to require researchers to consider the long-term- and potential adverse-effects of different technologies, which could be encouraged through editorial requirements in scholarly journals, ethics/institutional review processes, and funding stipulations.

Doing Nothing with AI by Emanuel Gollob in Science Gallery Melbourne’s MENTAL. Photo by Alan Weedon.

2.7 Transparency and explainability

Transparency is defined in various ways but essentially refers to the importance of technological systems being designed and implemented so that oversight is possible. Transparency is therefore closely linked to accountability. It could extend to matters such as the data that are generated in a particular setting, the system that processes the data, and (where relevant) the business model that makes use of them.³⁹⁰ As an example, Til Wykes and Stephen Schueller have proposed a transparency governance method for apps concerning health; they raised concerns about the overselling of health apps and so-called health apps that may provide little benefit and even harm.³⁹¹ They presented the ‘Transparency for Trust (T4T) Principles of Responsible Health App’ in the form of a list of questions that can be asked to reveal key matters concerning privacy, security, feasibility, and so on.³⁹² Wykes and Schueller promote the use of the T4T principles by app stores and for presentation ‘in a simple form so that all consumers can understand them.³⁹³

The related concept of explainability refers to ‘the translation of technical concepts and decision outputs into intelligible, comprehensible formats suitable for evaluation’.³⁹⁴ Explainability seems particularly important for software that analyses large datasets algorithmically,³⁹⁵ which, again, are a minority of digital initiatives in the mental health context. Explainability is particularly crucial for systems with potential to cause harm or significantly impact individuals, such as impacting health, access to resources and quality of life. A governance example on this point is the AI in the UK: Ready, Willing and Able? report, which notes that if an AI system has a ‘substantial impact on an individual’s life’ and cannot provide ‘full and satisfactory explanation’ for decisions made, then it should not be deployed³⁹⁶—a view which few people, if any, would challenge in the mental health and disability context. Explainability is often linked to promoting nondiscrimination given that the more comprehensible a system is, the more likely discrimination, bias or error can be identified, prevented and rectified.³⁹⁷

2.7.1 Open-Source Data and Algorithms

Open-source principles promote code, data and algorithms being made freely available for possible modification and redistribution. These principles can promote a collaborative, inclusive and community-minded approach to technology development, can facilitate equal distribution of the benefits across regions and populations, and can help avoid monopolies or power concentration associated with particular technologies. Positive examples exist of open-source technologies in the mental health context, such as apps that place a premium on transparency

There are some risks with open science principles, insofar as technologies may be re-purposed for bad ends. Risks may be acute concerning biometric monitoring. Consider the following comment by academic psychiatrist, Nguine Rezaii, who was discussing her biometric monitoring research:

Some technological practices made with good intention, which may at first appear as if they should be openly accessible, could be re-purposed in unexpected and harmful ways and may need to be prevented from being publicised. For example, researchers may develop an algorithmic tool to quickly identify social media users who appear to be LGTBQI+ young people, to whom specifically-designed crisis support can be directed. Such a tool carries inherent risks, including in the event it was used by bad actors who were hostile to LGBTQI+ people. Platform regulation that adequately combats online abuse, harassment and vilification would be the clearest path to address this risk (and would carry mental health benefits more broadly). In lieu of better platform regulation, at least one option to address the tension raised by open science principles is having disclosure processes so that algorithms may be subject to validation or certification agencies that can effectively serve as auditing and accountability bodies, and in ways that respect when some algorithmic technologies should not be made open source.⁴⁰⁴

2.7.2 Other Issues of Transparency and Explainability

Other points can promote transparency in the mental health and disability context, and include:

2.8 Promotion of Public Interest and Societal Good

Most people would agree that new technologies in the mental health context should promote the public interest. Various proposals exist for guiding this aim; for example, with reference to public law values like community, freedom and equality, or general democratic values of access to information, democratic governance, civic participation, and so on.⁴¹¹ Others refer to international human rights law (which we will discuss in the next section), or internationally recognised labor rights,⁴¹² and some have pointed to broad ethical aims like ‘advancing human well-being’ as a ‘primary success criterion for development’ beyond technology simply being profitable, legal and safe.⁴¹³

Within the mental health context, broad concepts like ‘recovery-oriented support’ and ‘trauma-informed care’, which have strongly influenced mental health policy in recent years, might offer guidance for publicly-minded digital crisis support initiatives; so may guidelines for good mental health practices, such as those prepared by bodies like the World Health Organisation.⁴¹⁴

For the purposes of this report, we will briefly discuss two notable issues, which did not fit easily elsewhere in the report but which seem noteworthy. The first concerns the importance of face-to-face support and the risk of automation depersonalising care, and the second concerns the tendency of many technological approaches to home in on the individual, at the expense of more socio-economic understandings of mental health crises, distress and disability.

2.8.1 Automation, Undermining Face-to-Face Care, and the Risk of Depersonalisation

One uniformly acknowledged risk is that digitising crisis support may reduce the type of human interaction and compassion that is indispensable to providing and experiencing care, support and healing. In 2018, Christopher Hollis and colleagues conducted what appears to be the largest participatory study in the world concerned with charting a research agenda about digital technologies in mental healthcare.⁴¹⁵ 664 ‘people with lived experience of mental health problems and use of mental health services, their carers, and health-care practitioners’ in the UK were consulted. The number one research priority for participants was determining ‘the benefits and risks of delivering mental health care through technology instead of face-to-face’ and considering the impact of removing ‘face-to-face human interaction’.⁴¹⁶ Participants’ concluded that – above all – technologies that emphasised connection should be prioritised. They warned against technologies, including well-intentioned ones, that would exacerbate isolation, loneliness and alienation.

The impact of new technologies on dynamics of care will be a key discussion in coming decades, both in terms of care in health care facilities and residential homes, but also home-based care. One common aim of new technology, such as AI, is to break down tasks into individual components which can be repetitively undertaken. Yet, care is not just tasks, it is also emotion; it is a fundamental part of human relationships and it is a highly complex social interaction.⁴¹⁷> Some have claimed that technology will supersede human care. For example, a major suicide hotline in Australia claimed that a ‘virtual or robotic counsellor’ could ‘speak to lots of people and provide support to people immediately’ whereas ‘phone counselors can only speak to one person at a time.’⁴¹⁸ Others have derided the benefit of human interaction in healthcare encounters, because it is not strictly clinical in nature and wasteful of healthcare resources. Nick Weston, the chief commercial officer at Lilli, a UK company behind biometric monitoring technology that was installed in the homes of older social care recipients, rejected claims that the monitoring could exacerbate the loneliness of older people who would have otherwise received care visits.⁴¹⁹ He stated that ‘We shouldn’t be relying on home care agency staff to provide the social interaction for somebody’.⁴²⁰ Such claims rest on the narrowest conceptions of human care, flattening its complexity in the extreme.

Care has been persistently devalued in many societies. This devaluation is often based on the sexist premise that care is ‘women’s work’, given that care for older persons, persons with disabilities and children has largely been performed by women, paid and unpaid.⁴²¹ Simplistic efforts to automate care can perpetuate this devaluation. To paraphrase Fiona Jenkins, relations of care are neither adequately recognised in many paradigms of digitised mental health care, and the language of market value, nor in our inherited traditions of contemplating the values and virtues that make for a truly human life.⁴²² Such concerns highlight the potential rise of cheap (if limited) software to replace more expensive, expert, and empathetic professionals, and the disruption of care service provision and public assistance for the provision of private mental health care.⁴²³

The potential for care technologies to dehumanise and objectify care-recipients was raised by the UN Independent Expert on the enjoyment of all human rights by older persons, Rosa Kornfeld-Matte.⁴²⁴ The risk of ‘automating care’, she wrote, includes ‘losing one’s sense of identity, self-esteem and control over one’s life’.⁴²⁵ Kornfeld-Matte argued that human dignity must be ‘integrated from the conception to the application of assistive devices and robotics’.426 Even on a purely practical level, new technologies may also mean increased pressure on mental health and other service providers to increase multi-tasking and workloads in generating, inputting, organising, and constantly updating data records. Paradoxically, this extra work may reduce care workers’ time for face-to-face engagement and collaborative work with care recipients and other care workers

2.8.2 Expanding the Frame from the Individual to the Social

Most algorithmic and data-driven technology in the mental health context appears to be directed at detection and diagnosis,⁴²⁷ which draws the focus to the individual who is identified as requiring expert intervention. This dominant framing has been challenged by some commentators who call for a shift in focus away from the individual, and its associated deficit-based concern with their deviation from presumed norms, toward the social networks and relational nature of distress, mental health, disability and healing.⁴²⁸ Jonah Bossewitch, for example, has elaborated on the way technology can re-direct attention to networked collaboration, which could significantly improve the training and development of providers offering support to those in crisis.⁴²⁹ He writes:

For those with lived experience or psychosocial disability, contemporary information communication technologies can have a strong collective dimension that can be used within social movements to create a sense of solidarity, to intervene politically and to provide a sense of belonging for groups that may have traditionally been socially and economically marginalised.⁴³⁰

There are examples of successful online initiatives that appear to boost local forms of mutual support, such as the online peer support network that emerged from the NGO USPKenya (discussed in case study on X). The online support group used a mainstream messenger service and was described as being ‘fully community-based, operat[ing] outside Kenya’s mental health system and [not linked] to any mental health institution.⁴³¹ The virtual support network involves crisis support for individual members, sharing of information about face-to-face meetups, the generation of fundraising for individual members, and so on.⁴³² Such informal initiatives may not make it into the public spotlight in the same way governments, large NGOs, and industry-funded initiatives do, but they may warrant resources or further research to determine how and why they are working, and how they can be supported.

The call for a focus on relationships over individual autonomy, community over individual rights, and interdependence rather than independence, can be found in much of the literature by people with first-hand experience of mental health initiatives, particularly in low- and middle-income settings.⁴³³ This focus is also common in literature concerning

the ethics of care (sometimes referred to as relational feminist ethics)⁴³⁴ and other communitarian-oriented approaches to ethics and justice. A relational focus also aligns with growing research on the impact of social, political, and economic structures on people’s mental health, which extends to querying whether most forms of distress should even be framed in terms of ‘mental health’. As an example, consider Morgan and Kienzler’s statement on the mental health impacts of COVID-19 on populations worldwide

In low- and middle-income countries, the impact will be even more stark. Manuel Capella describes how the Ecuadorian government ‘set up the “telepsychology” phone line with one hand, whilst (in the middle of a pandemic) paying millions of dollars of foreign debt and approving reductions in the public budget with the other’, noting that ‘[s]uch cuts negatively affect the well-being – including the mental health – of the vast majority of the population’.⁴³⁶ In this way, digital practices in mental health have clear potential to reinforce individualistic views of mental health, which invisibilise social determinants and the importance of communities. This might be described as a capitalist instrumental view of mental health (for example, ‘empowering’ people to take matters into their own hands or making damaging work practices more bearable to workers rather than relying on state resources or creating fair and equitable labour conditions).

In contrast, using technology to promote holistic, human growth requires attention to the impacts of socially and economically structured disadvantage. This could even extend to querying the monopolistic, anti-competitive and surveillance-driven nature of major parts of the information economy. Regardless, current pandemic conditions have amplified historic and structural inequalities, making it even more important to consider ways to use computer technology to harness social and economic resources that individuals draw from to cope with and navigate our challenging and changing social worlds.

2.9 International Human Rights

Many people and organisations have supported international human rights law as a basis for regulating algorithmic and data-driven technologies.⁴³⁸ Although not without critics,⁴³⁹ proponents view human rights as a helpful organising framework for the design, development and use of new technologies. This includes offering factors that governments and businesses should consider in order to avoid violating human rights.⁴⁴⁰ Lorna McGregor and colleagues list some of the many human rights engaged in the growing information economy:

In each of these examples, data concerning mental health may be decisive to high stakes decisions. For example, a person may be ‘red-lighted’ in automated credit scoring systems or in social security determinations due to data generated by mental health services or inferred based on data suggesting a person is experiencing distress. The same data might be used to assess risk ascribed to a person in relation to child protection, insurance, criminal sentencing, and so on.

Human rights violations persist against people with psychiatric diagnoses and psychosocial disabilities across low-, middle- and high-income countries. In 2019, Dainius Pūras, then UN Special Rapporteur on the Right to the Highest Attainable Physical and Mental Health, commented on the ‘global failure of the status quo to address human rights violations in mental health-care systems’.⁴⁴² He argued that this failure ‘reinforces a vicious cycle of discrimination, disempowerment, coercion, social exclusion and injustice’, including in the very systems designed to ‘help’. Pūras raised a very brief concern about impact on the right to health of expanding surveillance technologies, and warned against technologies that ‘categorize an individual for commercial, political or additional surveillance purposes’.⁴⁴³ He did not elaborate on the rise of algorithmic and data-driven technologies in mental health settings in general, and indeed there is little research on the human rights implications of these developments.⁴⁴⁴

In 2022, the UN Special Rapporteur for the rights of persons with disabilities, Gerard Quinn, published a thematic study on artificial intelligence and its impact on persons with disabilities.⁴⁴⁵ The report was delivered to the 49th session of the Human Rights Council, and takes a human rights lens to the ways AI, machine learning and other algorithmic technologies can both enhance and threaten the rights of disabled people worldwide.

Some fundamental rights that are relevant here include but are not limited to: prohibition of discrimination, the right to privacy, freedom of expression, the right to health, the right to a fair trial, and the right to an effective remedy. The Convention on the Rights of Persons with Disabilities is the most relevant international human rights instrument, given its role in applying established human rights norms to the disability context, and given the strong involvement of people with first-hand experience of lived experience and psychosocial disability in its development. Relevant sections of the Convention on the Rights of Persons with Disabilities include:

There is also a potential role for data-driven and algorithmic technologies in preventive monitoring to promote and protect human rights. Preventive monitoring of closed environments, like psychiatric wards, aged care homes, and disability residential facilities, for example, can help to promote Articles 16 (freedom from exploitation, violence and abuse) and 33 (monitoring and implementation), as the following case study suggests.

More research is needed to identify the range of human rights engaged by algorithmic and data-driven technology in the mental health and disability context,450 and ways that such technologies can promote and protect, rather than threaten, human rights. Disability-inclusive research is critical to realising this aim. Research that actively involves people with disability in the development, design and implementation of technology – as well as its governance – will help to ensure technology is enabling rather than further disabling.

‘Human rights by design’ represents an emerging approach to design that ensures human rights are built into all elements of technology and AI development.⁴⁵¹ The Oxford Handbook on AI Ethics identifies four pillars to human rights by design:⁴⁵²

Human rights by design could be pursued by governments and civil society actors, including technology developers and businesses.

The UN has also developed Guiding Principles on Business and Human Rights (‘Guiding Principles’), which are relevant here given the prominent and expanding role of the private sector in generating and processing data concerning mental health and disability.453 For example, Principle 5 sets out a special duty of governments to protect against human rights abuses when they contract with private businesses. Advocacy group Access Now have drawn on this principle in calling for open government procurement, recommending that:

The International Labour Organisation has developed guidance and tool kits, as well as establishing an ILO Global Business and Disability Network,⁴⁵⁵ which may assist businesses and others to uphold the Guiding Principles.

Returning to human rights more generally, the potential benefits of adopting a human rights lens include the following:

Doing Nothing with AI by Emanuel Gollob in Science Gallery Melbourne’s MENTAL. Photo by Alan Weedon

2.10 Future Efforts

This report could have explored much more. Issues requiring further research include digitised initiatives in low and middle-income countries, the growth of ‘emotion or affect recognition’ technologies, digital care labour platforms, automated resource allocation for social security and healthcare, and human rights by design methodologies

Digital mental health technology in low and middle-resource settings and countries is growing and is often framed as allowing intervention in the lives of large populations. Emerging practices include online training for professionals and laypersons, digitisation of population-level information and electronic health records, and ‘scalable’ digital therapies such as automated cognitive behavioural therapy.⁴⁵⁶ China Mills and Eva Hillberg have argued that any analysis of risks and benefits must include questioning assumptions about digital empowerment and – as is often the case in both information technology and mental health practice – of top-down imposition by high-income (and typically ‘Western’) countries.⁴⁵⁷ Advocacy groups like TCI-Asia have warned against perpetuating legacies of colonialism in response to disabled people in low- and middle-income countries; instead, they promote the importance of involving affected populations and carefully considering the unique historical, social, cultural and economic factors of different settings.⁴⁵⁸ Failed digital health interventions in low and middle-income countries suggest that technologies must be effectively localised in order to confer power to the communities they intend to serve—which raise tensions with ‘scalability’ as an aspiration.⁴⁵⁹ Much more work is needed on the potential and pitfalls of using algorithmic and data-driven technology to address distress and disability in low- and middle-income countries.⁴⁶⁰

Computational emotion or affect recognition, was discussed briefly in this report but requires attention to its role in mental health services and research. Some speculators project market value for ‘emotional AI’ at $91.67 billion by 2024, and it is being deployed by tech vendors, criminal justice agencies, advertisers, car manufacturers, and others.⁴⁶¹ Despite the exponential growth of such technologies, a more and more research is claiming that emotion recognition technology is founded on pseudoscientific claims.⁴⁶² Indeed the traditional research on emotion recognition based on facial features has been heavily criticised as lacking evidence.⁴⁶³ The impact of broader debates about affect recognition and its interaction with biometric monitoring work conducted in the mental health sciences remains uncertain and requires attention, particularly as mental health sciences risk lending a veneer of legitimacy to otherwise pseudoscientific claims.

Digital labour platforms are transforming care labour for people with disabilities who access home-based or community-based support, and other forms of support. This has been called the ‘Uberisation’ of care and therapy. Digital labour platforms can be designed in ways that institutionalise the exploitation of care and support labour, turning the interests of recipients against (generally low-paid) staff.⁴⁶⁴ Even as such platforms might equitably distribute care and support labour, they raise legitimate concerns, particularly in relation to health and safety, insurance, unpaid work and the long-term training needs of the workforce. These platforms will raise issues of platform regulation, but also accreditation and professional regulation, given therapy platform business models often depend on shrinking payment to therapists and increasing their caseloads, and endeavouring to de-medicalise therapy to hire cheaper, non-accredited counsellors.⁴⁶⁵

Algorithmic resource allocation for determining who receives state resources for healthcare or other forms of social security, including disability-based and mental healthcare support, is an area with serious implications. Lucy Series and Luke Clements reported on automated ‘personal budget decisions’ in the UK, suggest algorithmic resource allocation could be used as a mechanism for implementing spending cuts.⁴⁶⁶ Further, the budget allocations did not always respond to people’s needs, and the algorithmic nature of the system led to a lack of transparency. Advocacy organisation AlgorithmWatch cited the research, stating that it ‘serves not only to illustrate how flawed [automated] decisions can adversely impact people’s lives, but also how [automated decision-making] systems might be scrutinised and what obstacles are sure to arise in other domains of [automated decision-making] accountability research.’⁴⁶⁷

More broadly, research is required on the impact of the politics and ideology that shape the administration of mental health services, and its merging with the politics and ideology that drive the information economy

Photo by Note Thanun on Unsplas

Conclusion

There is cause for both optimism and pessimism in the application of algorithmic and data-driven technologies to assist people in extreme distress and crises, and to boost individual and collective opportunities for crisis support and flourishing. Vigilance is required to promote benefit and prevent harm, which won’t be possible without acknowledging the vast social inequalities and profit motives that are shaping technological development in this area. As populations reckon with new digital responses to age-old experiences of distress, anguish and disability, optimism comes from these technologies and their benefits being publicly controlled, genuinely shared, and firmly shaped by those most affected.

The development of robust data governance frameworks and a rich politics concerning the use of technology in care and crisis responses won’t be possible without inclusive public engagement, enforceable policies, and global cooperation. This can only be achieved with the assertion of collective claims over data, and acknowledgement of mental health as ‘indelibly connected to systems of technology, money and power’.469 Sharing benefits requires that the public and social value of data is directed toward the determinants of human flourishing and good mental health: equitable economic development, directing support where it is needed most, addressing discriminatory practices and histories of exclusion and marginalisation, improving the quality of care and service provision, and other measures known to boost societal wellbeing.

Kind Words by Ziba Scott in Science Gallery Melbourne’s MENTAL. Photo by Alan Weedon.