2.5 Human control of technology

Important decisions concerning mental health that are made with algorithmic technology must be subject to human control and review. Human control over technology is key to other themes discussed in this report, including maintaining safety and security, accountability, transparency, equity and non-discrimination.

The advance of algorithmic and data-driven technologies is often presented as inevitable, with thought leaders in computer science often depicting automation as a force of nature propelled by unstoppable technological change.³⁵⁷ This view risks delegating autonomy and responsibility for such systems away from humans to some higher and abstract authority. Even terms like ‘artificial intelligence’ can imply an external intelligence that is somehow ‘out there’. Relinquishing accountability in this way is sometimes described as the ‘substitution effect’,³⁵⁸ an effect which makes it harder to ensure human control over technologies by those who are actually designing and implementing them—and those who are using and/or subject to them.

Examples exist of civil society groups successfully challenging the introduction of some types of AI, such as facial recognition. Such groups have insisted that these technologies are not inevitable and demonstrated the power of public input to change the direction of, and even halt, the use of certain technologies. Similar examples are emerging in the mental health context (see Section 1.5).

Emphasising the importance of human control in the mental health context is important given the risk that algorithmic technologies like AI become a ‘substitute decision-maker’ over both professionals and those receiving care and support, which will impact individuals’ agency and autonomy.³⁵⁹ One ethical risk in professional decision support technology in healthcare, is that clinicians defer to algorithmic technology suggestions even in the face of a contrary opinion. An overreliance on automated systems may therefore displace human agency, moral responsibility, liability and other forms of accountability. Even the use of ‘algorithm’ to describe a decision-making system has been characterised by some advocates as ‘often a way to deflect accountability for human decisions’.³⁶⁰

More broadly, there is a risk of normalising and accepting technologies which reinforce potentially inaccurate, unhelpful or harmful categorisations of people’s mental states (as discussed in the biometric monitoring section of this report).

This case study raises questions about the appropriateness (and lawfulness) of certain claims being made about individuals, including whether it is appropriate to claim that behavioural sensing can ‘reveal’ an underlying mental state or diagnosable disorder through ‘silent digital biomarkers’. The claim rests on a presumption about the capacity of computer technology, combined with psychometrics, to capture reality. Even the common claim that various technologies ‘collect’ data suggests that there is some neutral, objective information-gathering process underway. Instead, it seems more accurate to say that new forms of data concerning mental health are being created and generated. This creation and generation is not value-neutral—it is value-laden and often rests on multiple social and political claims that may be unstated. The nature of this data-generation, the meaning given to different types of data, and the often uncritical presentation of these methods as neutral forms of data ‘collection’ warrants critical scrutiny—whether in technology sales materials, media, government policy documents, or in leading scholarly journals.

Another risk is that the push to generate ever more data to improve algorithmic and datadriven solutions can undermine human action and control by distracting from the need to change existing mental health services, policies and practices. An extraordinary amount of energy can go into efforts to generate, curate, store and use data. This process can undermine action on information that already exists, particularly information highlighting existing problems of resourcing, discrimination, coercive practices, and other prominent issues in the politics of mental health.³⁶³