When considering threats posed by artificial intelligence (AI), the focus usually rests on two.
Some foresee a terrifying Skynet dystopia, with sentient computers identifying mankind as their greatest enemy and turning the world into a killbot hellscape. Others, slightly less alarmist, fear automation rendering vast swathes of the population literally redundant.
But the most likely threats come from bias and failures in trust.
AI and machine learning should result in fairer, more evidence-based decision-making, since machines are supposedly free from human biases. But as machine learning is reliant on data input, bias at this stage is not only reflected in the machine’s model, but becomes ingrained.
This is not something that might happen – we are already seeing it come to pass.
AI is only as good as the data it’s trained on. Furthermore, it is increasingly widespread across many industries, with real effects on people’s lives.
AI is being used in areas from insurance and financial services to assess risk for policies and loans, to healthcare and justice. So far, evidence of “bad” decisions is sporadic. But evidence from higher-impact applications elsewhere may not inspire confidence.
One of the most high-profile AI failures was the Microsoft chatbot Tay, launched in 2016. Designed to mimic Twitter users, Tay was shut down after just 16 hours after becoming racist, sexist, and promoting drug use.
The same year, the online “beauty pageant” Beauty.AI attracted 6,000 entrants and picked 44 winners, nearly all of whom were white, and only one of which had dark skin.
Such failures are upsetting and suggest deeper biases in society. But other failures have been more insidious.
An investigation into the US algorithm COMPAS – used in the criminal justice system to predict reoffending rates and inform parole decisions – found evidence of racial bias against minorities. PredPol, an LA-based crime prediction tool for identifying crime concentration, wrongly identified black neighbourhoods as drug hotspots, contrary to actual arrest maps.
Nor is the bias only racial. A University of Washington study found that Google’s voice recognition struggled to understand female voices, while another study found job adverts targeted men with higher-status opportunities.
Since machines are prejudice free, clearly the datasets are reflecting society’s own biases – but the decisions they make entrench it.
It is vital we work harder to get this right. If AI is increasingly relied upon for high-level decision-making, it is not enough that it makes the right decisions – it must also be trusted to do so. This is where assurance could come in.
There are challenges to traditional approaches to assurance in this area, and in ensuring algorithms do what they are meant to.
It is frustratingly not always possible to explain how an algorithm has weighted different factors to come to any single answer. To some extent, “explainability” is traded against accuracy. Algorithms used for picture recognition, for example, rely on deep learning – many layers in a neural network between input and output. The more layers, the more accurate the likely result, but the harder it is to understand.
This is an ongoing debate in the technical community; do we want greater accuracy, or to be able to understand how a result has been generated? These issues are compounded by the “learning” element – as algorithms evolve, repeatable tests may be redundant, since you are no longer testing the same thing.
Despite these issues, much can be done by developers and users of algorithms to manage risks. They can be alert to risks of bias, and proactive in identifying issues. Ethics codes for data scientists and coders are being developed.
Diversity is a key part of this – not just in terms of considering diversity issues, but also ensuring diversity in hiring processes, so coders are not all from the same background, with the same set of unconscious biases.
Controls and Quality Assurance processes – checking input data for accuracy, completeness, and integrity – should be in place, as well as similar checks on how algorithms are selected and developed.
This kind of thinking is central to accountancy. Building such processes, asking challenging questions, and providing transparent, independent assurance that models have been correctly followed and information is reliable is core to what accountants do. AI is a fantastic tool with the potential to deliver countless benefits to businesses and society. But it cannot be left solely to the technological developers.
If we are to build in ethics from the ground floor, and eliminate bias – and we surely must – it is time to exercise more professional scepticism and judgement.
If the robots are going to start taking over, let’s make sure the values they have embedded are right.