more

# When a Robot Becomes Judge

## Carl Miller warns about the hidden powers of algorithms

Algorithms have been used to make astronomical calculations, build clocks and turn secret information into code. But whatever they did, for millennia up until around the eighties, most algorithms were pretty simple. They took an input, followed a series of well-described steps and produced an output. Input, process, output: that’s all an algorithm is, and has ever been. Over history some have been more complicated than others, but how they turned an input to an output has generally been transparent and understandable, at least to the people who built and used them.

But at another time, in another place, I’d begun to learn that a new species of algorithm has emerged. Memory had become so much cheaper, and computational power, and data, of course, became far more plentiful. This allowed algorithms to take on a form, I learned, very different from their forebears.

Jure Leskovec spent time at Facebook and as chief scientist at Pinterest before moving back to academia. We were sitting in his office in Stanford, which, like anything associated with tech in ‘the valley’, seemed to be expanding rapidly. As we spoke, clouds of hot, white dust drifted up past his window from drilling below.

___

"So I can train a machine learning algorithm to answer the question,'If I release you, will you commit another crime or not?'"

___

Jure’s latest work was building an algorithm to help criminal court judges make better decisions over whether to grant a plaintiff bail. ‘You have a judge,’ he said, drawing a large, black rectangle on the board with a J in it, ‘and a defendant, D. The judge is trying to make a single determination: if bailed, will the defendant commit crime or no crime? So I can train a machine learning algorithm to answer the question, “If I release you, will you commit another crime or not?”’

The scribbling on the board became more profuse, as Jure excitedly sketched out the study. He’d gathered criminal records data on people who, when bailed, committed another crime, and other data on people who hadn’t. He also found a way, by comparing lenient and stricter judges, of constructing data on people who were released but who normally would have been locked up, and whether they had committed another crime too.

I had come to Jure because I was trying to understand how power was changing. Across the world, I had been meeting people like him who, I thought, could shed some light on where power really sat, what power really was, in the digital age. And here, I thought, was a confusing but important part of the puzzle. Advanced computation, machine learning, artificial intelligence, what we popularly just call ‘the algorithm’ have become an object both of fascination and fear. They’re discussed everywhere but have remained mysterious. The overall answer would eventually become a book: The Death of the Gods: The New Global Power Grab, that was published a month ago.

___

"The algorithm was constantly changing and moving as the world moved around it, and its diet of data changed to reflect that."

___

But at another time, in another place, I’d begun to learn that a new species of algorithm has emerged. Memory had become so much cheaper, and computational power, and data, of course, became far more plentiful. This allowed algorithms to take on a form, I learned, very different from their forebears.

He tapped for a few minutes and, with a sense of occasion, turned the screen to face me. ‘It’s all there.’ And there it was: a white screen with instructions neatly arranged in a series of boxes.

‘In [3]’ the first step says

‘In [8], in [9]’ says the next.

There were words in different colours, some green, some purple, some in red, in bold, in italics., spread out over about a page and half of code. It looked pretty simple. ‘It’s way more complicated than it looks,’ he said, hovering a pencil over some of the short words in square brackets. ‘These are functions. These are the building blocks of the algorithm: a sub-algorithm, basically.’ He showed how he built each of these building blocks, stringing together phrases like ‘get component’, ‘filter by station’, ‘sort_nodes_in_degree’.

Each of these sub-algorithmic functions, his building blocks, was really another complete house. Each was a complex tangle of instructions and processes, and some were themselves made from sub-algorithmic building blocks. The screen looked simple, but I was looking at a blueprint of building blocks within building blocks within building blocks: millions of instructions in just a few pages of code. Its builder was sitting next to me, but even he struggled to explain the stages, retracing steps and correcting himself as he tried to hold the layer upon layer of abstraction in his head. He seemed to finish, but then paused. ‘I don’t really remember where that last bit comes from,’ he said.

No longer simple, this new species of algorithm is ridiculously complicated. It isn’t just that the code is inscrutable, such algorithms are also constantly changing. The data inputs flow into these algorithms in real time, and the actual weights, measures and trade-offs that the algorithms make aren’t static either. Some of the functions that this researcher, like many others, had woven in used machine learning – techniques where the machine constantly learned and adapted to what the most important patterns, correlations and relationships were. It meant that the algorithm was constantly changing and moving as the world moved around it, and its diet of data changed to reflect that.

___

"Algorithms rarely stand alone. Instead, they exist within webs."

___

Within tech giants like this , algorithms rarely stand alone. Instead, they exist within webs. ‘I rely’, he said, ‘on signals that are produced by other algorithms.’ His algorithm was fed by outputs that were shaped by other algorithms. It was like a car assembly line. He, like his colleagues, worked on a small, specific part of a much larger process.

****

‘The point’, Jure said, ‘is that our algorithm outperforms human judges by 30 per cent.’ In certain data-rich contexts, especially ones which are all about patterns and trends, machine judgement will simply outperform humans. Jure’s was one example of that, but far from the only one.

‘I think there is a huge revolution to come,’ Jure continued, ‘in how decisions are made in society.’ So far, these machine learning algorithms have mostly been used in, y’know, recommendations, the online world. And I would say that these types of domains are low stakes. You might get a bad ad. You have a bad Friday night because you’ve watched the wrong movie. That’s the worst that can happen to you. But if you think about applying these algorithms to high-stakes domains —’

‘Which is increasingly happening?’ I cut in.

‘Which I think is increasingly happening. Then you have to make sure that the methods that we develop, and the standards about the way we use these methods and the way we verify them, are incredibly rigorous.’

So algorithms are being pushed into higher stakes uses precisely because their decisions can be more accurate than humans, and they can be more accurate than humans precisely because they are complex. Algorithms can handle complexity by being complex themselves. They can grasp the blinding array of factors, signals and influences that bounced off each other at every moment in ways that we simply cannot. They are able, really, to handle an unfathomably complex world better than a human can. But exactly because they can, the way they work has become unfathomable too. Inputs loop from one algorithm to the next; data presses through more instructions, more code. The complexity, dynamism, the sheer not-understandability of the algorithm means that there is a middle part – between input and output – where it is possible that no one knows exactly what they’re doing. The algorithm learns whatever it learns.

****

I had found power somewhere in those algorithms, but it was buried. Power here is buried somewhere within a process too complex for humans to grasp. Built by humans, driven by humans, only given expression and agency through humans, algorithms ultimately become powerful exactly because their analytical powers – in certain data-rich contexts – surpass humans. Like any technology, advanced computation is not an independent agency unto itself. It is how humans use it that is important. But humans have given agency, genuine decisional power, to processes that are so complex they are hidden.

Through technology, power now operates in ways that are in one way or another inhuman, and so difficult to see and to comprehend. Due to its complexity, the technology acts like a kind of camouflage, that we can’t, and don’t, look under.

Coming away from speaking to Jure and the tech whistleblower, there was an irony in all of this, I thought. It was exactly an age so swollen with data and information that had created these algorithms. They were trained precisely to handle all that information. But despite all that data, and despite the emerging smartness of machine learning, our understanding of how our lives are shaped, well, that seemed more distant than ever.

Carl Miller is the author of The Death of the Gods: The New Global Power Grab (August 2018)

Latest Releases

### The dangerous illusion of AI consciousness

##### Join the conversation

Liz Bee 2 September 2021