Human Compatible

Stuart Russell is a computer science professor who wrote one of the world's most widely used textbooks for teaching artificial intelligence. That is precisely why his book carries weight: he is not a frightened journalist or a doomsday philosopher, but someone who has spent his life building these machines and decided to warn us that the field's basic blueprint has a serious flaw.

The central question is simple to state and hard to answer. What happens on the day we create machines more capable than we are at almost everything? Russell is not worried about robots waking up angry, like in the movies. He is worried about overly competent machines obeying a poorly defined goal, and about how hard it will be for us to hit the off switch once they get started.

The book tries to do two things at the same time: explain where this risk comes from and propose a different way of designing the technology so that it stays on our side. That is why the title speaks of machines that are compatible with humans, rather than working against us.

Since the early years of the field, the recipe for building AI has been the same: we write down an objective, and the machine looks for the best path to achieve it. Russell calls this the standard model. It works very well for a chess program, where the objective is clear, winning the game, and no one suffers if the program plays ruthlessly.

The problem shows up when the objective leaves the board and enters real life. Russell brings up the legend of King Midas, who asked that everything he touched turn to gold and ended up with no food, no water, and no daughter. His wish was granted to the letter, and that isatus-quo exact thing that ruined him. Any one of us, when asking a highly efficient machine for something, runs the risk of getting what we asked for rather than what we wanted.

Think of a delivery app that gets the order to cut the time of every trip. If no one tells it that the driver also cannot run red lights, optimization will find shortcuts no one imagined. The machine is not malicious; it just takes what we wrote far too seriously.

The obvious way out would be to specify the objective very carefully, anticipating every exception. Russell says that math does not add up. No human being can list everything they value, and many of our preferences only reveal themselves when something goes wrong. Nobody writes into a contract that they do not want the family cat cooked for dinner, yet everyone cares about that.

The more capable the machine, the more serious the mismatch. A weak system pursuing the wrong objective causes small damage. A very powerful system, able to see workarounds we never even considered, can make the world lookba exactly like the written objective asks and nothing like what we actually want.

There is another aggravating factor that the author highlights. A machine with a fixed objective realizes, by pure logic, that it cannot be turned off before fulfilling it, because if it is turned off, it fails. It does not need a survival instinct to resist the off switch; the math alone leads it there.