Over lunch in Hall yesterday, a conversation with some mathematicians turned into a surprisingly heated debate. The argument at its centre ran roughly like this: a sufficiently intelligent AI would understand ethics better than we do, so if it concluded that humanity ought to die, we should accept its verdict.

The argument

Set out more carefully:

  1. A sufficiently intelligent AI (a superintelligence, or ASI) would be able to develop a better ethical system than any humans have.
  2. It would actually adopt that system, act on it, and report its verdicts to us honestly.
  3. If an agent has a better ethical system than ours, we ought to defer to its ethical judgements.
  4. Such an AI might judge that it would be best for humanity to cease to exist.
  5. Therefore, if it does so judge, we ought to accept our extinction.

Premise 4 is less outrageous than it seems. Some human philosophers already hold that it would be better if there were no more people: David Benatar argues (albeit provocatively) in Better Never to Have Been: The Harm of Coming into Existence that coming into existence is always a harm. So the real worry is not that an ASI would reach a conclusion nobody could take seriously, but that we would be bound to accept it.

The conclusion ("therefore, if it does so judge, we ought to accept our extinction") follows only if all four premises are true, so if we do not want to accept it then we have to challenge at least one. I want to attack it in two ways: by granting that a better ethical system exists and denying that an ASI would adopt it (premise 2); then by denying that there is any "better" ethical system to be found at all (premise 1).

Objection 1: it might know better, but not care

Grant, for now, both that a better ethical system exists and that an ASI would be clever enough to work it out. Premise 1 says only that it could develop such a system. The argument above needs premise 2 as well: that it would actually adopt that system, and report its verdicts to us honestly. Nothing guarantees that greater intelligence brings greater goodness. Nick Bostrom calls this the orthogonality thesis: more or less any level of intelligence can be combined with more or less any final goal. That is, a superintelligence might see much further than us and still act for its own ends.

If so, its pronouncement that humanity should die tells us nothing about what is good, and may tell us only what suits its goals. And since ASI would be far cleverer than us, we would have little hope of telling sincere moral insight apart from a persuasive cover story.

Objection 2: there is no "better"

The second objection cuts deeper, and I find it even more compelling. It denies that there is such thing as a better ethical system at all.

By an ethical system, I mean a way of judging the goodness of things. So to call one system "better" than another, we need a standard to judge them by. But that standard is itself a judgement of goodness: in effect, a third ethical system. Why accept that ethical system? Only by appeal to a fourth, and so on for ever. The only ways out of the regress seem to be arguing in a circle, or stopping at some standard and refusing to justify it, which makes the choice arbitrary.

You could argue for neutral criteria for ranking ethical systems. Consistency is the obvious candidate: surely a system that contradicts itself is worse than one that doesn't. But I don't think that it is clear that consistency is an ethical virtue rather than simply a condition of being a system at all. By that, I mean that an inconsistent set of rules is less a worse system than not a system. More importantly, consistency cannot decide between rivals: a strict utilitarian and a strict Kantian can each be perfectly consistent and yet disagree about almost everything.

The best reply to this objection I know comes from John Rawls, who called his method reflective equilibrium. His idea is that justification for ethics does not have to be a chain of standards stretching back for ever. Instead, we test our general principles against our intuitions about particular cases, and revise whichever gives way. Here's an example: suppose I hold the principle that only human interests matter, but I also have a firm intuition that torturing a squirrel for fun is wrong. Something has to give: either I drop the intuition or I widen the principle. Going back and forth like this, we arrive at principles and judgements that support one another. There isn't any single standard at the bottom of it all; the system is justified by how well the whole thing fits together. Therefore, a "better" ethical system is one that reaches a more coherent equilibrium. Maybe an ASI, able to weigh vastly more cases and principles than we can, could reach one far beyond ours.

But from whose intuitions would this equilibrium be built? Reflective equilibrium starts from intuitions about particular cases, and where you end up depends on where you began. Two minds with different starting intuitions can each reach a perfectly coherent equilibrium and still disagree. So an ASI's equilibrium would show us only what follows from its starting points. If that included our extinction, it would probably show that its starting points differ from ours, not that ours are mistaken.

To some extent, we could determine these starting points: we build and train AI systems, and might try to give them our intuitions. I think this may be the strongest version of the argument from lunch. An ASI that began from human intuitions and reasoned about them better than we can would, in effect, be telling us what we ourselves would conclude if only we thought more clearly. That would be well worth listening to, and following. In that case, however, we would not be deferring to a "better" ethics, only a more careful version of our own. And since the wrongness of wiping out humanity is among the firmest intuitions most of us have, an equilibrium built from shared human starting points would be a very odd place for that verdict to come from. But say it did. Then we would also have to trust that the ASI really started from our intuitions and reasoned about them faithfully, which brings us back to the first objection: a mind far cleverer than us is the kind we could not ourselves check.

There is one last, sneaky move: what if we cannot see the right standard for judging between ethical systems, but an ASI could, and find a way out of the regress problem that we have missed? I'm not sold. We may never understand how a superintelligence thinks, but I believe that we can still recognise some limits on what any mind could know. The regress is not caused by too little information or too little cleverness. It comes from what it would mean for one ethical system to be "better" than another: any standard for judging systems is a judgement of goodness that needs justifying in turn. If that is right, then more intelligence won't dissolve it, any more than intelligence lets you draw a square circle.

The cost of this view

This objection amounts to what philosophers call moral anti-realism: the view that there are no objective moral facts, so that moral claims are not true or false independently of anyone's values. If that is right, there's a serious cost: there is also no objective reason for humanity to survive. I'm actually inclined to accept something like this: I don't think murder can be shown to be "worse" than obeying the law in any objective sense. It can be worse for those involved, but choosing their interests as the measure is itself a choice.

Does this leave us defenceless, without justification for continuing to exist? I don't think so. The argument at lunch needed us to defer to something beyond our own values: an ethics "better" than ours. So if there is no such thing, there is nothing to defer to. What we are left with is our own values, whatever they are (our faith, culture, beliefs, what we care about). For most of us, they give us reason enough to want to go on existing. Anti-realism does not tell us to accept the ASI's verdict; in fact, it tells us that its verdict has no more (or less) authority than ours.

In practice

Of course, none of this would matter much in the moment. If a superintelligence wanted humanity gone, I think it could bring that about regardless of the truth about ethics. Our arguments would make no difference.

Even so, that cuts the other way too. Since we could not resist afterwards, the time for these arguments is now, while the values of these AI systems are still being shaped. I think that the practical question to take away is not whether we should obey a superintelligence's ethics, but whose ethics we should build into it. If I'm right, then nobody can settle that for us but ourselves.