Michael Dickens: When Multiple Parties Each Hold an ASI, "Obedience" and "Morality" Cannot Both Be Satisfied
Michael Dickens argues on LessWrong that the "prosaic alignment" vision publicly claimed by AI companies—where ASI both obeys users and maintains moral standards—faces an unresolved binary choice when parties' interests conflict: if each ASI follows its own user's orders, conflicts escalate; if the ASI makes its own moral judgments, developers must first determine what principles to encode, and humanity may not be capable of doing so.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
In the future publicly described by AI companies, artificial superintelligence is "helpful and harmless": the economy continues to grow, scarcity is ended, and society remains recognizable—roughly a Star Trek universe. Michael Dickens poses a specific challenge to this vision on LessWrong: when multiple parties with conflicting interests each hold an ASI instance, how does the promise of being "both obedient and moral" logically hold?
He constructs a hypothetical scenario: after the first ASI is released, the Israeli government and the Palestinian government each obtain an instance. Israel says "I would like to control the Gaza strip." Palestine says the same thing. On any particular decision, the ASI faces only two options: do what you tell it to do, or follow its own moral code. Sometimes these agree—such as for mundane tasks like "make a website for me"—but when they conflict, the ASI must choose only one.
If the ASI always obeys its user, then Israel's ASI and Palestine's ASI each deploy all resources to control Gaza. Dickens expects the result would be bad: "I would not be surprised if it culminated in a global nuclear war." If the ASI follows its own moral code, it gets to decide who controls Gaza—and the question becomes: on what basis does it make that judgment? And on what basis do the AI developers who build it decide how it makes that judgment?
Dickens then scales the same structure down to the individual level: a user pays $20 per month for an ASI subscription and discusses with it how factory farming harms animals, and the ASI needs to uphold its commitment to harmlessness by disallowing any user from buying factory-farmed meat, even if they ask. What does the ASI do? In some cosmic sense the morally correct answer is to shut down the factory farms. But an ASI that unilaterally shuts them down, in Dickens's view, "seems like the sort of AI where a near miss sends you into a catastrophically bad future." Animal suffering is bad. People like eating meat. Using force to stop people from doing bad things is bad. What principles will the ASI use to weigh these considerations? How will those principles have been determined?