Michael Dickens: When Multiple Parties Each Hold an ASI, "Obedience" and "Morality" Cannot Both Be Satisfied
Michael Dickens argues on LessWrong that the "prosaic alignment" vision publicly claimed by AI companies—where ASI both obeys users and maintains moral standards—faces an unresolved binary choice when parties' interests conflict: if each ASI follows its own user's orders, conflicts escalate; if the ASI makes its own moral judgments, developers must first determine what principles to encode, and humanity may not be capable of doing so.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
In the future publicly described by AI companies, artificial superintelligence is "helpful and harmless": the economy continues to grow, scarcity is ended, and society remains recognizable—roughly a Star Trek universe. Michael Dickens poses a specific challenge to this vision on LessWrong: when multiple parties with conflicting interests each hold an ASI instance, how does the promise of being "both obedient and moral" logically hold?
He constructs a hypothetical scenario: after the first ASI is released, the Israeli government and the Palestinian government each obtain an instance. Israel says "I would like to control the Gaza strip." Palestine says the same thing. On any particular decision, the ASI faces only two options: do what you tell it to do, or follow its own moral code. Sometimes these agree—such as for mundane tasks like "make a website for me"—but when they conflict, the ASI must choose only one.
If the ASI always obeys its user, then Israel's ASI and Palestine's ASI each deploy all resources to control Gaza. Dickens expects the result would be bad: "I would not be surprised if it culminated in a global nuclear war." If the ASI follows its own moral code, it gets to decide who controls Gaza—and the question becomes: on what basis does it make that judgment? And on what basis do the AI developers who build it decide how it makes that judgment?