Someone posted a distilled version of the argument that I tried to outline in some of my previous posts. In this post I try to refine the argument even further.
Note: In this post AI stands for artificial general intelligence.
(1) An AI will not be pulled at random from mind design space but instead be designed by humans.
(2) If an AI is meant to behave generally intelligent then it will have to work as intended or otherwise fail to be generally intelligent.
(3) A significant part of general intelligence consists of deriving exact meaning from fuzzy meaning.
(4) An AI that lacks the capacity from step 3 cannot take over the world.
(5) By step 1, what an AI is meant to do will originate from humans.
(6) If not otherwise specified, an AI will always make use of the capacity required by step 3.
(7) By step 6, an AI will not be confused about what it is meant to do.
(8) Therefore the intersection of the sets of “intelligently designed AIs” and “dangerous AIs” only contains those AIs which are deliberately designed to be dangerous by malicious humans.
 An AI is the result of a research and development process. A new generation of AIs needs to be better than other products at “Understand What Humans Mean” and “Do What Humans Mean” in order to survive the research phase and subsequent market pressure.
 When producing a chair an AI will have to either know the specifications of the chair (such as its size or the material it is supposed to be made of) or else know how to choose a specification from an otherwise infinite set of possible specifications. Given a poorly designed fitness function, or the inability to refine its fitness function, an AI will either (a) not know what to do or (b) will not be able to converge on a qualitative solution, if at all, given limited computationally resources.
 An AI can be viewed as a deterministic machine, just like a thermostat, only much more complex. An AI, just like a thermostat, will only ever do what it has been programmed to do.
 If an AI was programmed to be generally intelligent then it would have to be programmed to be selectively stupid in order fail at doing what it was meant to do while acting generally intelligent at doing what it was not meant to do.
 “The two features <all-powerful superintelligence> and <cannot handle subtle concepts like “human pleasure”> are radically incompatible.”, The Fallacy of Dumb Superintelligence
An improved version of the above argument can be found here.
- Reply to Stuart Armstrong on Dumb Superintelligence
- Smarter and smarter, then magic happens…
- Goals vs. capabilities in artificial intelligence
- To beat humans you have to define “winning”
- Implicit constraints of practical goals
- The Fallacy of Dumb Superintelligence