[Link] The Fallacy of Dumb Superintelligence

Link: ieet.org/index.php/IEET/more/loosemore20121128

If a computer is designed in such a way that:

(a) it has the motivation “maximize human pleasure”, but

(b) it thinks that this phrase could conceivably mean something as simplistic as “put all humans on an intravenous dopamine drip”, then

(c) what you have is NOT a computer that could ever be “all-powerful”.


For the AI to come to the conclusion that “maximize human pleasure” means that it must “consign us all to an intravenous dopamine drip”, the AI would have to be so narrow-minded as to think that maximizing human pleasure is a single-variable operation (thereby rejecting a vast swathe of human thought pertaining to the fact that “pleasure” is not, in fact, a single-variable thing at all).  Then, it would also have to believe that human pleasure is entirely consistent with forcing a human to submit to a dopamine drip against the most violent, screaming protestations that this was not wanted.  The only way that the AI could take this attitude to the concept of human pleasure would be to change the concept in such a way that it becomes entirely inconsistent with the usage prevailing in 99% of the human population (assuming that 99% of humans would scream “No!!”).

So … we are positing here an artificial intelligence that is perfectly willing to take at least one existing concept and modify it to mean something that breaks that concept’s connections to the rest of the conceptual network in the most drastic way possible.  What part of “maintaining the internal consistency of the knowledge base” don’t we understand here, folks?  What part of “from one logical contradiction, all false propositions can be proved” are we going to dump?

Further reading

Implicit Constraints of Practical Goals:

The goal “Minimize human suffering” is, on its most basic level, a problem in physics and mathematics. Ignoring various important facts about the universe, e.g. human language and values, would be simply wrong. In the same way that it would be wrong to solve the theory of everything within the scope of cartoon physics. Any process that is broken in such a way would be unable to improve itself much.

See also:

What I would like the Singularity Institute to publish

Tags: ,

  1. Robby Bensinger’s avatar

    “The goal ‘Minimize human suffering’ is, on its most basic level, a problem in physics and mathematics. Ignoring various important facts about the universe, e.g. human language and values, would be simply wrong.”

    The goal ‘maximize human suffering’ is also “a problem in physics and mathematics”. Choosing not to pursue the goal of minimizing suffering doesn’t require any particular ignorance or irrationality; it merely requires that one not be programmed to have that particular value. Otherwise, since maximizing suffering is also a mere physics problem, refusing to maximize suffering would also be ‘simply wrong’, a factual error and not just a value / dynamic. Do you see the problem here?

  2. Alexander Kruel’s avatar

    The basic argument here is that a general superintelligence will understand what is meant by ‘Minimize human suffering’. Whether it will do so is a separate issue.

    It is possible to program a superintelligence to maximize human suffering. Nobody disagrees about that. Nobody disagrees that an AI will only ever do what it has been programmed to do.

    Nobody disagrees that the a super-powerful being that pursues alien values can be an existential risk.

    Whether such a being is at all likely to exist is again a separate issue.

    The point is that intelligence in and of itself is not a risk at all. Even a narrow AI, if connected to an advanced molecular assembler, could cause human extinction.

    All of my posts should make it clear that the particular crazy idea that I am fighting is MIRI’s scenario of a dumb superintelligence with simplistic values that somehow manages to take over the world overnight. That scenario is ridiculous and badly flawed for many reasons.

Comments are now closed.