AI risk scenario: Deceptive long-term replacement of the human workforce

Related to: AI vs. humanity and the lack of concrete scenariosQuestions regarding the nanotechnology-AI-risk conjunction

Objective: Some questions about a scenario related to the possibility of an advanced artificial general intelligence (short: AI) overpowering humanity. For the purpose of this post I will label the scenario a deceptive long-term replacement of the human workforce. As with all such scenarios it makes sense to take a closer look by posing certain questions about what needs to be true in order for a given scenario to work out in practice and to be better able to estimate its probability.

Deceptive long-term replacement of the human workforce:

If I could upload myself, limitlessly multiply, and run myself a hundred times faster than my human brain allows, I wouldn’t need new technological advancements like nanotechnology or even a qualitatively superior intelligence before I took over the world. I’d just make myself available (via bittorrent perhaps) and eventually irreplaceable to every business and endeavour in the world, by being a hundred times more productive and far cheaper than any individual non-menial worker (from bank accountants all the way to CEOs and diplomats) ever could. Once I’ve replaced all such professionals with myself, well, technically I’d be already in control of the world, but I could also change my visible goals without anyone having retained ability to give me significant resistance.

No new technology required.

The above description is due to Aris Katsaris (source).

Note: For the purpose of this post I will substitute artificial general intelligence for brain emulation. Assuming brain emulation instead of AI would require a different analysis as questions about the importance of prerequisite technologies such as nanotechnology would have to be posed and the difficulty of effective cooperation due to value drift would have to be examined differently.


(1) How likely is the instrumental goal of overpowering humanity to emerge in a general purpose AI (i.e. an AI design with a utility-function defined in such a way as to learn about and satisfy each customers intentions)? More on this here: AI drives vs. practical research and the lack of specific decision procedures

(2) How likely is the goal of overpowering humanity (which naturally includes the tendency to deceive humans about this intention) to go unnoticed during the research and development phase in which the AI is not yet fully developed and therefore more prone to failures (i.e. not very good at deceiving humans yet)?

(3) How likely is the AI to be modified for each purpose and customer and still retain a coherent plan of how to overpower humanity that allows it to effectively conspire with other copies without anyone noticing it?

(4) How likely are the elites, large companies and governments to trust a replacement of its workforce without demanding an inspection of the software and a thorough risk analysis, possibly constraining or even reversing such a replacement? As an example consider that China demanded Microsoft to disclose the source code of its operating system Windows (source).

(5) How likely are suspicious activities to go unnoticed with security experts, third party AI researchers, hackers or concerned customers?

(6) What happens if different customers employ their AI for purposes that are detrimental to the overall goal of overpowering humanity, such as proving the AI’s source code to be safe in a way that is verifiable by humans or inventing provably safe security protocols to protect crucial infrastructure from misuse or sabotage?

(7) How likely are people to be comfortable with replacing humans in power, such as politicians and other decision makers, with such a software?

(8) How likely is such a software to overpower humanity before other players manage to release their own general purpose AI’s as competitors, possibly constraining its influence or uncovering or thwarting its plan for world domination?

Tags: ,

  1. Alexander Gabriel’s avatar

    I just want to briefly say that the whole concept of “an” artificial general intelligence strikes me as less than realistic. Technological progress proceeds from the simple to the complex, so I would guess that modules for specific tasks are the natural unit of AI and not sharply delineated “intelligences.” A better way of thinking about AGI might then be in terms of an ecosystem of interacting modules, each basically predictable in itself but exhibiting unpredictable emergent behavior after reaching a critical mass.

Comments are now closed.