Alexander Kruel

Related to: AI vs. humanity and the lack of concrete scenarios, Questions regarding the nanotechnology-AI-risk conjunction, AI risk scenario: Deceptive long-term replacement of the human workforce, AI drives vs. practical research and the lack of specific decision procedures

Objective: Some remarks and questions about a scenario outlined by Mitchell Porter (source) on how an existential risk scenario involving advanced artificial general intelligence (short: AI) might be caused by a small but powerful network of organizations working for a great power in the interest of national security.

Mitchell Porter’s Elite Cabal:

…if we are interested in likely concrete scenarios, we should be considering something like: national-security elite of great power “X” have access to AI breakthroughs taken from the civilian world and then pushed over the edge by well-funded covert computer scientists. So the feedback loop of self-enhancement is not occurring solely within one single self-modifying program, but within a small but powerful network of organizations, whose value system is the “national interest” of one country…

Assumptions: I assume that we are still talking about an eventual technological existential risk scenario where some sort of artificially intelligent agency plays a role. Given that aforementioned assumption, the cabal behind this scenario must (1) be coherent enough to grasp the power of such a technology, in order for it to be funded (2) be smart enough to put all the pieces together (3) fail to notice that something that they believe to be very powerful could be very dangerous (4) succeed at creating such a powerful technology (5) fail in such a way that, in order to be powerful, the technology works perfectly well along a huge number of dimensions yet fails in such a way that it ends up deceiving and overpowering humanity.

Remarks: I like to analogize such a scenario to the creation of a generally intelligent autonomous car that works perfectly well at not destroying itself in a crash but which somehow manages to maximize the number of people to run over.

The failure mode, the mistake, would have to be selectively enough to only influence one or a few dimensions of how such an artificial general intelligence is supposed to work, causing it to fail in a highly complex, intelligent, rational yet catastrophically destructive way, while being indiscernible during the research and development process, i.e. before reaching the ability to influence the world in such a way.

For an artificial general intelligence to constitute a risk as a result of unintended consequences those unintended consequences would have to have no, or little, negative influence on the huge number of intended consequences that are necessary for it to be able to overpower humanity.

Now some people might object that the specific failure mode will be the emergence of certain instrumental goals, given a wide range of terminal goals, that are responsible for an artificial general intelligence to fail in a catastrophic way while an expert system would solve similar goals as intended or fail completely.

Questions:

(1) How likely is the conjunction of having a group of people who is smart enough to create an AI that is capable of taking over the world but who however fails to predict the possible emergence of such instrumental drives given that such drives have already been predicted by people who were not capable of creating such an AI? Consider that such a group would also have to be highly rational because a powerful AI would itself have to be equipped with a good formalization of epistemic and instrumental rationality to be powerful in the first place.

(2) How would the AI initially manage to hide any suspicious signs of working against the intentions of its creators given that during its initial stages it will either still be a sub-human intelligence or lack certain skills?

(3) If such a cabal acts in the interest of national security, how likely are they to ignore possible risks associated with such a technology or fail to take preemptive security measures? Consider that great powers have a lot of practice from dealing with other dangers such as biological weapons.

(4) How likely is it that a group funded by the government in the interest of national security would not be highly suspicious of any data traffic or other actions that they are unable to explain?

For other related questions see the previous posts linked to above or see the points outlined here.

Related to: AI vs. humanity and the lack of concrete scenarios, Questions regarding the nanotechnology-AI-risk conjunction

Objective: Some questions about a scenario related to the possibility of an advanced artificial general intelligence (short: AI) overpowering humanity. For the purpose of this post I will label the scenario a deceptive long-term replacement of the human workforce. As with all such scenarios it makes sense to take a closer look by posing certain questions about what needs to be true in order for a given scenario to work out in practice and to be better able to estimate its probability.

Deceptive long-term replacement of the human workforce:

If I could upload myself, limitlessly multiply, and run myself a hundred times faster than my human brain allows, I wouldn’t need new technological advancements like nanotechnology or even a qualitatively superior intelligence before I took over the world. I’d just make myself available (via bittorrent perhaps) and eventually irreplaceable to every business and endeavour in the world, by being a hundred times more productive and far cheaper than any individual non-menial worker (from bank accountants all the way to CEOs and diplomats) ever could. Once I’ve replaced all such professionals with myself, well, technically I’d be already in control of the world, but I could also change my visible goals without anyone having retained ability to give me significant resistance.

No new technology required.

The above description is due to Aris Katsaris (source).

Note: For the purpose of this post I will substitute artificial general intelligence for brain emulation. Assuming brain emulation instead of AI would require a different analysis as questions about the importance of prerequisite technologies such as nanotechnology would have to be posed and the difficulty of effective cooperation due to value drift would have to be examined differently.

Questions:

(1) How likely is the instrumental goal of overpowering humanity to emerge in a general purpose AI (i.e. an AI design with a utility-function defined in such a way as to learn about and satisfy each customers intentions)? More on this here: AI drives vs. practical research and the lack of specific decision procedures

(2) How likely is the goal of overpowering humanity (which naturally includes the tendency to deceive humans about this intention) to go unnoticed during the research and development phase in which the AI is not yet fully developed and therefore more prone to failures (i.e. not very good at deceiving humans yet)?

(3) How likely is the AI to be modified for each purpose and customer and still retain a coherent plan of how to overpower humanity that allows it to effectively conspire with other copies without anyone noticing it?

(4) How likely are the elites, large companies and governments to trust a replacement of its workforce without demanding an inspection of the software and a thorough risk analysis, possibly constraining or even reversing such a replacement? As an example consider that China demanded Microsoft to disclose the source code of its operating system Windows (source).

(5) How likely are suspicious activities to go unnoticed with security experts, third party AI researchers, hackers or concerned customers?

(6) What happens if different customers employ their AI for purposes that are detrimental to the overall goal of overpowering humanity, such as proving the AI’s source code to be safe in a way that is verifiable by humans or inventing provably safe security protocols to protect crucial infrastructure from misuse or sabotage?

(7) How likely are people to be comfortable with replacing humans in power, such as politicians and other decision makers, with such a software?

(8) How likely is such a software to overpower humanity before other players manage to release their own general purpose AI’s as competitors, possibly constraining its influence or uncovering or thwarting its plan for world domination?

Objective: (1) Outlining how to examine the possibility of the emergence of dangerous goals in generally intelligent systems in the light of practical research and development. (2) Determining what decision procedures would cause generally intelligent systems to exhibit catastrophic side effects. 

There are arguments supporting the possibility that an advanced artificial general intelligence (short: AI) might exhibit specific universal drives which could interfere with human matters in catastrophic ways. It is for example argued that <self-protection> is important in order to achieve a wide range of goals. It is not my intention to discuss those arguments in particular but rather to look at various goals and how likely it is that different AI designs might follow decision procedures that cause them to exhibit catastrophic side effects given those goals.

To examine the possibility that a wide range of AI designs might exhibit catastrophic side effects, given a wide range of goals, several factors have to be considered. Factors such as (1) a cost benefit analyses of interfering with human matters (2) the necessity of a spatiotemporal planning horizon, given computationally limited agents, possibly limiting unbounded protectionism (3) how any given goal is interpreted given vagueness and uncertainty.

Simple and complex natural language goals such as <calculate 1+1> and <keep the trains running> should be examined to see if to expect more dangerous outcomes with more complex goals or vice versa.

Various questions should be asked to pinpoint the expected failure mode:

(1.0) How is a goal likely to be interpreted by an AI design: (1) arbitrarily (2) verbatim (3) as a problem in physics and mathematics that needs to be solved correctly?

(1.1) If an AI is interpreting a goal arbitrarily, how does it choose one interpretation over another?

(1.2) What does it mean for an AI to interpret a goal literally? Suppose the goal given is <build a hotel>. Is the terminal goal to create a hotel that is just a few nano meters in size? Is the terminal goal to create a hotel that reaches the orbit?

(1.3) If an AI design is going to interpret a goal as a problem in mathematics and physics, would it make sense to ignore various important facts about the universe such as what its creators intended it to do? Would it make sense to simply assume the most resource expensive interpretation and very likely end up doing more than necessary?

(2.0) What instrumental goals are implied by a terminal goal when interpreted by a specific AI design?

(2.1) Does a cost benefit analysis imply that it would be rational to take over the world?

(2.2) Would taking over the world, or some other far-reaching action, make sense if it is not even clear that it is instrumentally rational to allocate massive resources to do so? Does it for example make sense to build a bunker and kill all humans to make sure that you are unobstructed in calculating 1+1? Or would it make sense to turn everyone into paperclips if you are only supposed to create more paperclips than the best competitor without interfering with the world at large?

(4.0) If a specific AI design does exhibit catastrophic side effects given a goal that present day software tools can master with ease, such as the calculation of a driving route from Los Angeles to San Francisco by Google maps, is it possible to pinpoint what specifically causes that AI design to fail in such a way and how its creators did not foresee that failure mode? 

(4.1) If you were to alter a narrow AI expert system such as Google maps and incrementally turned it into a the kind of AI design that you expect to exhibit catastrophic side effects, given the same goal as the expert system, can you locate the tipping-point where on the way towards your AI design the well-behaved expert system starts to act in a catastrophic yet highly complex and intelligent way?

Example 01: 

Assume an ultra-advanced version of Google or IBM Watson.

If I was to ask such an answering machine how to prevent human suffering, would it be reasonable to assume that the top result it would return would be to kill all humans? Would any product that returns similarly wrong answers survive even the earliest research phase, let alone any market pressure?

Example 02: 

Assume an ultra-advanced version of Siri, an intelligent personal assistant and knowledge navigator which works as an application for Apple’s iOS. 

If I tell the present day version of Siri, “Set up a meeting about the sales report at 9 a.m. Thursday.”, then the correct interpretation of that natural language request is to make a calendar appointment at 9 a.m. Thursday. A wrong interpretation would be to e.g. open a webpage about meetings happening Thursday or to shutdown the iPhone.

The question here becomes at which point of technological development there will be a transition from well-behaved systems like Siri, which are able to interpret a limited amount of natural language inputs correctly, to superhuman artificial generally intelligent systems that are in principle capable of understanding any human conversation but which in contrast to their narrow AI counterparts fail in catastrophic ways.

Further reading

• Implicit constraints of practical goals
• The Fallacy of AI Drives
• Being specific about AI risks
• Taking over the world to compute 1+1
• Furniture robots as an existential risk? Beware cached thoughts!
• AI risks as a result of vagueness
• Human-UFAI Conversation

About this post: This post is supposed to be a preliminary outline of how to analyze concrete scenarios in which an advanced artificial general intelligence attempts to transform Earth in a catastrophic way.

Objective: Analyzing concrete scenarios helps to (1) better estimate the probability of possible catastrophic side effects associated with the invention of an advanced artificial general intelligence and helps to (2) design preemptive security measures.

Assumptions: For the purpose of this post I will assume an artificial general intelligence (short: AI) that is very roughly more intelligent than all of humanity and can process a greater amount of knowledge in a shorter period of time. I further assume that this agent does care about using all of the resources in the solar system for some goal unrelated to human values. Humans are considered a mere resource.

The question: Does such an AI constitute an existential risk? In other words, will such an AI cause human extinction?

This question can obviously be answered positive if such an AI is likely to achieve its goal. But how do we determine the probability of such a scenario? We have to carefully look at how such an AI could accomplish to defeat humanity (taking over the world).

We have to pay attention to a lot of factors if we want determine concrete scenarios of how an AI could overpower humanity and how probable each scenario is. Factors such as (1) the AI’s fragility to human counter strikes (2) dependency on the global infrastructure (3) ability to take control of external resources and to keep hold of those resources while remaining productive.

One of the most important question is how the advantage of greater intelligence scales with the task of taking over the world.

If we consider simple games such as Tic-tac-toe we can definitely say that superhuman intelligence would not be instrumentally useful at beating humans. You also won’t get a practical advantage by throwing more computational resources at the travelling salesman problem and other problems in the same class. The same might be said about improving a conversation in your favor by improving each sentence for thousands of years of subjective time. You will shortly hit diminishing returns. Especially if you lack the data to predict human opponents accurately.

Another example is due to Holden Karnofsky (source):

I find it somewhat helpful to analogize UFAI-human interactions to human-mosquito interactions. Humans are enormously more intelligent than mosquitoes; humans are good at predicting, manipulating, and destroying mosquitoes; humans do not value mosquitoes’ welfare; humans have other goals that mosquitoes interfere with; humans would like to see mosquitoes eradicated at least from certain parts of the planet. Yet humans haven’t accomplished such eradication…

Example scenario: Inventing new technologies to overpower humanity.

Consider that we are already at a point where we have to build billion dollar chip manufacturing facilities to run our mobile phones. We need to build huge particle accelerators to obtain new insights into the nature of reality. It takes a whole technological civilization to produce a modern smartphone.

In order to come up with new technologies an AI would somehow have to acquire large amounts of money. And even if it manages to do so, it is not easy to use the money. You can’t “just” build huge companies with fake identities, or use a straw man, to create revolutionary technologies easily. Running companies with real people takes a lot of real-world knowledge, interactions and feedback. But most importantly, it takes a lot of time. How like is it for an AI to simply create a new Intel or Apple over a few years without its creators noticing anything?

Further questions:

• What is the net advantage of eidetic memory if you consider that humans can use tools to effectively achieve the same?
• What advantage is there between humans who can extent their working memory using their tools and an AI? We can make a certain kind of psychological distinction between things we can hold in our mind without tools, and things we can’t. Does this mean there is some radical qualitative advantage (as opposed to the obvious speed advantages) in increasing the capacity of working memory? If an AI that we invented can hold a complex model in its mind, then we can also simulate such a model by making use of expert systems. Does being consciously aware of the model make a great difference in principle to what you can do with the model? If your brain had a 1000 times larger working memory, would you be better at problem solving? Probably. Would you be 1000 times better?
• What is the advantage of more serial power? Do important problems related to taking over the world fall into complexity classes where throwing more computational resources at a problem does not lead to diminishing returns? Increases in raw processing power don’t translate to proportional increases in actual utility. Your brand new PC does not improve your life twice than the PC you bought 18 months ago.
• What is the advantage of parallel computation? It is not clear how many tasks are easily decomposable into smaller operations. Consider that the U.S. has many more and smarter people than the Taliban. The bottom line is that the U.S. devotes a lot more output per man-hour to defeat a completely inferior enemy. Yet their advantage does scale sublinearly. 
• What evidence do we have that most evolutionary designs are vastly less efficient than their technological counterparts? A lot of the apparent advantages of intelligent design is a result of making questionable comparisons like between birds and rockets. We haven’t been able to design anything that is nearly as efficient as natural flight. It is true that artificial flight can overall carry more weight. But just because a train full of hard disk drives has more bandwidth than your internet connection does not imply that someone with trains full of HDD’s would be superior at data transfer.
• What is the advantage of copying? The first artificial general intelligence might be a state of the art technology which might run on state of the art hardware, rather than one AI of a huge ecosystem of different AI’s that run everywhere from smartphones to personal computers. To imagine that an AI could simply copy itself would be similar to imaging that IBM’s Blue Brain Project could simply be copied in such a way that not only nobody notices the unexpected use of bandwidth and surge up of everyone’s CPU load but that it would run effectively enough to make it worthwhile to take the risk of detection and increased instability due to using highly volatile infrastructure that was never adapted to run such a software. Further consider that a collective of humans and their tools can also think much faster than a single human being. Yet how great is the advantage? Sometimes a single human being can outsmart humanity. Yet humanity can kill a single human being. What does this indicate about the relation between (1) greater intelligence (2) faster thinking and (3) greater power?