Related to: AI vs. humanity and the lack of concrete scenarios, Questions regarding the nanotechnology-AI-risk conjunction, AI risk scenario: Deceptive long-term replacement of the human workforce, AI drives vs. practical research and the lack of specific decision procedures
Objective: Some remarks and questions about a scenario outlined by Mitchell Porter (source) on how an existential risk scenario involving advanced artificial general intelligence (short: AI) might be caused by a small but powerful network of organizations working for a great power in the interest of national security.
Mitchell Porter’s Elite Cabal:
…if we are interested in likely concrete scenarios, we should be considering something like: national-security elite of great power “X” have access to AI breakthroughs taken from the civilian world and then pushed over the edge by well-funded covert computer scientists. So the feedback loop of self-enhancement is not occurring solely within one single self-modifying program, but within a small but powerful network of organizations, whose value system is the “national interest” of one country…
Assumptions: I assume that we are still talking about an eventual technological existential risk scenario where some sort of artificially intelligent agency plays a role. Given that aforementioned assumption, the cabal behind this scenario must (1) be coherent enough to grasp the power of such a technology, in order for it to be funded (2) be smart enough to put all the pieces together (3) fail to notice that something that they believe to be very powerful could be very dangerous (4) succeed at creating such a powerful technology (5) fail in such a way that, in order to be powerful, the technology works perfectly well along a huge number of dimensions yet fails in such a way that it ends up deceiving and overpowering humanity.
Remarks: I like to analogize such a scenario to the creation of a generally intelligent autonomous car that works perfectly well at not destroying itself in a crash but which somehow manages to maximize the number of people to run over.
The failure mode, the mistake, would have to be selectively enough to only influence one or a few dimensions of how such an artificial general intelligence is supposed to work, causing it to fail in a highly complex, intelligent, rational yet catastrophically destructive way, while being indiscernible during the research and development process, i.e. before reaching the ability to influence the world in such a way.
For an artificial general intelligence to constitute a risk as a result of unintended consequences those unintended consequences would have to have no, or little, negative influence on the huge number of intended consequences that are necessary for it to be able to overpower humanity.
Now some people might object that the specific failure mode will be the emergence of certain instrumental goals, given a wide range of terminal goals, that are responsible for an artificial general intelligence to fail in a catastrophic way while an expert system would solve similar goals as intended or fail completely.
(1) How likely is the conjunction of having a group of people who is smart enough to create an AI that is capable of taking over the world but who however fails to predict the possible emergence of such instrumental drives given that such drives have already been predicted by people who were not capable of creating such an AI? Consider that such a group would also have to be highly rational because a powerful AI would itself have to be equipped with a good formalization of epistemic and instrumental rationality to be powerful in the first place.
(2) How would the AI initially manage to hide any suspicious signs of working against the intentions of its creators given that during its initial stages it will either still be a sub-human intelligence or lack certain skills?
(3) If such a cabal acts in the interest of national security, how likely are they to ignore possible risks associated with such a technology or fail to take preemptive security measures? Consider that great powers have a lot of practice from dealing with other dangers such as biological weapons.
(4) How likely is it that a group funded by the government in the interest of national security would not be highly suspicious of any data traffic or other actions that they are unable to explain?
For other related questions see the previous posts linked to above or see the points outlined here.