Realistic AI risk scenarios

Scenarios that I deem to be realistic, in which an artificial intelligence (AI) constitutes a catastrophic or existential risk (or worse), are mostly of the kind in which “unfriendly” humans use such AIs as tools facilitating the achievement of human goals. Whereas I believe the scenario publicized by certain AI risk advocates to be illogical and practically impossible, a scenario in which an consequentialist AI (expected utility maximizer) undergoes uncontrollable recursive self-improvement in order to e.g. turn the universe into paperclips.

Yet what some AI risk advocates imagine could partly come true, in the shape of a grey goo scenario. But such a scenario, if possible at all, would not require full-fledged general intelligence. I expect that the intelligent tools that are required to eventually create true general intelligence will be sufficient in order to solve molecular nanotechnology, and that, shortly after those tools are invented, someone will use those tools to do just that. Which makes it an existential risk that is distinct from the one that those people imagine.

But the possibility of intelligent tools, enabling humans to solve molecular nanotechnology, suggests that less intelligent tools will be sufficient to bring about other existential risk scenarios such as synthetic bioweapons.

Much to my personal dismay, even less intelligent tools will be sufficient to enable worse than extinction risks, such as a stable global tyranny. Given enough resources, narrow artificial intelligence, capable of advanced data mining, pattern recognition and of controlling huge amounts of insect sized drones (a global surveillance and intervention system), might be sufficient to implement such an eternal tyranny.

Such a dictatorship is not too unlikely, as the tools necessary to stabilize it will be necessary in order prevent the previously mentioned risks, risks that humanity will face before general intelligence becomes possible.

And if such a dictatorship cannot to established, if no party was able to capitalize a first-mover advantage, that might mean that the propagation of those tools will be slow enough to empower a lot of different parties before a particular party can overpower all others. A subsequent war, utilizing that power, could easily constitute yet another extinction scenario. But more importantly, it could give several parties enough time to reach the next level and implement even worse scenarios.

But even given that the scenario makes no sense and is unfeasible, and if less than general intelligence was not sufficient in order to bring about other existential risks, there are other ways to create artificial general intelligence. Some of those ways might be worse than anything imagined by AI risk advocates.

Neuromorphic AI, mimicking neuro-biological architectures, is one such possibility. The closer in mind design space a general intelligence is to humans, the higher is the probability that humans will suffer. As the drives and values of such agents might be similar enough to not ignore or kill humans, yet alien enough to catastrophically interfere with human values.

What can be done to prevent such negative scenarios mainly seems to be (1) research on strong and beneficial forms of government (governments which will foster and protect human values and regulate technological development) (2) research on how to eventually implement such government (3) political activism to promote awareness of risks associated with advanced technologies.

Tags: ,

  • Xagor et Xavier

    If the global tyranny is ruled by baseline humans with their hands on the levers of the AI, it might still be unstable since the ruling group could become infested with backstabbing and deception, much like the courts of the old absolutist monarchies. The narrow AI might be very good at canceling out attempts at affecting the system from the outside, but it can’t easily tell a legitimate shift in power from a coup within the ranks of power if told only to obey the entire oligarchy.

  • Alexander Gabriel

    I would agree that narrow AI could shake up the international balance of power. Insect-sized drones could make it tough to keep secrets or hide where top commanders are, making a rapid decapitation attack possible against a great power. A world government might then be forced. But I wouldn’t bet on that.

    Overall I think a competitive dystopia is more likely than any tyrannical world government, given current politics. Which would you say is more likely, I’m curious?

    The most basic problem now I think may be the destruction of human social bonds like the family. It may eventually be possible to create healthy clones of great geniuses, against whom natural children will stand no chance in competition. We might then face a choice between starting families sure to be disadvantaged and foregoing a family. This might create great difficulties for ordinary people, or even existential despair.

    Nicholas Agar gives a preliminary discussion in the last chapter of his latest book on “radical enhancement.” I think this is a topic that needs more fleshing out.

    There’s also inequality. In a posthuman world I worry this might be far nastier than inequality today, even assuming perfect enforcement of property rights and political liberalism.

  • Pingback: Alexander Kruel · MIRI/LessWrong Critiques: Index()

  • Pingback: Alexander Kruel · Probability of unfriendly and friendly AI()

  • Joe

    Genocide Man web comic explores this very issue. Although to be honest, I’d expect dangerous nanotech to find itself way behind existing biology simply because of a billion+ years of evolution. Such a gap wouldn’t survive long, obviously. From an engineering POV, it’s a mixed bag since even a simple animal cell would probably be way more complex than early computer-based programmable assemblers. That same billion years of evolution just might make it a lot easier to adjust to the changes we made to it.

  • Joe

    Think: Explosive/shock/monitoring collars.

    No need for prisons as buildings. You’d just have immediate punishment and correction. Only for the more stubborn cases that are an existential threat, would there be the death penalty (rather the entire human or just their neurological identity). Individuals would be too valuable in this cradle-to-the-grave scenario, to just go and waste. It would be more economical to precisely control the number of people in each social position and avoid training them with knowledge that is on a need-to-know basis. In pure economics terms, it would make no sense to have to retrain another person.

    Yeah, disturbing, I know, but that’s the end-game of every strong ‘dark ages’ statist theory. You eventually reach a hegemony and then reduce costs to maximize profit for the PTB.

  • kt

    One counterexample to this argument is North Korea. I think North Korea’s evolution into an amazingly stable system of seemingly unchallenged central control and extreme oppression of nearly all North Koreans demonstrates a form of existential risk that people outside of the foreign policy sphere should be taking seriously. The state control in NK is more extreme than anything found in old-style monarchies and has lasted 60 years and counting without any effective coup attempts.

  • Xagor et Xavier

    I suspect that North Korea is unusually stable for two reasons. First, it is, to steal a phrase, “too big to fail”: China in particular doesn’t want the country to explode on them. Second, the people greatly value conformity. Part of this is from the pervasive indoctrination, but part of it may also be Confucian. Consider, for example, the elaborate mythology about signs from heaven around the birth of jong-Il; also, consider that Marx’s works is restricted in NK.

    If I’m right, then North Korea can play Ingsoc because it can rely on outside assistance to keep the regime running (because it would be disastrous to them were it to fail) and because the people will put up with it to a much greater degree than would otherwise be the case (because of the partly Confucian, partly propaganda-based mindset).

    In a narrow AI dystopia, the rulers could possibly get the stability (that NK gets from outside support) by AI support, or use the AI to force the outside world to keep the regime going – say by some kind of automatic doomsday weapon. But implanting the equivalent of the Koreans’ religious background from scratch is going to be considerably more difficult… unless the narrow AI’s speciality happens to be persuasion or the regime can simply let the hostile elements die off, and reiterate enough to figuratively breed out the resistance.

    On the other hand, a sufficiently general (medium wide?) AI could provide abundance to the elite — enough to keep them from engaging in coups, while also providing the means to harshly oppress the rest of the people. But that would imply an O’Brien logic: the elite would have to consciously acknowledge that what they truly want is power, “the boot stamping on the human face forever”, and that the semi-narrow AI can give it to them. You’d have a more or less equal “pantheon” on top of a very oppressive life for the mortals.

  • kt

    Full disclosure: at this point I’m far more knowledgeable about NK than AI, although I intend for that to change soon.

    Confucianism, indoctrination into the Kim cult, and possible influences from the Japanese god-emperor tradition (see Brian Myers’ book The Cleanest Race if you’re interested in this connection) may have been the initiating factor in NK’s situation, but the disturbing part is how this situation has been able to propagate itself long after most ordinary North Koreans had lost faith in these religions. A much more important factor than indoctrination and culture (beyond, as you mentioned, the Chinese and global interest in maintaining regime stability) is airtight social engineering (stovepiped information for everyone except a tiny handful at the top, deliberate starvation of the population, and an enforced culture which arguably places a higher premium on citizen surveillance and informing on others than any society ever to exist).

    Look at the demographics of inequality in NK. Expert Adrian Wong estimates that the nation of 25 million is run (ahem, brutalized) for the sole benefit of a mere 2000 or fewer elites. It appears that NK has already achieved the social equivalent of your medium-wide AI scenario with only luck and the social engineering talents of Kim Il Sung. I think it could be done with narrow AI; again, the circumstances would have to be right, but it would seem that considering the current disbelieving yet still brutalized state of NK’s citizens, a KIS type teamed with a decent narrow AI could take the place of a KIS type paired with a susceptible cultural milieu, especially given that all the NK-style dystopia seems to need is sufficient activation energy. At least it’s a possibility worth considering, in my view

  • Xagor et Xavier

    The question seems to be how much NK depends on particular conditions that, if an AI were to reproduce them, that AI would have to be more general-purpose than narrow.

    I would guess that current North Korea is fairly stable. It may not be entirely so, but I know too little about the prevalence of gray markets, how effective Chinese imports and cellular phone connections near the border are to speculate further, so I’ll assume that it is stable. That implies that the particular governance mode that the NK state runs under is stable once it can establish itself, at least given the international too-big-to-fail support that I have mentioned earlier.

    But establishing it may be another matter. In North Korea, the circumstances were right: the North Koreans had the Confucian/god-emperor background you mentioned. Perhaps one could consider the NK regime an extreme of the sort of totalitarianism that Europe faced in 1930-1940 and then developed an immunity to – except that NK never did develop the immunity because things were so that the system reinforced itself. (There are of course weaknesses to this argument, most obviously that South Korea avoided that kind of totalitarianism.)

    Anyway, my point is that it seems to me that NK both required a very particular set of initial conditions, and still requires support from the rest of the world to work. A narrow AI might manage to stabilize an already existing North Korea to the point where outside support would not be required, but I doubt it could turn, say, Canada into a North Korea. It would either have to be a /very/ good social engineering / propagandist AI, or be rather more broad than I imagine is implied by the concept of “narrow” AI.

    There’s also the matter of whether other countries would just let a narrow AI dystopia establish itself. One could suppose that there would be some equivalent of a Soviet Union to shield the nascent NK-alike, but that makes the circumstances more particular, and the target one has to hit correspondingly harder. Maybe a techno-coup would be more likely: e.g. an intelligence agency using AI to snoop, and then going from just snooping to actively regulating.

    And perhaps there’s a more general observation that covers what you’re saying. Systems (of which government is one) stay alive by reacting to and canceling out change that could destroy them or alter them to the point where they’re no longer the same system. A narrow AI could be good at this – but along a very particular, narrow field.

    So I imagine that a narrow AI dystopia would be stabilized along that particular narrow field. Perhaps it could arise out of a cult that was voluntary in the beginning, and the AI was very good at monitoring people. Or perhaps the dystopia consists of a county-sized prison nobody can leave or enter lest the AI kill them, and life inside the prison becomes Somalia-style anarchy. The original programmers need not even be around anymore.