Yet another article about existential risks repeating the usual cached thoughts:
People who worry about these things often say that the main threat may come from accidents involving “dumb optimizers” – machines with rather simple goals (producing IKEA furniture, say) that figure out that they can improve their output astronomically by taking control of various resources on which we depend for our survival. Nobody expects an automated furniture factory to do philosophy. Does that make it less dangerous? (Would you bet your grandchildren’s lives on the matter?)
First of all, we are computationally bounded and cannot afford to take into account highly specific, conjunctive, non-evidence-backed speculations on possible bad outcomes. And even if that was feasible, it does not work out in practice.
Anyway, the above quote again exemplifies the dangers of jumping to conclusions. Some sort of black box full of technological magic is conjectured on the basis of which unwarranted assumptions are being inferred which are then subsequently used to draw action relevant conclusions.
To correctly estimate risks associated with artificial intelligence it is important to take into account real world research and development processes and to pinpoint specific failure modes. It is important to narrow down on how specifically an artificial intelligence is supposed to behave in a catastrophic way by taking apart the mode of operation of the magic black box and the assumptions hidden in words such as <artificial general intelligence> and <explosive recursive self-improvement>. It is important to show how specifically it is possible to arrive at such a scenario by avoiding quantum leaps in thinking about complex scenarios and to instead approach those scenarios incrementally to locate the alleged tipping-point where a well-behaved system starts to act in a catastrophic yet highly complex and intelligent way.
How many different scenarios can you come up with where an artificial intelligence causes an extinction type event if you have to do so in an incremental fashion and have to take into account the real-world research and development process leading up to such a system?
Don’t just assume vague ideas such as <explosive recursive self-improvement>, try to approach the idea in a piecewise fashion. Start out with some narrow AI such as IBM Watson or Apple’s Siri, or from scratch if you like, and add various hypothetical self-improvement capabilities, but avoid quantum leaps. Try to locate at what point those systems start acting in an unbounded fashion, possibly influencing the whole world in a catastrophic way. And if you manage to locate such a tipping-point then take it apart even further. Start over and take even smaller steps, be more specific. How exactly did your well-behaved expert system end up being an existential risk?
The purpose is to break free from recalling old conclusions made by people such as Eliezer Yudkowsky and to start thinking for yourself in a concrete and specific fashion rather than participating in furious handwaving.
At one point software is going to write new, unique and better software all by itself. But that will not happen overnight. There will be a complex developmental and evolutionary process leading up to that outcome. Only if you conjecture the outcome independently of its origin can you imagine a software that makes better software irrespective of human intention.