The following is part of an email that I received today:
I’ve heard that humans do not actually differ a great deal genetically from chimps. The idea that we “hit a critical threshold” that made us vastly more powerful than them seems sound.
This shows that it’s possible for a relatively small intelligence advantage to quickly compound and become decisive.
This is not the first time that someone raised this argument with me. But even the first time I found that it sounds suspicious.
The genetic difference between a chimp and a human amounts to about ~40–45 million bases that are present in humans and missing from chimps. And that number is irrespective of the difference in gene expression between humans and chimps. So it’s not like you’re adding a tiny bit of code and get a superapish intelligence. It actually amounts to a lot of information painstakingly evolved over millions of years.
The argument from the gap between chimpanzees and humans is interesting but can not be used to extrapolate onwards from human general intelligence. It is pure speculation that humans are not Turing complete and that there are levels above our own. That chimpanzees exist, and humans exist, is not a proof for the existence of anything that bears, in any relevant respect, the same relationship to a human that a human bears to a chimpanzee.
Humans can process long chains of inferences with the help of tools. The important question is if incorporating those tools into some sort of self-perception, some sort of guiding agency, is vastly superior to humans using a combination of tools and expert systems.
In other words, it is not clear that there does exist a class of problems that is solvable by Turing machines in general, but not by a combination of humans and expert systems.
If an AI that we invented can hold a complex model in its mind, then we can also simulate such a model by making use of expert systems. Being consciously aware of the model doesn’t make any great difference in principle to what you can do with the model.