1. Since *I’m* not familiar with GAI criticism, I wouldn’t have been able to avoid the pitfall had I wanted to.

2. Theologians *ought* to take the problem about the heavy rock more seriously. God’s ability to limit his own power is the only internally consistent conclusion to the “problem of evil.”

3. I’ve followed Less Wrong for years, intrigued, and the religion hypothesis only recently occurred to me. The “sequences” are actually what largely convinced me. I’d mostly ignored them for a long time, without considering why. Recently, I recognized they consisted of writing that is simply terrible. A rambling mess. Don’t all religions have holy books so prolix they’re painful to read; that it takes *dedication* just to read? I can’t find a better explanation for the popularity of that prolix mess called the sequences beside the Bible analogy. I think that to understand the appeal of the sequences, you have to realize that the bad writing is *part* of the appeal. How often is that pattern encountered outside a religious context? (The content of the sequences is mostly second-hand scholarship, so content doesn’t outweigh the manner of presentation.)

(I also like that you have become somewhat of an unofficial propaganda arm for my crap.)

– Stephen R. Diamond

Readers familiar with GAI criticism would have taken you more seriously if you hadn’t aired that hypothesis. It’s equivalent to asking a theologian “can God make a rock so heavy he can’t lift it?” There are informed criticisms one can make, but that particular criticism is the mark of someone stuck on the far side of the pons asinorum.

This is the kind of reaction I call “militant stupidity.”

But if you try to criticize Yudkowsky based on ideas patent in a given essay (such as the 7,000-word metaethics *summation*), the response will be that you’re not entitled to criticize unless you’ve read the entire sequence. A rather perverse concept of “intellectual honesty.” (See http://tinyurl.com/75xvp4e)

Watson, Deep Blue and other supercomputer programs simply boil down to being able to do extremely intensive searches through sets of data that we already know. The rules of Chess are very straight forward, and Googling most questions gets you close to the answer, but this may be completely different from how actual intelligent cognitive creativity works.

Until we understand how our minds come up with completely new ideas, it may be impossible to purposefully design a computer that is *actually* intelligent as measured by an ability to come up with completely new ideas that show an actual understanding of the real problem that the ineptly succinct directive “prevent human suffering” is intended to convey.

We will know we have a real artificial super-intelligence when we ask it, “what can be done to end human suffering?” and it replies with, “I give you an answer, but you’re asking the wrong question. What you really want to know how to do is…”

Evolution: I made you do that so you’d eat high value but rare fruits when they were available.

H: Yeah, but I feel like something sweet. I think I’ll have ice cream.

E: You’ll get fat, it will ruin your chances of breeding. That’s not what I made it for.

H: I had my tubes tied. Mint choc chip ice cream sounds nice.

Morality is the function the brain computes to determine what’s right, given a completely informed subject.

A motive to read the spoiler might be that the essay contains more than 7,000 words–all to say only the above. The alternative to faith in my summary is te read the last few paragraphs, which do all the work. You’ll be skipping his argument for consequentialism, but the argument makes no contribution to his conclusion.

Here’s what I don’t understand. Why do people read and follow this guy? If it’s a cult, where’s the charisma? Yudkowsky is a bad writer. Most of what he writes is second-hand scholarship on rationality without attribution. When he tries to make an original contribution, this theory of metaethics is the sort of result to expect. Why do readers wade through this prolixity to read the “sequences”?

I have an unsupported hypothesis, but it’s not completely satisfying. He’s constructed something that appeals to the religious longings of many new atheists with religious upbringing. The Singularity is god; “rationality” is ritual.

Once these systems become “intelligent”, I feel safe to classify this as “an existential risk”. In essence the combination of psychopathic people in positions of extreme power or affluence implies an existential threat trajectory – not because these people will cause existential risk themselves – but because they will order the most ruthless devices to propagate their goals and values, and will do so indiscriminately and heedlessly, much in the same manner as they already indiscriminately pollute or exploit most of the world.

I am fairly sure this is a runaway effect of concentrated power and in itself this will cause the premature and horrible deaths of “a few billion” before halfway this century.

I sure hope I turn out wrong. But considering my current age there is a fair chance myself and most readers here may come to regret this prediction, and everyone’s utter disinterest in this prediction, and similar predictions.

If this is true, what to do? Write angry letters?