The two that bothered me most (and still do) are stroke and death. There are other serious things that can go wrong, but if their effects are temporary, then for me that puts them in a different league from a stroke, which could end my productive life, and death, which would end my life altogether.
The risk of death is put at one in a thousand, and this is where things get interesting. How worried should I be about a 0.1% risk? How do I even think about that question? Perhaps if my life expectancy from now on is around 30 years, I should think of this as an expected loss of 30/1000 years, or about 10 days. That doesn’t sound too bad — about as bad as having a particularly nasty attack of flu. But is it right to think about it in terms of expectations? I feel that the distribution is important: I would rather have a guaranteed loss of ten days than a 1/1000 chance of losing 30 years.
In the end, what convinced me that I shouldn’t worry too much about this risk was looking up what the risk of death is anyway over, say, the next year. I found on this site that the average risk of death in the UK for a man between 45 and 54 is 1/279, much higher than 1/1000. So if I am worried about a 1/1000 mortality rate from an operation, I should be about as worried that I will die from some other cause over the next four months or so. And yet I don’t lose any sleep over that possibility.
But maybe the problem is that I am concentrating four months’ risk into a few hours. Doesn’t that change everything?
This is a perfect example of the kind of scenarios that I would love to see being dissolved by lesswrong.com. I’d love to learn how to rationally handle such real world situations, rather than how to think about hypothetical distant superintelligences…