The thought experiment:
Jürgen is a brilliant artificial general intelligence (AGI) researcher who investigates the world from a purely mathematical point of view.
He specializes in a formal theory of fun and creativity and acquires, let us suppose, all the physical information there is to obtain about what goes on when we enjoy music, or create art, and utter sentences like ‘This is fun!’, ‘I feel great!’, ‘This music is very emotive!’, and so on.
He discovers, for example, just which musical pieces are more interesting or aesthetically rewarding than others, and exactly how this can be mechanistically described and produced by a computable algorithm that results in the creation and perception of optimally appealing art and music.
What will happen when Jürgen eventually computes the algorithm? Will he learn anything or not?
The past two posts – The value of philosophy in a universe ruled by a friendly AI and Utopia is dull – have been written in order to analyze the expected value of living in a world after a benevolent technological singularity took place.
The expected value of the event of a technological singularity itself is distinct from the value of the time between this event and the heat death of the universe. Given that a positive technological singularity could end all suffering, it is intrinsically valuable to achieve it.
Jürgen, our AGI researcher, has two options:
(1) Jürgen could decide that the discovery of a algorithm that can produce optimal fun and creativity makes any human attempt to have fun, and to be creative, futile and of no additional value.
(2) Jürgen could decide that it would be valuable to turn the whole universe into a computational substrate computing his algorithm in order to maximize fun and creativity.
With which option you agree partly depends on the answer to the thought experiment outlined above. Will Jürgen learn anything new from computing the algorithm?
(1) Given that you believe that Jürgen will not learn anything new from computing his algorithm, what difference is there between a universe that contains his algorithm, and a universe that computes the algorithm as often as possible? In other words, Jürgen solved and proved a mathematical problem. What value is there in solving and proving it over and over again?
(2) Given that you believe that Jürgen will learn something from computing his algorithm, then once his algorithm computed an optimal, or nearly optimal, result, what difference would it make to compute it N times?
Again, the question here is not about the value of discovering such an algorithm. The question is not even about computing it once. The question is about the expected value of living in a universe where such an algorithm already exists.
To clarify the above, consider a different algorithm. Let’s call it Much-Better-Life Simulator™. Running Much-Better-Life Simulator™ is equivalent to the most enjoyable life a human being could ever experience.
What difference would it make to run Much-Better-Life Simulator™ (1) once (2) N times (3) not at all? What do you estimate is the expected value of 1, 2 and 3? And how confident are you about that estimate? Can you explain what difference it makes?
More specifically, consider the value humans assign to music and art. As described in my previous posts, the value of creating music and art will be diminished by (1) the instant availability and integrability of the best possible and perceptible permutations of art and music (2) a perfectly understood, integrable and implementable mechanistic algorithm which can yield the most emotive and appealing music and art that is provably possible. In other words, anything you could ever achieve has already been achieved in the best possible way when that algorithm had been discovered.
But what about enjoying art and music? As human enjoyment is perfectly understood as well, it will be possible to generate an optimal experience of either listening to music, enjoying art, or composing and creating it. All other permutations will be provably less desirable. There will be exactly one perfect experience of either enjoying or creating music and art.
You could either integrate such an experience, as if it has already happened, or run a simulation. Afterwards you could run all less desirable permutations of it, or run it over and over again. Which raises the question of what difference it makes to have a universe in which all matter is converted in order to be excited, compared to a universe where you perfectly understand what excitement is, but choose not to compute it?