The Satan's Apple problem is a problem in decision theory, an attempt at a paradox. In this post I'm going to explain what's wrong with it and why it doesn't pose a real problem.
The problem has two forms. I'll explain them both here, but I'm going to focus on the simpler form and then briefly address the other one at the end.
Satan's Apple, form A: There are an infinite number of apple slices, indexed by the whole numbers. You may take any initial segment of them, including all of them. Taking n of them is worth n utility. However, taking all of them is worth -∞ utility.
Satan's Apple, form B: You are presented with an apple slice. You may take it or not; if you do, the process repeats. If you stop after having taken n slices, you get n utility. But if you take ω slices, we stop and you get -∞ utility.
Obviously these are just the same problem rephrased, so I'm going to focus on form A. Basically the idea of expressing it in form B is to make the following "paradox": At any given step, you should take the slice, yet if you consistently follow this strategy, you lose! What's going on?
Now, if you've read much of what I've written on the internet, you can probably guess that I'm going to say the problem is unbounded (and even infinite) utility values, duh! Except, no! That has nothing to do with the problem! To see this, change it so that taking n slices is worth 1-1/(n+1) rather than n, and so that taking all of them is worth -1 instead of -∞. The problem is still there!
So what is the problem, then? Well, it's easier to see if you put it in form A, looking at the whole thing at once. It's got nothing to do with infinite processes; the basic problem is really just, how do you make a decision when there's no maximum, when the supremum is not achieved? If you like, imagine a continuous version -- pick x∈[0,1], if you pick x≠1 you get x, if you pick x=1, you get -1. What do you do? Or rather, what does the VNM/Savage theory of expected utility say you do?
The answer is, it doesn't say anything! It simply isn't built to handle situations like this -- not in the sense that it gives inconsistent answers, but in the sense that is simply provides no guarantee; any answer would be consistent with it, because it only describes binary (or finitary) decisions. Really, the whole reasoning is backwards -- the theory of expected utility describes an agent's preferences, not determines them!
OK, but that's not really a satisfying answer, is it? How should one handle infinitary decisions, then? Is it really just totally unconstrained, unrelated to how one handles finitary decisions? That would be ridiculous. But is it determined by it, or is there some additional freedom?
The answer is, there's no answer to that, because there are no infinitary decisions! But wait, don't we often consider optimizing over, say, a real interval? We do! But that doesn't mean such decisions actually occur; one cannot act with infinite precision.
But wait, does that mean one should never consider such decision problems? Not exactly -- they can still be informative. When the function being optimized is continuous, then solving the continuous problem is a good guide to solving the finite-precision problem. Though not always -- imagine for instance that the function drops off steeply on one side of the maximum. If you have finite precision, is that maximum what you want to aim for?
The extreme case of this is when the function isn't continuous at all -- as here. (And yes you can apply this to the original form of the problem as well a real-valued one, considering ω as the limit of the natural numbers, although I'll admit it's maybe a bit strained.)
So, that's why I reject the Satan's Apple problem as uninformative and not a paradox.
With this it should be pretty clear what's wrong with form B, but let's treat that briefly anyway. Well, OK, first off, what's wrong with form B is that it's equivalent to form A. But let's say we ignore that and start analyzing form B from a more intuitive perspective that leads to the "paradox". So -- obviously it doesn't matter which slice we're at; our strategy should be the same regardless, right? But picking deterministically won't work. So, with what probability p do we take the slice? We find ourselves back at the same problem! The resulting expected utility is discontinuous in p.
Of course, this is considering a restricted set of strategies; in reality it's equivalent to problem A. But one way or another, the same problem reoccurs -- you can't actually make infinitary deicions; and when your continuous approximation is, well, discontinuous, it ceases to be informative.
ADDENDUM: Sorry, actually I think there's a bit more to say here...
Now I'm saying infinitary decisions don't actually happen... but what if some godlike entity actually presented you with Problem A, you just need to name the number? Then one has to account for the fact that different numbers take different amounts of time to name! (And it's a question of what forms it accepts the numbers in; if it presents it in the form of Problem B, it's effectively only accepting it in unary.) So at any time step, you still only have finitely many choices, in laying out the string that will name the number.
Moreover, here the boundedness of utility does come into play, since it means that it can't actually constantly be worth your time to take more and more slices. You only actually want to name the largest number you can when doing so is costless.
...hm, this is making me wonder if my original answer was correct. I mean, if we're considering an AI, while it doesn't technically have infinitely many choices for strategies over its lifetime, I think it has enough that infinite may be a good model. So maybe the real answer is that boundedness + time costs prevents Satan's Apple from coming up?
Um... hm. OK I'm not confident in my answer after all. But, uh, I still suspect one way or another it's not a problem? Hm, that's not a very satisfying answer... :-/
ADDENDUM AGAIN: Blargh! No, wait, my second "answer" I think is seriously flawed. It doesn't "take seriously" the boundedness of utility -- because that applies to everything else as well! Um... hm... blech. Yeah OK. I guess I'm just confused again.
-Harry
The problem has two forms. I'll explain them both here, but I'm going to focus on the simpler form and then briefly address the other one at the end.
Satan's Apple, form A: There are an infinite number of apple slices, indexed by the whole numbers. You may take any initial segment of them, including all of them. Taking n of them is worth n utility. However, taking all of them is worth -∞ utility.
Satan's Apple, form B: You are presented with an apple slice. You may take it or not; if you do, the process repeats. If you stop after having taken n slices, you get n utility. But if you take ω slices, we stop and you get -∞ utility.
Obviously these are just the same problem rephrased, so I'm going to focus on form A. Basically the idea of expressing it in form B is to make the following "paradox": At any given step, you should take the slice, yet if you consistently follow this strategy, you lose! What's going on?
Now, if you've read much of what I've written on the internet, you can probably guess that I'm going to say the problem is unbounded (and even infinite) utility values, duh! Except, no! That has nothing to do with the problem! To see this, change it so that taking n slices is worth 1-1/(n+1) rather than n, and so that taking all of them is worth -1 instead of -∞. The problem is still there!
So what is the problem, then? Well, it's easier to see if you put it in form A, looking at the whole thing at once. It's got nothing to do with infinite processes; the basic problem is really just, how do you make a decision when there's no maximum, when the supremum is not achieved? If you like, imagine a continuous version -- pick x∈[0,1], if you pick x≠1 you get x, if you pick x=1, you get -1. What do you do? Or rather, what does the VNM/Savage theory of expected utility say you do?
The answer is, it doesn't say anything! It simply isn't built to handle situations like this -- not in the sense that it gives inconsistent answers, but in the sense that is simply provides no guarantee; any answer would be consistent with it, because it only describes binary (or finitary) decisions. Really, the whole reasoning is backwards -- the theory of expected utility describes an agent's preferences, not determines them!
OK, but that's not really a satisfying answer, is it? How should one handle infinitary decisions, then? Is it really just totally unconstrained, unrelated to how one handles finitary decisions? That would be ridiculous. But is it determined by it, or is there some additional freedom?
The answer is, there's no answer to that, because there are no infinitary decisions! But wait, don't we often consider optimizing over, say, a real interval? We do! But that doesn't mean such decisions actually occur; one cannot act with infinite precision.
But wait, does that mean one should never consider such decision problems? Not exactly -- they can still be informative. When the function being optimized is continuous, then solving the continuous problem is a good guide to solving the finite-precision problem. Though not always -- imagine for instance that the function drops off steeply on one side of the maximum. If you have finite precision, is that maximum what you want to aim for?
The extreme case of this is when the function isn't continuous at all -- as here. (And yes you can apply this to the original form of the problem as well a real-valued one, considering ω as the limit of the natural numbers, although I'll admit it's maybe a bit strained.)
So, that's why I reject the Satan's Apple problem as uninformative and not a paradox.
With this it should be pretty clear what's wrong with form B, but let's treat that briefly anyway. Well, OK, first off, what's wrong with form B is that it's equivalent to form A. But let's say we ignore that and start analyzing form B from a more intuitive perspective that leads to the "paradox". So -- obviously it doesn't matter which slice we're at; our strategy should be the same regardless, right? But picking deterministically won't work. So, with what probability p do we take the slice? We find ourselves back at the same problem! The resulting expected utility is discontinuous in p.
Of course, this is considering a restricted set of strategies; in reality it's equivalent to problem A. But one way or another, the same problem reoccurs -- you can't actually make infinitary deicions; and when your continuous approximation is, well, discontinuous, it ceases to be informative.
ADDENDUM: Sorry, actually I think there's a bit more to say here...
Now I'm saying infinitary decisions don't actually happen... but what if some godlike entity actually presented you with Problem A, you just need to name the number? Then one has to account for the fact that different numbers take different amounts of time to name! (And it's a question of what forms it accepts the numbers in; if it presents it in the form of Problem B, it's effectively only accepting it in unary.) So at any time step, you still only have finitely many choices, in laying out the string that will name the number.
Moreover, here the boundedness of utility does come into play, since it means that it can't actually constantly be worth your time to take more and more slices. You only actually want to name the largest number you can when doing so is costless.
...hm, this is making me wonder if my original answer was correct. I mean, if we're considering an AI, while it doesn't technically have infinitely many choices for strategies over its lifetime, I think it has enough that infinite may be a good model. So maybe the real answer is that boundedness + time costs prevents Satan's Apple from coming up?
Um... hm. OK I'm not confident in my answer after all. But, uh, I still suspect one way or another it's not a problem? Hm, that's not a very satisfying answer... :-/
ADDENDUM AGAIN: Blargh! No, wait, my second "answer" I think is seriously flawed. It doesn't "take seriously" the boundedness of utility -- because that applies to everything else as well! Um... hm... blech. Yeah OK. I guess I'm just confused again.
-Harry
