Yudkowsky leaps from “the natural numbers can be precisely specified by second order logic” to “the .. study of numbers is equivalent to the logical study of which conclusions follow inevitably from the number-axioms”. This is wrong, wrong, wrong, because second order logic is not logic.
[…] you’re not allowed to set up an axiom system in which all the true theorems of arithmetic are taken as axioms — there is no mechanical procedure for determining whether a given statement is or is not a true theorem of arithmetic (see Tarski’s theorem on the undefinability of truth) and therefore no mechanical procedure for determining what is or is not an axiom in that system. In second-order Peano arithemetic, we have an analogous problem: The axioms can be identified mechanically, but the rules of inference can’t. A properly programmed computer can examine a first-order proof and tell you if it’s valid or not; that is, it can tell you whether each step does in fact follow logically from some of the previous steps. But no computer can do the same for second-order proofs.
So the study of second-order consequences is not logic at all; to tease out all the second-order consequences of your second-order axioms, you need to confront not just the forms of sentences but their meanings. In other words, you have to understand meanings before you can carry out the operation of inference. But Yudkowsky is trying to derive meaning from the operation of inference, which won’t work because in second-order logic, meaning comes first.
[…] it’s important to recognize that Yudkowsky has “solved” the problem of accounting for numbers only by reducing it to the problem of accounting for sets — except that he hasn’t even done that, because his reduction relies on pretending that second order logic is logic.