1 thinking can only be infinitesimally understood¹

1. The following questions about thinking are sometimes conceived of as ones to which satisfactory answers could be provided (in finite time)²:
   1. “What’s intelligence?”
   2. (a probably better version of the above:) “How does thinking work?”
   3. (a probably still better version of the above:) “How should one think?”
2. I think it is a mistake to think of these as finite problems³ — they are infinite.
3. We can define a “complexity class” of those endeavors which are infinite; I claim the following are some central endeavors in this class: figuring out math, understanding intelligence, and inventing all useful technologies.⁴
   1. Why are these three endeavors infinite? The next two notes will (among other things) provide an argument that if one of these endeavors is infinite, then so are the other two. And I will assert without providing much justification (here) that math is infinite.
4. What do I mean by an infinite endeavor? I could just say “an endeavor such that any progress that could be made (in finite time) [won’t be remotely satisfactory]/[won’t come remotely close to resolving the matter]”. I’d maybe rather say that an infinite endeavor is one for which after any (finite) amount of progress, the amount of progress that could still be made is greater than the amount of progress that has been made, or maybe more precisely that at any point, the quantity of “genuine novelty/challenge” which remains to be met is greater than the quantity met already.
   1. Hmm, but consider the endeavor of pressing a button very many times in some setup which forces you into mindlessness about how you are to go about getting the button pressed very many times — maybe you wake up each morning in a locked room with only a button in it, getting rewarded one util (whatever that means) for each day on which you press the button, maybe with various other constraints that make it impossible for you to make an art out of it or take any kind of rich interest in it beyond your interest in goodness-points or really think about it much at all — basically, I want to preclude scenarios which look like doing a bunch of philosophy about what it is for something to be “that button” and what it is for that button to “be pressed” and what “a day” is and developing tech/science/math/philosophy around reliably getting the button pressed and so on. Are we supposed to say that pressing the button many times is an infinite endeavor for you, because there is always more left to do than has been done already?
   2. I want to say that pressing the button many times (given the setup above) is not an infinite endeavor for you — I want to say that largely because that conflicts with my intuitive notion of an infinite endeavor and because it makes at least one thing I’d like to say later false. I might be able to get away with saying it is not infinite because it only presents a small amount of genuine novelty/challenge, saying that we should go with the last definition in the superitem (i.e., the parent item to this item on this list), but I’m sort of unhappy with the state of things here.
   3. This button example makes me consider changing the definition of an infinite endeavor to be the following: an infinite endeavor is one which cannot be finitely well-solved/finished on any meta-level — i.e., it itself cannot be remotely-well-[solved/finished], and figuring out how to go about working on it cannot be remotely-well-[solved/finished], and figuring out how to go about that cannot be remotely-well-[solved/finished] either, and so on.
      1. Actually, it should be fine to just require that the meta-[problem/endeavor] cannot be remotely-well-[solved/finished] — i.e., that there is no \([\geq \text{some positive const}]\)-satisfactory finite protocol for the object-[problem/endeavor], because I think this would imply that the object-endeavor itself cannot be remotely-well-finished (else we could have a finite protocol for it: just provide the decent finite solution) and that the higher meta-endeavors cannot be remotely-well-finished either (because finitely remotely-well-finishing a higher meta-endeavor would in particular provide a finite protocol for the object-endeavor).
5. We could speak instead of “practically infinite endeavors” — endeavors for which for any amount of progress one could ever make on the endeavor in this universe, the importance-weighted-quantity of “stuff” which is not yet figured out will be greater than the importance-weighted-quantity of “stuff” which has been figured out. That an endeavor is practically infinite is [a weaker assertion than that it is infinite], and we could retreat to those assertions throughout these notes — such claims would be sufficient to support our later assertions. Actually, we could probably even retreat to the weaker assertions still that certain endeavors couldn’t be remotely finished in (say) 1000 years of humans doing research. But going with the riskier claim seems neater.
6. If you conceive of making progress on an endeavor as collecting pieces with certain masses in \(\mathbb{R}_{\geq 0}\) and only being able to collect finitely many pieces in any amount of time, then you should imagine the sum of the masses of all pieces being infinite for an infinite endeavor.
   1. With this picture in mind, one can see that whether an endeavor is infinite depends on one’s measure — and e.g. if all you’re interested in in mathematics is finding a proof of some particular single theorem, then maybe “math” seems finite to you. For these notes, I want to get away with just saying we’re measuring progress in some intuitive way which is like what mathematicians are doing when saying math is infinite, marking the question of how to think of this measure as an important matter to resolve later. For example, maybe it would be natural to make the measure time-dependent (that is, changing with one’s current understanding), since it might be natural for what is potentially important (as progress? or for making progress?) to depend on where one is?
   2. I should clarify that there is a different conception of an infinite endeavor as one in which there is an infinite amount to do in a much weaker sense, such that there is some sort of convergence toward having finished the endeavor anyway. In this picture where one is collecting pieces, this is like there being infinitely many pieces, but with the sum of all their masses being finite, so even though one can only collect finitely many pieces by any finite time, one can see oneself as getting arbitrarily close to having solved the problem. And I want to clarify that this is not what I mean when I say an endeavor is infinite — I mean that it is much more infinite than this!
7. Generally, given some reasonable and/or simplifying assumptions, it should be equivalent to say that an infinite endeavor is one in which there is an infinite amount of cool stuff to be understood (whereas (by any finite time) one could only ever understand a finite amount). Given some assumptions, it should also be equivalent to say that an infinite endeavor is one which presents infinitely many challenges of at least some constant significance/importance (in the above picture with pieces, one should be allowed to merge pieces into one challenge to have that equivalence work out).
8. An infinite endeavor can only be infinitesimally completed/solved.
   1. If we were to make the measure time-dependent, I’d maybe instead want to say “an infinite endeavor can only be (let’s say) \([\leq \frac13 ]\)-completed/solved”, and maybe instead call it a forever-elusive/slippery/escaping question/quest/problem/endeavor. Maybe I should be more carefully distinguishing quests/questions/problems from corresponding pursuits/endeavors-to-address/solutions — I could then speak here of, say, it being an infinite endeavor to handle/address an elusive question. Anyway, a problem continuing to slip away in this sense wouldn’t imply that it is at each time only infinitesimally solved according to the measure appropriate to that same time, though it would of course still imply that the problem will always seem mostly unsolved, and it would (given some fairly reasonable assumptions) also imply that for each state of understanding, there is another (more advanced) state of understanding from the vantage point of which the fraction of progress which had been made by the earlier state is arbitrarily close to \(0\).
9. Math is infinite.
   1. I should say more about what I mean by this. For math in particular, I want to claim that always, the total worth/significance/appeal of theorems not yet proven will be greater than the total worth/significance/appeal of theorems already proven, the total worth/significance/appeal of objects not yet discovered/identified/specified/invented will be greater than the total worth/significance/appeal of objects not yet discovered/identified/specified/invented, and analogously for proof ideas and for broad organization (and maybe also some other things).
   2. Like, I think that when one looks at math now, one gets a sense that there’s so much more left to be understood than has been understood already, and not in some naive sense of there having been only finitely many propositions proved of the infinitely many provable propositions, but in a much more interesting significance-weighted sense; my claim is that it will be like this forever.⁵
   3. an objection: “Hmm, but isn’t there a finite satisfactory protocol for doing math, because something like the present state of this universe could plausibly be finitely specified and it will plausibly go on to”do math” when time-evolved satisfactorily? (And one could make something with a much smaller specification that still does as well, also.) We could plausibly even construct a Turing machine “doing roughly at least as much math” from it? So isn’t math finite according to at least one of the earlier definitions?”
      1. my response: If it does in fact get very far in math (which is plausible), it would be a forever-self-[reprogramming/reinventing] thing, a thing indefinitely inventing/discovering and employing/incorporating new understanding and new ways of thinking on roughly all levels. I wouldn’t consider it a fixed protocol (though I admit that this notion could use being made more precise).⁶
   4. It would be good to more properly justify math being infinite (which could benefit from the statement being made more precise (which would be good to do anyway)); while Note 2 and Note 3 will provide some more justification (and elaboration), I’m far from content with the justification for this claim provided in the present notes.
10. “How should one think?” is as infinite as mathematics (I’ll provide some justification for this in the next few notes), so the endeavor to understand thinking will only ever be infinitesimally finished. Much like there’s no “grand theorem/formula of mathematics” and there’s no “ultimate technology (or constellation of technologies)”, there’s no such thing as a (finite) definitive understanding of thinking — understanding how thinking works is not a problem to be solved.
    1. “How does thinking work?” should sound to us a lot like “how does [the world]/everything work?” (asked in the all-encompassing sense⁷) or “how does doing stuff work?”. I hope to impart this vibe further and to make the sense in which these should sound alike more precise with the next three notes.
11. All that said, this is very much not to say that it is crazy to work on understanding thinking. It’s perfectly sensible and important to try to understand more about thinking, just like it’s perfectly sensible and important to do math. Generally, we can draw a bunch of analogies between the character of progress in math and the character of progress in understanding thinking.
    1. In both math and investigating thinking, progress on an infinite thing can still be perfectly substantive.⁸ A mathematical work can be perfectly substantive despite being an infinitesimal fraction of all of math.
    2. It can still totally make sense to try to study problems about intelligence which relate to many aspects of it, much like it can make sense to do that in math.
    3. It can still totally make sense to prefer one research project on thinking to another — even though each is an infinitesimal fraction of the whole thing, one can still easily be much greater than the other.
    4. However, working on finding and understanding the structure of intelligence in some definitive sense is like working on finding “the grand theorem of math” or something.
12. I’m very much not advocating for quietude on an endeavor in response its infinitude. I think there are many infinite endeavors which merit great effort, and understanding thinking is one of them. In fact, understanding thinking is probably a central quest for humanity and pretty much all minds (that can get very far)!
    
13. One could object to my claim that “I will understand intelligence, the definite thing” is sorta nonsense by saying “look, I’m not trying to understand thinking-the-infinite-thing; I’m trying to understand thinking as it already exists in humans/humanity/any-mind-that’s-sorta-smart, which is surely a finite thing, and so we can hope to pretty completely understand it?”. I think this is pretty confused. I will discuss these themes in Note 4.
    1. And again, I think it is perfectly sensible and good to study intelligence-the-thing-that-already-exists-in-humans (for example, as has already been done by philosophers, logicians, AI researchers, alignment researchers, mathematicians, economists, etc.); I just think it is silly to be trying to find some grand definitive formula for it (though again, it totally makes sense to try to say broad things that touch many aspects of intelligence-the-thing-that-already-exists-in-humans, just like it makes sense to try to do something analogous in math).

onward to Note 2!