8 making an AI which is broadly smarter than humanity would be most significant

  1. The previous notes might leave the impression that I think the future of thinking looks like business as usual. The point of this note is to clarify some ways in which I very much do not think this.
  2. Yes, I think it is unlikely that “the history/development of thought will end”1, but I very much don’t think that the future just brings more of the usual.
  3. First, what I’ve been saying is “in many ways, the future will be like it has been before — that is, it brings completely crazy things which cannot be remotely-well-anticipated, like arithmetic, the printing press, probability, formal logic, computers, or maybe even like primitive compositional language2, very many times”, which is hardly well-captured by “the future just brings more of the usual” alone.
  4. Second, these notes are written during a period with a high probability per year of humanity making an artificial system more intelligent/capable than us3. That would very much not be “business as usual” — I think it would plausibly be comparably important to evolution or culture getting started, given the central role that intelligence/capability has in the world and the fact that gains in intelligence and understanding will likely be much faster (by default) [once an artifact smarter than humans/humanity has been created] than they have been historically in humanity/humans.
    1. One reason things will be crazy once such an artifact is created is that it makes thinking much more open to being intelligently reshaped than before (including just scaling a mind up). A great deal of somewhat intelligent design of thinking has happened already (for example, humanity is doing some of that (roughly) every time we create a new useful word-concept), but making an artifact more intelligent than us and distinct from us will almost certainly make much more powerful forms of reshaping thought available — it would open up many aspects of thought to much more potent kinds of improvement.
    2. For a very weak lower bound on what happens by default, imagine just being able to make very many “people” at least as smart as the smartest people who have appeared so far, and running them \(100\) times faster, potentially making as much scientific/mathematical/philosophical progress in a year as might have taken a century without artificial intelligence. In particular, you can have this much progress be made in the field of artificial intelligence itself. Really, things will be much crazier than 100 years of usual progress happening in 1 year.
    3. Generally, intelligence is just very powerful! Like, the structures in our world are almost entirely made by thought-like processes. When you look around, almost all the objects you see had to be invented by people or by evolution (or the two together) — if you look right, you see “reified/implemented ideas” everywhere — and there is very very much more significant stuff to invent.4 The world is only going to become much more shaped/built by thinking than it is now (assuming thinking sticks around). Your own mind is made using evolution’s “thinking” and humanity’s thinking and in particular your own thinking and your parents’/teachers’/friends’ thinking.
  5. Furthermore, by default, the intelligent system(s) we’d create would be radically different from us — this contributes further to the creation of such a system being a drastic event.
  6. So, given that by default, an artifact more intelligent/capable than us is created this century, by default, it is “the big thing” of our time.

I’d link to Note 9 here, but it hasn’t been posted yet :) (well, either that, or maybe I forgot to update this).

  1. I don’t mean anything that mysterious here — I largely just mean what I’ve already said in previous notes, though this theme and the broader them that history won’t end could be developed much further (it will be developed a bit further in later notes).↩︎

  2. Instead of ”primitive compositional language”, I originally wanted to say ”language” here, but since I don’t think language is that much of a definite single thing — I think it is a composite of many ”ideas”, and very open to new ”ideas” getting involved — I went with ”primitive compositional language”, trying to give a slightly less composite thing. But that is surely still not that unitary.↩︎

  3. or maybe multiple such systems, in parallel or in succession↩︎

  4. There are also some things made by/of other life (such as a beaver dam and a forest ecosystem) and some things made by physics (such as stars, planets, oceans, volcanoes, clouds). There’s often (almost always?) some selection/optimization process going on even to make these structures “of mere physics”.↩︎