Superintelligence

This review was written in May of 2019

Superintelligence is a good, but annoying book. First, it must be said that their are two samples of readers--- those who had absorbed most of it by osmosis in a social network or by associated/preceding papers before reading it, and those who encountered it fresh. I know nothing of what the experience would be like to the latter group, but I'm glad I finally read it.

Because before I read it, wasn't grokking the ideas as well as I am now. It's a dense book, the occasional passage which demands a full day by itself, and one that simmers on your mind-- providing insights months after you read it rather than while you read it.

Look, intellectually I come from that whole "singularity is fake, the real problem is algorithmic bias/capitalism/government" thing, so I think I can speak to you if such a take describes you: it is less about showing that superintelligence is a problem, it is much stronger to point out that we can't show that it isn't a problem. Bostrom elaborates on all the ways in which singularity isn't species-threatening, and argues that those outcomes are tiny dots in a very large space, and that none of the other dots in the space look very good.

But there are a ton of inference steps even between accepting the scalar-centric arguments--- more on that below --- and the full-blown mindset that believes alignment foundations research is of equal or greater importance when compared to the other top causes. Frankly, I'm suspicious of anyone who believes it without a lot of feet-dragging, but this isn't fair to people who dragged their feet a lot before I met them or have the ability to drag their feet faster than I can drag mine.

What gets lost, and this is a great misfortune, is plain old longtermism. Longtermism is a consequentialism showing that while you shouldn't apply punishing discount rates to the lives of people who haven't been born yet, almost every nonzero discount rate shows that the future is more valuable than the present, so it barely even matters what the discount rate is. I've recently started consuming Bostrom's older papers, from way before this book, and it occurred to me: What if Bostrom shouldn't have gotten famous for superintelligence when longtermism is his more substantial contribution, and superintelligence is just one book you might build on top of the longtermist framework (i.e. one cause area that takes an exceptional hit to civilizational expected value)?

This question is quite sad to me, because I actually think the reason people reject alignment as a philosophically or consequentialistly coherent and useful field of study is because they haven't grokked longtermism, and I wish that people would stop arguing on the object level and focus on the metalevel, which is to say, for more people to be converted to longtermism and let their AI opinions just stew on the backburner. A longtermist reasoning about which cause areas might be worse than AGI is infinitely better than a non-longtermist reasoning about anything.

Lastly, there's on embarrassingly basic thing that I haven't really grokked yet: scalars. Perhaps you have seen the chart that says "an MNIST classifier has n neurons; there is a k where ant intelligence has kn neurons; there is a K where human intelligence has Kkn neurons"-- this doesn't seem very sophisticated, so why is Bostrom doing something similar? Perhaps it works on certain abstraction levels and not others (indeed, a whole book review could be written about the role of compression in the reader's suspension of disbelief!!!!!!). Perhaps what we are meant to interpret is "if the scalars provide the fuel, different architectures can approximate or surpass certain intelligence levels in infinitely various qualitative ways", but I couldn't help but wish I was reading a book about that part he glazed (or steamrolled) over-- the qualitative. I know he didn't write that book because he didn't want to write scifi about what the next 10 years of research might look like, but if it's not clear I must tell you: the book is quant as all heck, it's about specific datapoints, from the number of asteroids between here and sirius to postmalthusian population models. Philosophically, I just don't understand complexity at all well enough to know; who is more vulnerable to mythmaking about "emergence"-- the one who says all is scalars and nothing else, or the one who says there must be some nonscalar factor in here somewhere? That is ambiguous to me.