Maarten Boudry, a philosopher and independent author writing on human progress and human (ir)rationality, via Seeds of Science:

‘In the debate about catastrophic AI risk, surprisingly little attention is paid to evolution by natural selection. Even when imagining scenarios in which humanity is subjugated by selfish, dominant AIs, most commentators ignore evolution and tend to focus on other arguments to reach that conclusion, such as “instrumental convergence”. On this view, self-preservation and a drive for dominance might come along for the ride once systems reach a sufficient level of intelligence, because such traits are conducive to achieving almost any goal—captured in Stuart Russell’s slogan, “You can’t fetch coffee if you’re dead.”

But this neglect of evolution is strange, because Darwinian selection is a proven mechanism for producing selfish and aggressive creatures, and it is known to be perfectly substrate-neutral: it works in silico just as much as in carbon. By comparison, as I argued in my last post, the argument from instrumental convergence strikes me as weak and speculative.

Last year I published a paper about evolutionary scenarios of AI doom in Philosophical Studies together with the philosopher Simon Friederich. This was largely in response to a seminal paper by the AI safety researcher Dan Hendrycks, with the disturbing title “Natural Selection Favors AI over Humans”. Now three heavyweights in evolutionary biology and AI are weighing in with an important new paper in Proceedings of the National Academy of Sciences (for a good discussion of the paper, see Rob Brooks’s piece here)…’