Order is the very dangerous one. If it turns out that models are actually capable of patching all the exploits, it’s hard to imagine what will disrupt the status quo. Fortunately, this seems unlikely, but that doesn’t mean order won’t prevail in the short term. At the end of my Yudkowsky debate we found a tangible point of disagreement. He thinks AI will be able to find a solution to the Prisoner’s dilemma. I know there already is a solution (it’s defect), EA people just aren’t happy about it.
If it won’t be order (and you better pray it won’t be order) that leaves chaos. AI that can play a whole society in hours can manufacture entire histories, years of archives of daily newspapers, and an Internet that never happened. The dead Internet theory is coming true before our eyes. Now I can still see the bots, soon I won’t be able to. Scratch that, I probably already often can’t.
