However, until one proposes a concrete methodology for modeling the world which includes the method that is doing the modeling (self-awareness), a decision making process, and a way to take action autonomously and without intervention, it's very hard to talk generally about preventing catastrophic bugs.