The problem with postponing optimization is that getting really good performance out of a system sometimes depends on picking a particular underlying model, which may subject your system to some unexpected constraints.

For example, who remembers DOOM? The distinctive 2-1/2 D level designs were done that way to simplify rendering, as an optimization. That's how the rendering engine could run at a remarkable framerate on the hardware of the time. If ID had started off with a full 3D rendering system and said "well, it runs at 2 FPS, lets go design some levels, we'll optimize the rendering engine later" ... well, the game would have worked out quite differently.