Funny, coming back to this, if I recompile without native compilation and any other flags, I get faster startups. By upwards of 1 second. (So, from almost 2 seconds to almost 1 second.)

An emacs -q startup is roughly the same. And I maintain that many things feel snappier with the optimization flags.