I basic used `:defer t` for a while (Doom Emacs lazy loaded under the hood), but when I switched to my own config I decided to just run it in daemon mode and use `:demand t` on everything. I didn't like pauses sometimes when opening a new file while it loaded a package.
Actually, I just checked my init time with that command, and it's ~2.5 seconds. I wouldn't call my config heavyweight, but it does have evil, lsp, vterm, ivy, magic, etc. I am using the native-comp branch though.
FWIW, my Emacs init time (run `emacs-init-time`) is between 2–7 seconds, and I'm not doing anything particularly special to get that working.
I'd start out with a fresh .emacs config file and then add the packages you want with the excellent `use-package` macro, setting `:defer t` as much as possible. (Often this will be implicit if you have `:mode`, `:hook`, or `:bind` configured, if I'm not mistaken.)
It would be better if Emacs were less sensitive to how your config is setup, but alas, we live in a fallen world. You can also ask around on r/emacs and get some better tips than mine.