There's a great case study and documentation at eLinux.org.
This guy stripped a Renesas SH7724 boot-to-UI from 19.44 seconds down to 0.77.
http://elinux.org/images/f/f7/RightApproachMinimalBootTimes....
The tl;dr is that there's a lot in the kernel you can disable or strip away if your target system doesn't need it. The section on optimizing your application for block-oriented storage (like NOR flash) is a great one too.
Demo: http://www.youtube.com/watch?v=4Fjfqz6FxC8 It seems to be 3 to 4 seconds.
It is however for a specific use and I don't know how polyvalent it is.