On my i5-3550 with 16 GB of RAM and an SSD it takes a couple seconds to start the first time and less than a second for subsequent times.
Both machines are running Windows 10.
Right now, the machine with the spinning rust is loading a bunch of files with an I/O priority of "background" because it just got booted into Windows; that might slow it down a bit because of the seek times and I don't know if Windows is willing to starve background I/O for seconds at a time to speed up interactive requests (I doubt it).
Update: once all the background preloading is done, PowerShell restarts in three seconds on the spinning-rust machine.
Long story short, I think getting an SSD will be the thing that makes PowerShell start acceptably fast.
Jeffrey Snover [MSFT]
Left click in fwvm, select xterm, window appears in less than my blink response time.
Seriously: I think I might pop Win10 on an old Dell i5 that came with Win7 and play with this.
As people have mentioned the biggest factor here is probably your hard drive since you are loading maybe couple of 100's of small files when you load the ISE.