Actually, VLC can still display a separate playlist and playback window. You just need to uncheck "Show video within the main window" in the Preferences. http://feepk.net/customizing-vlc-for-mac has some other tips for making things look more like they do in 1.x.