Ah my bad, that was unrelated. Windows 10 had an option to show the Window titles of items on your taskbar if there was enough room.

Most of the time I end up having 2-3 VSCode windows open which you can't switch between unless you hover over the icon first.