On multi-tasking OSes with memory protection, everything that accesses hardware requires drivers; it's 'just' the amount of driver code that differs (which, for USB vs optical, probably is quite a big difference)

Also, originally USB didn't support isochronous channels at all, so one couldn't guarantee that one could send audio out continuously.