I completely agree. Just think about your own behavior: do you choose the empty cafe or the one with many people inside?

Unless the all-dayers are actually blocking space that other customers could use, they are likelier to profit the business than harm it.