Physicists working on physics research might be a problem if those physicists could just divert money elsewhere in the economy into their own research programs. As it is, there's a complex system of controls so that public research money is generally used for some reasonably good purpose.