>> they certainly get used, at least in my experience
As a taxpayer, this made me feel good, thank you for sharing.
Or was it a combination of that with "well its there, and we want to do this thing that would happen quicker on the big machine" - IE not necessarily enabling new research, just adding a nice speedup to existing work?
So no, these machines are often used for problems which really do utilize the large-scale nature of the computers.
So lots of the "big data" type problems with lots of uncoordinated jobs aren't going to take advantage of the primary component of these big computers, the node interconnect. The computers really shine with big fluid dynamics simulations where there's lots of chatter between different components of the simulation to communicate boundary conditions and the like.
Having done exclusively HPC for almost 16 years now, I can state that there is one and only one immutable law of HPC: grad students will find a way to squeeze out every second of available CPU time on a large system. The amount of effort going into taking advantage of those empty spots in the clusters where smaller jobs can run quickly while the scheduler drains resources for a large job (called backfilling) is tremendous.
The thing that sticks with me is what a fantastically complicated problem HPC job scheduling is. I've seen dozens of fresh-faced undergrads or first year grad students come in with a full head of steam and decide that they're going to "solve" HPC job scheduling, but every single one has gotten bogged down in the minutiae and the muck, and I've never seen software that "gets it right". Every scheduler sucks in its own unique way.
Whether all of those jobs need an old-fashioned supercomputer is certainly up for debate, but they certainly get used, at least in my experience.