There isn't necessarily a lot of research into packing these better - the basic algorithms have been unchanged for quite some time, and a lot more effort goes into deciding how to prioritize different groups that are sharing access into the same system.
97% is the highest specific value I can recall for any of the larger sites with a heavily mixed workload, absent having a nearly infinite supply of short+small jobs at hand to use to fill those gaps. And a lot of users aren't trying to chase that - for these large scale "capability" systems the goal is to scale out as large as you can anyways, the smaller stuff is usually relegated to "capacity" systems elsewhere with a less expensive architecture.
One thing that at least some schedulers can manage is the idea of a min+max runtime for a job, combined with a min+max node/cpu count. If you have users willing to 'scavenge' otherwise wasted time by running under such a regime that can put you closer to full usage.
There are separate debug queues for smaller development and test jobs that prioritize latency over throughput, and this is a main source of inefficiency (but, as someone who develops on supercomputers, trust me, it's necessary to make them usable).
As for the efficiency of the codes--there are a lot of barnacle-encrusted Fortran codes in astrophysics, chemistry, and elsewhere. Some of them probably contain plenty of inefficiency, but the cost of re-writing them is high. Oftentimes either the people writing the codes aren't interested in maintaining good software quality (because they're focused on the science) or the incentives aren't aligned to re-write code that's already established, but from the 80s (re-writing old code is generally not how one gets a Ph.D.). But they do very important things, like simulate nuclear fusion or the interaction of chemicals in the atmosphere that create weather patterns and storms, so it's very important to run them.