The problem for first stages is that some high ISP propellants are very non-dense. Liquid hydrogen is 70 kg per cubic meter. Kerosene is about ten times as dense. So the tanks need to be a lot bigger for the same payload. The same holds for pumps and turbines too. So the engines need to be bigger for the same thrust. Hydrogen is also annoying to deal with as it's much colder than liquid oxygen, the molecule is also much smaller and leaks much more easily.
Technically the fueled all-up hydrogen rocket might be lighter, so the engines should be smaller, but the density disadvantage eats back all that advantage. The dry mass, which is the actual cost driver, is about the same.
For upper stages, it makes sense because the cost driver is not the dry mass but the wet mass. Also they need less thrust so engine mass is a smaller proportion of stage mass.
In outer space, the rocket equation says that the delta-v is only a function of propellant mass and exhaust velocity. It is always better to have higher exhaust velocity, no exceptions.
But until you get to orbit, you experience the “gravity drag”. Gravity acts in the opposite direction of your motion, and reduces your speed, so you end up needing overall more delta-v.
In order to reduce the effect of the gravity drag, you want to accelerate as quickly as possible. That’s why astronauts have to be able to withstand high Gs for a few minutes. Ballistic missiles, which are not limited by the human limits can (and do) acceletate much faster. A Trident II gets to about 50 Gs.
To get to these insane accelerations, you need huge thrust. You achieve that at the cost of lower exhaust velocity (like in the case of the Space Shuttle). Lower exhaust velocity means lower delta-v, but higher thrust means lower delta-v losses due to the gravity drag, so in many cases this tradeoff makes sense.