If the plane is going fast as well, that exhaust is more or less stationary relative to the ground. The engine works to exchange the position of the plane with the position of the air in front of it.
If the plane is going slow, it's accelerating the air backwards. That's where the work is going, making the engine less efficient.
Think about it this way: if the jet airplane is tied to the ground, its engines are running at 0% efficiency, working hard to blow the air backwards. You wouldn't want to stand behind a jet engine when the plane is about to take off, when that's effectively the case.
The same applies to propeller-driven planes, of course. But those can vary the prop speed as well as propeller pitch, having more control on how fast the air is being pushed backwards. This allows the engine to be efficient at a wider ranger of speeds, particularly, at the slower range.
But the propeller has a limit of how fast it can push the air back. When the prop blades start reaching the speed of sound, weird shit starts happening [1]. So propeller-driven aircraft have a limit on speeds at which they can go efficiently.
Jet engines (turbofans when it comes to airliners) trade off low efficiency at low speed / low altitude (where the airplane is spending a small percentage of flight time) for higher efficiency at high speed / high altitude.
Variable pitch turbine fans[2] aim to address this tradeoff, but the tech has yet to catch on.
[1] https://en.wikipedia.org/wiki/Republic_XF-84H_Thunderscreech
Jet needs to suck air from front. If air is stopped, sucking is hard. If air is already being thrown at you, you don't even need to suck, just let it come in.