This is why ground source heat pumps are suprisingly effective. You're drawing on the yearly solar inflow to the ground, at a depth where the temperature is quite stable, due to accumulating and averaging over years.

My curiosity in using things like heat pumps is that what happens when you move too much of that heat energy away. The heat might be doing nothing but, it has been there for a long time. At what point is too much heat displaced to the surface and what is the outcome of that?

You can read about what happened to the London underground over the years.

Except the process has been sort of reversed: there was heat being introduced into the deep layers of clay and rock.

https://en.wikipedia.org/wiki/London_Underground_cooling

For them to actually work the energy needs to be replaced, flowing from the hot areas to the (relatively) cold areas, mostly coming from the sun.

If you have a small area and continually pump the heat out then you end up with the inside of a freezer, which uses the same tech for exactly that purpose, but also intentionally insulates to prevent the heat getting back in.