In freight "backhaul" typically refers to transporting goods during the return journey. During the principle (non-return) journey, often the starting location is more central, like a distribution center, and the destination is a smaller satellite location like a store. So when something is backhauled, that tends to mean it's transported from the smaller satellite location to the central location.

Maybe that's where the term came from?

Maybe actually, or at the very least it could explain the confusion. Better than some of the other explanations I’ve heard for sure.

IME I think backhaul is just sending data to the main internet, not all backend. I thought it just meant it hauled the data back into the core network.

Main Internet, across main Internet, between networks, intra-domain. Intra-station. Anything that joins it all together that isn’t “front facing” i.e wireless network towards handsets

I've heard that term used where an ISP is piggbacking on a larger service. Sonic.net offers some of their services over AT&T infrastructure. Data to and from home DSL lines is "backhauled" to Sonic HQ in Santa Rosa, CA and then goes out over the bulk Internet backbone from there. This is a different path the data would take than if handled entirely by AT&T.