For example, that's why a railway operator (a private company) can fine you when you cannot present a valid ticket. When you get on the train, this action creates a transportation contract between you and the railway operator.
There is also some additional fun extras in contract law, like the concept of fair terms. This very old idea is that a contract should be a balanced deal. Terms that unfairly give one party undue benefits can then be challenged as invalid.
US and state laws have a concept of "unconscionability" for extreme cases, as well as prohibitions on penalties, as opposed to prior agreements to fix damages based on reasonable estimates, and so on. But "the deal wasn't fair" isn't any general defense against breach of contract claims. Courts generally avoid digging into the business or other advisability of contracts. They err on the side of giving parties the deals they agreed to.
Perhaps you were speaking from the perspective of a different jurisdiction?
Here's an illustrative example that may or may not accurately correspond to the current state of the law in any particular jurisdiction, but it show why this implied contract stuff doesn't work in practice.
A farmer might put up a big notice saying: "By camping overnight on this field you agree to pay me £100 per vehicle per night." Then a bunch of travellers might camp there. Can the farmer sue the travellers for the £100 per vehicle per night? My guess is that he could only sue them for damaging his grass, because, despite what the notice said, the travellers didn't "agree"; they camped illegally and without permission, like they usually do. Probably they smashed up the notice to show what they thought of it. Therefore the farmer can only sue them for trespassing, not for breach of contract. (He might also sue them for the cost of replacing the notice if they did smash it up!)
Likewise, you could put a notice on the front door and every window of your house saying that by breaking into the house burglars agree to pay you X pounds. Would it help you in any way? Of course it wouldn't. You'd get laughed out of court if you tried it.
I'm sure that's legally valid. Perhaps it is in the USA.