Yes, the very fact the the OP is now considering (however causally) the idea of finding a smaller host with more personal care is something I'd be concerned about if I were his current host. Who knows, maybe he'll go out and find an awesome small host he never knew existed. Customer gone, and he'll probably start recommending the new host to all his friends and associates too.

If it were me, and a good customer made a mistake, I'd eat the cost to keep the goodwill.