I've had this thrown at me on multiple occasions, at gigs where I've solved and addressed long-standing problems, hugely improved efficiencies, and taken systems from states of perpetual chaos to vastly smoother functioning conditions.
And the simple truth is, I will lose my effectiveness and sanity if I'm required to put in "face time" simply to maintain order elsewhere.
Ultimately I view complaints on that basis as an exceptionally strong indicator of management incompetence. Time to look elsewhere.
I'm being paid for my outputs, not inputs.
They may argue you could have accomplished the task 1 day early, saving them $X, etc.
I agree with your value proposition -- but I feel it's difficult to pull off in reality. I feel a lot more comfortable with billing Time and Materials rates, so basically if I skip out early one day, they aren't being billed for it, and the project goes on at it's pace.
Maybe close a couple of issues/send a bunch of emails at night so that people see you are not done at 5PM sharp? Visibility does not have to be physical, like "body in the office", but digital also.
There are cases when timing matters, such as if you're expected to collaborate with internal teams who have set schedules, but that has less to do with how many hours you spent doing X.